Overview
The client needed to bring Jira data from an on-premises environment into Azure Data Lake Storage Gen2 for centralized reporting, analytics, and downstream data modeling. Jira contained important project, issue, user, workflow, and activity data, but the environment was not cloud-native and could not simply be accessed through a public endpoint.
Challenge
- Jira hosted on-premises with no public inbound access allowed.
- Required custom extraction across issues, projects, users, comments, worklogs, boards, sprints, and custom fields.
- Outbound HTTPS/TLS access from the SHIR to ADLS DFS/Blob endpoints was blocked.
Solution
MSPowerhouse implemented a secure on-premises integration architecture using Azure Data Factory and a Self-hosted Integration Runtime. The runtime server acted as the secure bridge between the internal Jira environment and Azure, allowing the pipeline to reach Jira without exposing the Jira server publicly.
Instead of relying only on a generic connector, MSPowerhouse designed the integration using Jira REST API patterns where needed. This allowed better control over JQL, custom fields, pagination, comments, worklogs, and incremental extraction logic. Data was landed into Azure Data Lake Gen2 in raw form first, preserving the original source output for traceability before downstream transformation or reporting.
When the pipeline encountered connectivity issues, MSPowerhouse isolated the problem to outbound network access from the runtime server to Azure storage endpoints. Once outbound HTTPS/TLS access was corrected, the copy process resumed and data started flowing successfully into the lake.
Technical Execution
- Azure Data Factory orchestration.
- Self-hosted Integration Runtime for on-premises Jira access.
- Jira REST API extraction for issues and related entities.
- JQL-based filtering for targeted and incremental pulls.
- Pagination handling for large Jira result sets.
- Raw JSON landing into Azure Data Lake Gen2.
- Secure outbound-only connectivity.
- Validation of Azure Data Lake DFS and Blob endpoint access.
- Troubleshooting across Jira, SHIR, firewall, TLS, and ADLS layers.
- A repeatable pattern for future Jira objects such as projects, users, comments, worklogs, boards, and sprints.
Outcome
The client gained a working integration path from on-premises Jira into Azure Data Lake Gen2. Jira data that was previously locked inside an on-premises system became available for reporting, analytics, and future modeling.
Impact
This project demonstrated MSPowerhouse's ability to handle real-world hybrid integration problems. The work was not just about creating an ADF pipeline. It required understanding Jira APIs, on-premises networking, self-hosted runtime behavior, Azure Gov/GCC-style endpoint access, storage permissions, and data lake landing patterns.
