Scalable ETL platform ingesting clinical, claims, and operational data into a governed lakehouse for population health analytics and value-based care reporting.
An accountable care organization needed a unified analytics foundation to measure quality metrics, track attributed patient populations, and report on shared savings under value-based contracts. Data resided in siloed EHR extracts, claims feeds, and manual spreadsheets with inconsistent patient identifiers and delayed refresh cycles. I architected a medallion lakehouse pipeline using Azure Data Factory for orchestration, Databricks for transformation, and Delta Lake for ACID-compliant storage with time travel for audit replay. The platform serves curated datasets to Power BI and external quality registries through row-level security aligned to care team assignments.
Quality metric reporting lagged 45 days behind clinical activity, preventing timely care gap closure interventions. Analysts spent 60% of their time reconciling conflicting patient counts across claims and clinical sources. Value-based contract calculations lacked auditable lineage, creating disputes with payer partners over attributed population denominators. HIPAA compliance reviews flagged uncontrolled PHI copies on analyst workstations fed by ad hoc extracts.
Azure Data Factory pipelines ingest HL7 extracts, X12 claims, and reference data on scheduled and trigger-based cadences, landing raw files in bronze Delta tables with ingestion metadata. Databricks notebooks apply standardized de-identification, patient matching, and clinical concept mapping (ICD, CPT, LOINC) in silver layers. dbt models build gold-layer quality measure calculations with unit tests and schema contracts enforced in CI. Access is brokered through Synapse serverless SQL with Azure AD groups and column-level masking for sensitive attributes.
The pipeline follows medallion architecture: bronze for immutable raw ingestion, silver for cleansed and conformed entities, gold for business-level aggregates and measure definitions. Orchestration dependencies are DAG-modeled in Data Factory with checkpoint files enabling idempotent replays after failure. Spark jobs run on autoscaling Databricks clusters sized by workload class, with job clusters terminated after batch completion to control cost. Metadata and lineage are captured in Azure Purview, linking source systems to downstream reports for audit requests.
Key interfaces and system views from the engagement.
Additional work in adjacent domains and technology stacks.
Modular .NET order orchestration platform replacing a legacy monolith with bounded contexts, event-driven workflows, and multi-region deployment.
FHIR-native interoperability platform enabling secure clinical data exchange between hospitals, payers, and specialty networks with full audit trails.
Low-latency settlement and reconciliation platform processing equities and fixed-income trades with T+1 regulatory compliance and intraday risk controls.