Active-passive disaster recovery design spanning Azure primary and AWS secondary regions with automated failover, RPO under 15 minutes, and quarterly game-day validation.
A SaaS provider serving 2,800 enterprise tenants operated entirely from a single Azure region with backup tapes as the only DR mechanism, yielding an effective RPO of 24 hours and RTO untested beyond tabletop exercises. Customer contracts increasingly mandated sub-hour recovery objectives and geographic redundancy independent of a single cloud vendor. I designed an active-passive architecture with Azure as primary and AWS us-east-1 as warm standby, replicating application state, configuration, and data through a combination of native replication services and custom sync workers.
A regional Azure outage during a prior year caused 11 hours of customer-facing downtime, triggering SLA credits exceeding $800K. Enterprise renewal negotiations stalled on DR clauses requiring RPO ≤ 15 minutes and documented failover testing. Single-vendor concentration introduced procurement and compliance concerns for customers in regulated industries. Existing backup restore procedures had never been validated against production-scale data volumes within RTO targets.
Critical .NET APIs and worker services deploy to both Azure AKS and AWS EKS from the same Helm charts with environment-specific value overlays. PostgreSQL uses cross-region logical replication to Aurora PostgreSQL with lag monitoring and automatic promotion scripts gated by health check quorum. Blob storage replicates via Azure-to-S3 sync jobs with checksum verification, while DNS failover routes through Route 53 health-checked weighted records. Runbook automation via Terraform and Ansible executes failover in documented stages with manual approval gates for production promotion.
The DR topology employs a hub-and-spoke network model with VPN and Direct Connect bridging Azure VNet and AWS VPC for replication traffic isolated from customer-facing paths. Configuration management uses GitOps repositories synced independently to each cluster, with sealed secrets replicated via HashiCorp Vault replication. A DR orchestration service coordinates failover phases: traffic drain, replication lag validation, database promotion, cache warm-up, and DNS cutover. Observability stacks in both regions feed a unified incident dashboard with synthetic transaction probes validating end-to-end tenant workflows.
Key interfaces and system views from the engagement.
Additional work in adjacent domains and technology stacks.
Modular .NET order orchestration platform replacing a legacy monolith with bounded contexts, event-driven workflows, and multi-region deployment.
FHIR-native interoperability platform enabling secure clinical data exchange between hospitals, payers, and specialty networks with full audit trails.
Low-latency settlement and reconciliation platform processing equities and fixed-income trades with T+1 regulatory compliance and intraday risk controls.