Context & constraints
This was a modernization of both the hardware and the software architecture of a mission-critical system for a national telecom operator with more than 100 million subscribers. The platform pairs a CBS (Convergent Billing System) with a CRM, sustaining roughly 500 transactions per second of live customer and billing activity.
The hard constraint set the tone for every decision: billing cannot lose orders. If a recharge, an activation, or a charging event is dropped, that is real money and real customer trust gone. On top of that, the cutover from the legacy platform to the new one had to be invisible to subscribers — no outage window, 100% functional on day one. Losing a whole data center to fire, power, or network had to be a survivable event, not a catastrophe.
The three sites
The platform runs across three data centers in the same country but different cities, each more than 500 km apart — far enough to survive a regional disaster, which in turn rules out synchronous replication over that distance. The topology is Active-Active-Standby; the interactive 3D scene above maps one-to-one to it (click a site for its role):
- DC-1 — Active. Serves live production traffic. Full Oracle-backed CBS/CRM stack running on Kubernetes, with dedicated storage: ~1 PB for the database tier and ~500 TB for the application tier, each shadowed by a backup storage device of the same size.
- DC-2 — Active. Serves live traffic alongside DC-1 and is deliberately sized so that, in the worst case, a single active site can absorb 100% of national traffic on its own.
- DC-3 — Warm standby. Continuously receives database and application replication so its data is current, but does not take live traffic until promoted during a full active-site loss.
Integration & failover design
Traffic enters through an F5 that fronts DC-1 and DC-2 under a single DNS name, applying weighted, proximity-aware load balancing — a request is served by whichever active site is closest to it. The F5 continuously health-checks each site by probing an HTTPS endpoint and requiring an HTTP 200 OK from the Nginx ingress in front of the application. If a site stops returning 200, the F5 simply stops sending it traffic and the surviving active site carries the full load — automatically, with no human in the loop.
Inside each site, Kubernetes is the engine: it runs the CBS/CRM workloads and orchestrates many other supporting nodes. Traffic splits into two planes — east-west traffic flows between the CRM and CBS solutions, while north-south traffic arrives through an API gateway that lets external channels reach the application through the Nginx layer.
The honest tradeoff lives at DC-3. The active pair fails over to each other instantly because they live behind one F5 and one DNS record — but DNS replication was limited to two sites. Promoting the warm standby therefore requires a DNS change that has to propagate, so DC-3 activation is a deliberate, slower step reserved for the loss of an active site, and failover/switchover can also be triggered manually when operators need control. To protect billing through that window, the system is designed to capture 99.99% of orders and hold them in a wait state during a disruption, draining the queue once service is restored — so no billing event is silently dropped.
Steady state: DC-1 & DC-2 serve live traffic behind one F5; both replicate continuously to the DC-3 warm standby.
Results
The platform went live with zero downtime during cutover and full functionality from day one, and the VM-to-Kubernetes migration completed without data loss. Disaster-recovery runbooks were not left on paper: recovery plans and failover drills were executed against the system months ahead of go-live, and functionality was verified repeatedly before delivery. The single most satisfying milestone was moving the project's status from delivery to maintenance — the signal that the design held under real operations.
Leadership & lessons learned
As Infrastructure DevOps Team Leader I also mentored several early-career engineers through this delivery, and watching their work mature into production-grade contributions was as rewarding as the architecture itself. A practical organizational lesson: small teams are easy to coordinate, but a large delivery only works when those small teams operate as one whole toward shared development and deployment goals.
If I did it again, I would carry less unnecessary worry and manage my time more deliberately. The tool I picked up during the delivery — and wish I'd had at the start — is the Eisenhower (importance vs. urgency) matrix: Do, Decide, Delegate, Delete. It is now how I triage under pressure.
Anonymization note
Operator identity, exact locations, and proprietary configuration are intentionally omitted. Figures are rounded and the architecture is described at a design level; industry-standard components (Kubernetes, Oracle, F5, Nginx) are named because they are generic and non-sensitive. Everything here is presented to be honest and defensible in a technical interview.