AWS Disaster Recovery: 128-Minute Failover

The entire platform ran in a single AWS Region, with backups stored there too. The ISO 27001 auditor was waiting for evidence that the platform could be recovered, and two major live broadcasts were already on the calendar.

Multi-Region Disaster Recovery and ISO 27001 (NDA)

An ISO 27001-certified live streaming platform with an international audience. Kubernetes on AWS, API capacity of up to ~10,000 RPS and live broadcasts with 20,000–30,000 viewers on YouTube.

TEAM

2 Alpacked DevOps/Platform engineers

PERIOD OF COLLABORATION

August–September 2026; engagement ongoing

CLIENT’S LOCATION

Germany

Live Streaming on AWS, with Backups in the Same Region as Production

Our client is an international live streaming platform delivering video and interactive content through the web, mobile applications. Streamers drive its traffic: a single popular broadcast can multiply the load within minutes. The frontend is built with Next.js, API services run on Kubernetes on AWS, data is stored in MongoDB Atlas, and Redis handles caching.

The entire infrastructure ran in a single AWS Region, Frankfurt, along with secrets, container images, and backups. The backups had been fixed earlier during ISO audit remediation, but no one had ever performed a full recovery from them. A backup that has never been restored is an assumption, not a safeguard.

The work was triggered by an ISO 27001 audit that identified two nonconformities: backup testing and disaster recovery. The first was closed in August, while the second was held up by internal communication with the auditor as the deadline approached. Meanwhile, the platform was preparing for two major live broadcasts that required additional capacity. Two of our engineers worked on these tasks alongside the client’s development, security, and sysadmin teams.

Over two months, we built infrastructure in a second region, failed over the core customer-facing services, and measured the time it took. Regular releases continued throughout.

Key Results in Numbers

icon

128 min to fail over core services during testing

icon

2 of 2 ISO 27001 findings closed; audit passed

icon

9 automated disaster recovery tests

icon

20–30K concurrent YouTube viewers; no noticeable rise in platform errors

Without Tested Disaster Recovery: Why It Matters for Live Streaming

In live streaming, peak load coincides with peak audience attention. A broadcast cannot simply be pushed back an hour. A viewer who cannot open the stream may not return, and everyone the event attracted sees the failure at once.

For the platform, relationships with streamers are also at stake. If a streamer brings an audience but the platform struggles, they may lose trust in it and take future broadcasts elsewhere.

A regional outage raises the stakes further. A platform that can handle a traffic spike but cannot survive the loss of a region remains vulnerable to a less frequent but far more extensive failure.

“Everything we have is in Frankfurt: the databases, secrets, images, and backups too. If we lose that region, we lose both the platform and the way back. Nobody could tell us how long recovery would take, let alone prove it.” – Client representative

The constraints were tight, too. Development could not stop: the platform continued shipping features and running live broadcasts throughout. The external auditor set the timeline, not the engineers. The recovery region had to be sized for recovery rather than duplicate production and double the AWS bill.

The test had to involve an actual failover on live infrastructure, rather than a tabletop exercise that walked through the scenario on paper. A tabletop exercise would neither satisfy the auditor nor provide a measured recovery time.

Our analysis identified three technical root causes.

1

Challenge #1: A Single AWS Region as a Single Point of Failure

Databases, secrets, container images, and backups were all stored in one region. A regional outage would have made both the system and its recovery resources unavailable.

3

Challenge #3: S3 Cross-Region Replication Depended on the Development Team

For this platform, enabling native S3 Cross-Region Replication required changes to how the application wrote objects. Those changes needed the development team’s involvement, so the infrastructure team could not complete the setup on its own

2

Challenge #2: No Recovery Region or Runbook

There was no infrastructure in a second region, so even the best procedure would have had nowhere to fail over the services. There was no procedure either: recovery depended on which engineers were available and what they remembered.

Multi-Region Disaster Recovery: A Recovery Region First, Then a Runbook and a Timed Failover

Documentation alone was not enough for this project: the auditor expected evidence that recovery worked. So we started with the infrastructure. First, we built an environment to recover into, then documented the procedure, performed an actual failover, and measured the time.

An AWS Recovery Region Sized for Recovery
We chose London as the recovery region and replicated data, secrets, container images, and S3 objects there. The infrastructure was sized for recovery rather than a full production replica, so the AWS bill did not double. We handle projects like this through our AWS infrastructure consulting services.

A Real Failover and 9 Automated Tests
During testing on live infrastructure, we failed over core customer-facing services to the second region in 128 minutes. To make the exercise repeatable, we built a suite of 9 automated tests. They separately cover regional failover and resilience to Availability Zone failures (multi-AZ).

A Runbook and Policies for the ISO 27001 Audit
We wrote a step-by-step recovery runbook and 15 internal policies so engineers who had not built the recovery process could carry it out. We escalated the audit blocker to management representatives rather than waiting for it to resolve itself.

Why a DR Plan Without Infrastructure Would Not Have Resolved the ISO 27001 Nonconformity

Rejected approach: documentation alone. Starting with documentation alone was not enough here. The auditor expected evidence that recovery worked, and a plan to fail over to an empty region could not provide it. The infrastructure had to come before the procedure.

Chosen approach: build recovery infrastructure and perform an actual failover. This gave us a measured recovery time and evidence for the auditor. It also gave the team a procedure they could rely on during a real disaster.

S3 Cross-Region Replication Instead of Lambda: Fewer Points of Failure During Recovery

The main obstacle was coordination with the development team, rather than infrastructure. For this platform, enabling native S3 replication required changes to how the application wrote objects, and the development team was reluctant to make them.

The obvious workaround was a Lambda-based copy pipeline. It would have worked, but we deliberately rejected it. A Lambda copy pipeline adds a separate custom component that can fail independently, within the very mechanism that must work when the primary region is unavailable.

We followed up with the development team daily for a week until the application changes were shipped. S3 objects now replicate natively, without a separate custom copy service.

An Organisational Audit Blocker: When More Documentation Will Not Help

The DR nonconformity was held up by communication with the auditor, rather than engineering tasks. Additional documentation would not have changed that.

Escalating to management representatives resolved the issue. We initiated that escalation alongside the technical work. Distinguishing technical blockers from organisational ones helped us avoid delays and meet the deadline.

The Hardest Part: Making Regional Failover Repeatable

Performing a regional failover on a live platform without disruption is difficult. Making it repeatable is even harder.

A one-off failover demonstrates a result for a specific infrastructure state. Changes that affect recovery require another test. To turn the exercise into automated tests, we had to answer several questions.

What exactly does “recovered” mean for each class of service? In what order must services come back online? Which settings are still tied to Frankfurt without that dependency being documented?

Answering these questions took most of the work, rather than the failover itself. At the next audit, the team will be able to present the results of repeated recovery tests.

Custom Solutions: DR Tests, a Runbook, and Replication 

A suite of 9 automated disaster recovery tests. Built specifically for this platform, the tests separately exercise regional failover and Availability Zone failure scenarios. They verify that core customer-facing services actually serve requests afterwards, rather than merely having pods scheduled on Kubernetes nodes. The client retains the suite and can run it independently.

A recovery runbook with an ordered sequence of steps. It defines the recovery order for each class of service and the dependencies between them. Alongside it, we wrote 15 internal policies to make the procedures clear to the auditor and executable by other engineers.

Replication to the recovery region. The configuration covers data, secrets, and container images. We included images and secrets as essential recovery dependencies. For S3, we modified the application to enable native replication without an intermediary copy service.

Architecture Before and After: From Single-Region to Multi-Region

Before

  • A single region, Frankfurt, held everything: services, databases, secrets, images, and backups.
  • No infrastructure in a second region.
  • No S3 cross-region replication.
  • No end-to-end recovery test or documented procedure.
  • Recovery knowledge held by individual engineers.

After

  • A primary region in Frankfurt and a recovery region in London.
  • Data, secrets, images, and S3 objects replicated to the recovery region.
  • Native S3 replication without a separate copy service.
  • Core services failed over to London in 128 minutes during testing on live infrastructure.
  • 9 automated tests covering regional failover and multi-AZ resilience separately.
  • A recovery runbook and 15 internal policies.

Technology Stack

  • Cloud: AWS across two regions (EKS, S3, CloudFront, Route 53, Secrets Manager, ECR, WAF, managed data stores)
  • Containers and orchestration: Kubernetes, Helm
  • IaC and CI/CD: Terraform, GitHub Actions
  • Data: MongoDB Atlas, Redis
  • Application and analytics: Next.js, Metabase
  • Monitoring: Datadog

Technical Highlights

  • Native S3 Cross-Region Replication instead of a Lambda-based copy pipeline.
  • Separate testing of multi-region and multi-AZ resilience as distinct failure domains, rather than a single vague “high availability” claim.
  • Replication of container images and secrets as essential recovery dependencies.
  • Disaster recovery tests maintained as code.

Results: A 128-Minute Failover, a Passed ISO 27001 Audit, and Disruption-Free Broadcasts

The platform runs on the same technology stack as before. The difference is that it now has a tested recovery procedure for a regional outage, with a recovery time measured during testing.

1

Reliability: Recovery Time Measured on Live Infrastructure

During testing, core customer-facing services failed over to the second region in 128 minutes. Resilience to Availability Zone failures was verified separately. All stateful resources, secrets, and images are now available in the recovery region, so recovery does not depend on Frankfurt being available.

The backup nonconformity was closed in August, the disaster recovery nonconformity in September, and the audit passed. The suite of 9 tests enables repeat validation of recovery time after infrastructure changes. 

3

Costs: A Recovery Region Without Doubling the AWS Bill

The recovery region is sized for recovery rather than duplicating production, so the 128-minute result was achieved without doubling infrastructure costs. Native S3 replication avoided Lambda invocation costs and the maintenance and monitoring of a separate copy service.

The main financial benefit, however, does not appear on the bill. The platform’s revenue is concentrated in live events, yet previously it had no prepared way to recover in another region during a regional outage. 


2

Delivery: From Zero to Working DR in Two Months

Moving from no recovery infrastructure to a timed failover took roughly two months, without a development freeze. In September alone, alongside the DR work, the team closed 26 tickets and merged 18 pull requests from DevOps engineers. Recovery testing can now be a scheduled task rather than a separate project.

4

Viewers and Business: 20–30K Viewers Without Disruption 

Viewers noticed no disruption from these changes. In the same month as the recovery test, the platform ran two live events, including a broadcast by one of Brazil’s largest casino streamers with 20,000–30,000 concurrent YouTube viewers. The platform’s error rate showed no noticeable increase, and there were no capacity issues. Load tests with caching disabled reached 1,000 RPS; after optimisation, the API handled roughly 10,000 RPS on the same resources.

The company can now demonstrate its security and recovery controls to the auditor through test results. 

Have Backups but Never Tested Recovery?

We’ll test what happens to your platform if an AWS Region goes down and how long recovery actually takes.