1. Why near-zero recovery targets are a business decision
The recovery point objective (RPO) is how much recent data a business can afford to lose. The recovery time objective (RTO) is how long a service may stay unavailable. Near-zero values for both sound attractive, but each step toward zero costs money, network capacity and operational effort. The first task is to classify systems. A production order system and an internal wiki do not need the same targets, and treating them alike wastes budget on the wrong places.
Active-active means that two sites serve production traffic at the same time, not that one site sits idle as a cold standby. It can reduce downtime when a site fails, but it also means applications must tolerate data written at both sites. A network partition between the sites then forces hard choices about which side keeps accepting changes. Define those choices in writing before buying hardware, because the software behavior, not the hardware, determines the real outcome.
2. How storage replication behaves across sites
Ceph is a distributed storage system that spreads data across nodes and can replicate between clusters. The critical question is when a write is acknowledged to the application. In synchronous replication, the write is confirmed only after it reaches the remote site. This protects recent writes but adds latency to every write, and the distance between sites matters. In asynchronous replication, the write is confirmed after the local site stores it. Performance is better, but recent writes can be lost if the primary site fails.
Geo-replication is not a backup. A deleted table, a corrupted file or a volume encrypted by ransomware is copied just as faithfully as valid data. Keep point-in-time copies that are separate in time and in access rights from the replicated copy. Also consider quorum. A site that cannot see the majority of the cluster should stop accepting writes rather than diverge silently, because two diverging copies are much harder to reconcile than a short, planned pause.
3. Comparing replication and failover models
The right model depends on the data class and on the business cost of an interruption. Backup-only designs are simple and cheap to run, but recovery takes time and loses everything since the last copy. Asynchronous replicas with manual promotion balance cost and protection. Synchronous stretched clusters protect recent writes at the price of latency and network dependence. Active-active designs offer the best availability when the application is built for it.
Use the table as a starting discussion, not as a final decision. Ask each system owner which data they can lose, how long they can wait, and what they would do manually during an incident. Their answers usually show that a few critical services need the stronger models, while most workloads are well served by a tested asynchronous replica. Write the answers down.
| Model | Data at risk on site failure | Effect on write latency | Operational complexity |
|---|---|---|---|
| Backup only, restored on demand | Everything since the last usable backup | None during normal operation | Low |
| Asynchronous replica with manual promotion | Writes not yet replicated | Low | Medium |
| Synchronous stretched cluster | Minimal, but depends on the link | High, grows with distance | High |
| Active-active with conflict rules | Depends on how the application resolves conflicts | Moderate | Very high |
4. Common mistakes in active-active designs
The most frequent mistake is failing over the data but not the dependencies. Name resolution, identity services, certificates, message queues and software licenses may still point to the failed site. A failover test that only checks the storage layer proves very little. Test the whole service path, from the user's browser to the database and back, including the external integrations that the service needs to function.
Another mistake is never testing split-brain conditions, where both sites believe they are primary. Decide in advance which site wins when the link fails, and make the application enforce that decision rather than trusting an operator to notice. Also avoid assuming that a drill in a quiet period proves behavior during a real incident with full load, because queues, caches and retries behave very differently under pressure.
5. Checklist for a failover design
A checklist turns a design discussion into something a team can verify. The items below cover the decisions that most often remain implicit: targets, dependencies, replication modes and the rules that apply when the sites cannot agree. Each item should end with a written answer that the business owner has approved, because an unapproved target is only a hope. Keep the approvals with the checklist.
Run the checklist before any hardware purchase and again before go-live. Repeat it after major changes such as a new application, a new network provider or a change in the data volume. The goal is to keep the documented recovery behavior aligned with what the platform can actually do. Treat any gap as an open risk, with an owner and a date, until it is closed.
- Classify each service with an agreed recovery point and recovery time target.
- Map every dependency: name resolution, identity, certificates, queues and external integrations.
- Choose the replication mode per data class and document the expected data loss.
- Define the quorum and primary rules for a link failure, and who may override them.
- Keep at least one independent, point-in-time backup outside the replicated storage.
- Run a scheduled failover drill with the people who will act during an incident.
- Record how long a fail-back takes and who approves it.
6. What to ask a supplier or internal team
Ask which replication mode is used for each data class and what happens to writes when the link between sites is lost. Ask how the system decides which site is authoritative, and whether that decision is automatic or requires a person. Ask for the expected data loss in the worst documented case, stated in plain terms. A headline availability figure says little about what happens to your orders during a bad afternoon.
Ask how backups are kept separate from the replicated copy, how long they are retained and whether an administrator account can modify or delete them. Ask where each site is located and under which legal jurisdiction the data is held. Where data is stored and who can legally access it are separate questions, and a clear answer to both is part of a credible recovery design.
7. How to start: map one critical service
Pick one service whose outage would stop revenue or legal obligations. Write its current recovery point and recovery time based on what it actually achieves today rather than on what the plan says. This baseline often reveals that the existing backup is weaker than everyone assumed, and that is a useful finding in itself. Share the result with the owner of that service.
Next, run a tabletop exercise: walk through a site loss with the team, step by step, and note every manual action. Use the list of gaps to choose the first improvement, such as an independent backup copy or a documented promotion procedure. The first concrete step is to write down the current recovery figures for that service and have its owner sign them off.
Frequently Asked Questions
Is active-active always better than active-passive?
No. Active-active can reduce downtime when a site fails, but it adds consistency and operational complexity. Many organizations get most of the benefit from a well-tested primary and standby design. Choose the model by the real cost of an outage and by the engineering effort the team can sustain.
Does geo-replication protect against ransomware?
Not on its own. Encryption or deletion made on the primary site is copied to the replica. Protection comes from independent, point-in-time backups that cannot be changed by the production administrator account, and from regular restore tests that prove the copies are usable.
How far apart should the sites be?
It depends on the risks you want to cover and on the latency your applications can tolerate. Sites close together may share the same regional hazards. Sites far apart increase the latency of synchronous writes. Ask the supplier to show measured latency between the actual sites, not a generic figure.
What happens if the link between the sites fails?
That depends on the design. Some systems stop accepting writes on the minority side to avoid divergence. Others accept writes on both sides and reconcile later, which requires application rules for conflicts. Ask for the exact behavior, write it down and test it before you rely on it.
Do we need a specialist provider to run this?
Often yes, because the operations are demanding: monitoring, drills, capacity planning and incident handling all need steady attention. A managed service provider can take on part of that work, but the business must still define the recovery targets and their limits.
Is there a low-cost first step?
Yes. Document the recovery targets for the most critical services, restore one backup into an isolated environment to confirm it works, and make sure that dependencies such as name resolution and identity have a documented alternative if the primary site is lost.