Skip to main content

17. High Availability, Disaster Recovery & Business Continuity Planning

Learning objective: design, test and operate Databricks platforms that remain available through component failures, recover predictably from regional disasters, protect business-critical data, and provide evidence that recovery objectives are achievable.

Phase 15 taught you how to operate the platform.

Phase 16 taught you how to engineer performance and cost.

This phase answers a harder enterprise question:

What happens when the platform, a region, a dependency, or the data itself fails — and how do we prove that the business can recover?

This is not a backup tutorial. It is a resilience architecture discipline.

Enterprise Databricks resilience architecture

Databricks distinguishes high availability inside a region from disaster recovery across regions. Current Databricks guidance recommends managed disaster recovery for supported cross-region deployments; DIY patterns remain relevant where managed DR does not cover the required resources or where an organization needs active-active, cross-cloud or finer-grained control. Managed DR is currently gated, so an architect must know both the managed model and the engineering principles behind a customer-controlled design.

1. HA, DR and BCP are not the same thing​

High availability​

HA reduces interruption from failures inside the normal operating environment.

Typical failure domain:

Region
  ├── AZ-A
  ├── AZ-B
  └── AZ-C
       ↓
Service continues

HA answers:

"Can the service continue when a component or availability zone fails?"

Disaster recovery​

DR restores service after a larger failure, typically involving a separate region.

PRIMARY REGION                         SECONDARY REGION
Active workload                       Recovery workload
       │                                      ▲
       ├────── replicated data ───────────────┤
       ├────── replicated metadata ───────────┤
       └────── replicated assets ─────────────┘
                         │
                     FAILOVER

DR answers:

"Can we restore the business service after losing the primary deployment?"

Business continuity​

BCP is broader than Databricks.

It includes:

  • people and ownership;
  • alternate operating procedures;
  • critical business processes;
  • upstream sources;
  • downstream consumers;
  • BI;
  • identity;
  • networking;
  • communication;
  • vendors;
  • compliance evidence.

A Databricks DR design that cannot reconnect the source ERP or BI platform is not a complete business continuity solution.

2. The four recovery objectives​

Every enterprise workload should have explicit objectives.

ObjectiveMeaningArchitect question
RPOMaximum tolerable data lossHow much data can we lose?
RTOMaximum tolerable recovery timeHow quickly must service return?
RLORecovery level objectiveWhat exact functionality must return?
SLA/SLONormal operating commitmentWhat availability/freshness do we promise?

Example:

Finance reporting: RPO ≤ 15 minutes, RTO ≤ 60 minutes, RLO = executive dashboards + regulatory extracts.

This immediately changes architecture decisions.

A 15-minute RPO cannot be satisfied by a nightly copy.

A 5-minute RTO cannot be satisfied by a completely manual rebuild.

3. Start with business criticality​

Do not make every workload mission-critical.

Classify workloads first:

TierExampleRPORTOTypical pattern
Tier 0Regulatory / revenue-criticalnear-zero to minutesminutesactive-passive or active-active
Tier 1Enterprise operational analyticsminutesless than 1 hourwarm/passive standby
Tier 2Department analyticshourshoursscheduled replication
Tier 3Development / experimentsdaysdaysreconstruct from source

These are design targets, not universal Databricks defaults.

The architect owns the business conversation that establishes them.

4. Failure-domain thinking​

Always ask:

  1. What can fail?
  2. What is the blast radius?
  3. What state is lost?
  4. What state is replicated?
  5. What state can be reconstructed?
  6. How do we detect the failure?
  7. How do we prevent split-brain?
  8. How do we recover?
  9. How do we prove recovery?

A useful hierarchy is:

TASK / PROCESS
      ↓
COMPUTE NODE
      ↓
AVAILABILITY ZONE
      ↓
WORKSPACE
      ↓
REGION
      ↓
CLOUD / PROVIDER
      ↓
EXTERNAL DEPENDENCY

Each level requires a different resilience strategy.

5. What Databricks HA already gives you​

Do not build a second DR system to solve a problem that HA already solves.

Databricks documents intra-region HA for control-plane services and describes compute-plane behavior such as replacement/restart across availability-zone failures. Customer-managed infrastructure still requires deliberate choices around storage, networking and other dependencies.

The architect's rule:

First understand the platform's native failure guarantees. Then add resilience only for the remaining failure domains.

This avoids unnecessary complexity and cost.

6. The enterprise DR boundary​

A Databricks DR architecture is not just:

Workspace A → Workspace B

It is:

                    BUSINESS SERVICE
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
       Sources           Databricks          BI
          │                │                │
     replication       workspace/UC       reconnect
          │                │                │
          └──────────── REGION A ───────────┘
                           │
                     DR replication
                           │
          ┌──────────── REGION B ───────────┐
          │                │                │
       Sources           Databricks          BI
          │                │                │
          └──────────── recovery ────────────┘

The DR boundary must include dependencies that are actually required to deliver the business service.

7. Unity Catalog changes the architecture​

Unity Catalog is regional: Databricks documents one metastore per region, with workspaces in that region attached to that metastore. For cross-region sharing, Databricks recommends Databricks-to-Databricks OpenSharing rather than registering the same shared tables as external tables in multiple metastores.

Therefore:

Region A
  Workspace A1 ─┐
  Workspace A2 ─┴── Metastore A

Region B
  Workspace B1 ─┐
  Workspace B2 ─┴── Metastore B

             │
       cross-region
         sharing

Do not assume that a single global metastore eliminates regional architecture.

Also distinguish:

  • metadata replication;
  • managed table data replication;
  • external data replication;
  • workspace asset replication;
  • permissions and identities;
  • secrets and configuration.

They are different recovery domains.

8. Managed DR vs DIY DR​

Current Databricks managed DR can replicate selected Unity Catalog metadata, managed-table data and workspace assets between supported primary/secondary deployments and provides failover orchestration. It is gated and requires specific prerequisites.

Decision framework​

RequirementManaged DRDIY
Standard supported cross-region recoveryPreferredPossible
Custom replication logicLimitedStrong
Active-activeEvaluate scopeRequired where supported
Cross-cloudNot the default answerRequired
Fine-grained replicationLimitedStrong
Operational simplicityBestHigher effort
Custom compliance workflowEvaluateFlexible

Architectural principle:

Use the managed capability when it satisfies the business requirement. Do not build custom replication merely to prove engineering capability.

9. Active-passive is the default enterprise starting point​

Most organizations should begin with active-passive.

REGION A                         REGION B
PRIMARY                          SECONDARY
────────                         ────────
Users                            Standby
Jobs                             Replicated jobs
Compute ACTIVE                   Compute OFF / MINIMAL
Data                             Replicated data
UC metadata                      Replicated/recreated metadata

              ───── FAILOVER ────►

REGION A                         REGION B
Unavailable                      ACTIVE

Benefits:

  • simpler operations;
  • lower steady-state cost;
  • easier ownership;
  • lower split-brain risk;
  • easier testing.

Trade-off:

  • potentially higher RTO;
  • some capacity is idle;
  • recovery requires a controlled promotion.

10. Active-active is not "two copies"​

Active-active means both regions process production workloads concurrently.

That introduces:

  • duplicate processing;
  • conflict management;
  • idempotency requirements;
  • cross-region consistency questions;
  • independent failure detection;
  • global routing;
  • job completion semantics;
  • higher cost.

A simplified model:

              GLOBAL TRAFFIC
                /       \
               /         \
          REGION A      REGION B
          ACTIVE         ACTIVE
             \           /
              \         /
             CONSISTENT DATA

Databricks notes that active-active is more complex and costly, but can offer the lowest RTO/RPO when engineered correctly.

Architect rule: choose active-active because the business requires its recovery characteristics, not because it looks more advanced.

11. Data recovery is different from platform recovery​

This distinction prevents major DR mistakes.

Platform recovery​

Recover:

  • workspace assets;
  • jobs;
  • permissions;
  • SQL warehouses;
  • configuration;
  • networking;
  • identities;
  • secrets;
  • monitoring.

Data recovery​

Recover:

  • Delta tables;
  • source data;
  • checkpoints;
  • streaming state;
  • external systems;
  • historical versions;
  • audit records.

Databricks explicitly warns that DR does not protect against logical data corruption: corrupted primary data can be replicated to the secondary. Use Delta time travel and appropriate backup mechanisms for corruption scenarios.

Therefore:

DR protects availability. Backup/versioning protects recoverability from corruption.

12. Delta Lake and recovery​

Delta provides valuable recovery primitives:

  • transaction history;
  • versioned commits;
  • time travel;
  • ACID transactions;
  • idempotent design patterns.

But time travel is not a complete enterprise backup strategy.

Ask:

  • Is the required historical version still retained?
  • Was VACUUM already executed?
  • Is the underlying object storage available?
  • Was the corruption propagated to the DR copy?
  • Do we need an independent backup?
  • Can we restore a consistent business state across multiple tables?

For critical workloads, define retention and recovery policies deliberately.

13. Streaming recovery​

Streaming introduces state.

A recovery plan must understand:

SOURCE
  │
  ▼
CHECKPOINT
  │
  ▼
STREAM PROCESSING
  │
  ├── STATE
  │
  ▼
DELTA TARGET

During DR:

  1. identify the last known safe checkpoint;
  2. determine what source data remains available;
  3. restart from a known recovery point;
  4. make processing idempotent;
  5. detect possible duplicates;
  6. reconcile target state;
  7. validate business completeness.

Databricks' DR guidance specifically calls out finding the streaming recovery point and preparing duplicate elimination during failover testing.

14. Infrastructure as Code is a DR capability​

If production infrastructure exists only through manual UI clicks, recovery becomes a memory exercise.

Version-control:

  • workspace configuration;
  • Unity Catalog definitions;
  • grants;
  • jobs;
  • pipelines;
  • compute configuration;
  • network configuration;
  • environment parameters;
  • monitoring configuration;
  • secrets references;
  • deployment policies.

Use CI/CD and Declarative Automation Bundles where appropriate.

The recovery equation becomes:

SOURCE CODE
+ INFRASTRUCTURE DEFINITION
+ CONFIGURATION
+ DATA REPLICATION
+ RUNBOOK
= RECONSTRUCTABLE PLATFORM

15. Separate code from regional configuration​

Never hard-code:

  • bucket names;
  • workspace URLs;
  • region-specific endpoints;
  • secret values;
  • external locations;
  • network identifiers.

Prefer:

COMMON CODE
    +
ENVIRONMENT CONFIG
    +
REGION CONFIG
    +
SECRET REFERENCE

This makes the same deployment artifact usable in primary and secondary regions.

16. Failover is a controlled state transition​

A mature DR process looks like:

NORMAL
  │
  ▼
INCIDENT DETECTED
  │
  ▼
ASSESS IMPACT
  │
  ├── recover locally ──► NORMAL
  │
  └── regional failure
          │
          ▼
     DECLARE DR
          │
          ▼
     FREEZE PRIMARY
          │
          ▼
   VERIFY SECONDARY
          │
          ▼
     PROMOTE / FAILOVER
          │
          ▼
    REPOINT DEPENDENCIES
          │
          ▼
      VALIDATE DATA
          │
          ▼
      RESUME SERVICE
          │
          ▼
       OBSERVE
          │
          ▼
      FAILBACK PLAN

Never allow both regions to unknowingly process the same business stream unless the architecture explicitly supports active-active semantics.

17. Failback is a separate engineering problem​

Recovery is not complete when the secondary is running.

Eventually:

PRIMARY FAILED
      ↓
SECONDARY ACTIVE
      ↓
PRIMARY REPAIRED
      ↓
DATA / ASSETS RECONCILED
      ↓
PRIMARY VALIDATED
      ↓
CONTROLLED FAILBACK
      ↓
SECONDARY RETURNED TO STANDBY

Databricks' current DR guidance includes explicit failback procedures and recommends testing them rather than assuming failover automatically proves reversibility.

18. DR runbook design​

Every production service should have a concise runbook.

Before incident​

  • confirm replication health;
  • confirm RPO;
  • confirm secondary capacity;
  • validate credentials;
  • validate network paths;
  • validate source availability;
  • validate BI connectivity;
  • verify monitoring.

During incident​

  • declare incident owner;
  • determine blast radius;
  • stop unsafe processing;
  • protect against split-brain;
  • record recovery point;
  • activate secondary;
  • repoint dependencies;
  • execute smoke tests;
  • communicate service status.

After recovery​

  • reconcile data;
  • measure actual RPO/RTO;
  • identify missed workloads;
  • preserve evidence;
  • document deviations;
  • decide failback timing;
  • conduct post-incident review.

19. Testing strategy​

A DR architecture that has never been tested is an assumption.

Use progressive tests:

Level 1 — Documentation test​

Can a new engineer follow the runbook?

Level 2 — Component restore​

Restore one asset or workload.

Level 3 — Controlled failover​

Switch a non-critical workload.

Level 4 — Full business service failover​

Recover the complete dependency chain.

Level 5 — Regional simulation​

Exercise the entire operating model.

Measure:

  • actual RPO;
  • actual RTO;
  • detection time;
  • decision time;
  • failover time;
  • validation time;
  • data reconciliation time;
  • failback time.

20. Chaos and resilience testing​

Do not only test the happy path.

Inject scenarios such as:

FailureExpected learning
Driver failureCompute recovery
AZ disruptionPlatform resilience
Workspace unavailableWorkspace recovery
Object storage inaccessibleData dependency recovery
Source system unavailableIngestion resilience
Secret expiredIdentity/config recovery
Network route brokenConnectivity recovery
Corrupted tableData recovery
Bad deploymentRelease rollback
Region unavailableFull DR

The objective is not to create outages.

The objective is to discover assumptions before production does.

21. DR cost engineering​

Resilience has a price.

Typical cost drivers:

  • standby compute;
  • replicated storage;
  • cross-region data transfer;
  • duplicated environments;
  • monitoring;
  • network infrastructure;
  • operational staffing;
  • testing;
  • backup retention.

Compare:

EXPECTED BUSINESS LOSS
        vs
COST OF RESILIENCE

A Tier 3 development environment may not justify warm standby.

A revenue-critical payment analytics workload may.

22. Architecture decision matrix​

PatternRPORTOCostComplexityTypical use
Rebuild from IaChours/dayshours/daysLowLowDev/non-critical
Backup + restorehourshoursLow/MediumMediumModerate criticality
Active-passiveminutesminutes/hoursMediumMediumEnterprise default
Warm standbylowlowMedium/HighHighMission-critical
Active-activevery lowvery lowHighVery HighExtreme requirements

Do not select a pattern before defining the business requirement.

23. Multi-region Unity Catalog design​

A strong enterprise topology might look like:

                    ACCOUNT
                       │
          ┌────────────┴────────────┐
          │                         │
       REGION A                  REGION B
          │                         │
    ┌─────┴─────┐             ┌─────┴─────┐
    │ Metastore │             │ Metastore │
    │     A     │             │     B     │
    └─────┬─────┘             └─────┬─────┘
          │                         │
    Workspaces                 Workspaces
          │                         │
          └──── regional sharing ───┘

Use regional boundaries deliberately. Databricks recommends one metastore per region as the default pattern and Databricks-to-Databricks OpenSharing for cross-region access.

24. Architect checklist​

Before approving a production DR design, ask:

Business​

  • What is the critical business service?
  • What are RPO/RTO/RLO?
  • What data loss is acceptable?
  • Who can declare disaster?

Platform​

  • What is protected by native HA?
  • What requires regional DR?
  • What does the selected DR capability actually replicate?
  • What must be rebuilt?

Data​

  • Where is the source of truth?
  • How is data replicated?
  • How is corruption handled?
  • How are streaming checkpoints recovered?

Security​

  • Are identities available in DR?
  • Are secrets available?
  • Are permissions reproduced?
  • Are network controls equivalent?

Operations​

  • Is there a tested runbook?
  • Who owns each step?
  • How is split-brain prevented?
  • How is failback performed?

Economics​

  • What is the standby cost?
  • What is cross-region transfer cost?
  • What is the cost of testing?
  • Is the resilience level proportional to business value?

25. Hands-on labs​

Lab 1 — RPO/RTO classification​

Create five fictional workloads and assign:

  • business tier;
  • RPO;
  • RTO;
  • RLO;
  • DR pattern;
  • estimated cost category.

Lab 2 — Build a regional architecture​

Design:

  • primary workspace;
  • secondary workspace;
  • regional Unity Catalog metastores;
  • object storage;
  • CI/CD;
  • monitoring;
  • source replication;
  • BI failover.

Lab 3 — Streaming recovery​

Simulate:

  1. interrupted stream;
  2. checkpoint recovery;
  3. duplicate input;
  4. target reconciliation.

Lab 4 — Failover runbook​

Write a runbook that another engineer can execute without asking the architect questions.

Lab 5 — Tabletop exercise​

Scenario:

Primary AWS region becomes unavailable during the month-end finance load.

Produce:

  • incident timeline;
  • decision tree;
  • failover actions;
  • validation queries;
  • stakeholder communication;
  • RPO/RTO measurement;
  • failback plan.

26. Enterprise architect exercise​

Scenario​

A global enterprise has:

  • SAP S/4HANA;
  • SAP BW/4HANA;
  • Databricks;
  • Unity Catalog;
  • SAC;
  • enterprise BI;
  • streaming operational data;
  • regulatory reporting.

Finance requires:

  • RPO ≤ 15 minutes;
  • RTO ≤ 60 minutes;
  • no duplicate regulatory records;
  • auditable recovery;
  • quarterly DR testing.

Design the target architecture.

Your answer must explain:

  1. region selection;
  2. workspace topology;
  3. Unity Catalog topology;
  4. managed vs DIY DR;
  5. source replication;
  6. streaming recovery;
  7. identity and security;
  8. BI reconnection;
  9. failover decision authority;
  10. failback;
  11. testing;
  12. cost controls.

Architect-level requirement: explain why you rejected at least two alternative architectures.

27. Troubleshooting matrix​

SymptomLikely areaFirst investigation
Secondary is staleReplicationMeasure replication lag
Jobs fail after failoverConfigurationCompare regional parameters
BI cannot connectNetwork/endpointValidate workspace routing
Duplicate recordsStreamingCheck checkpoint/restart point
Permissions missingGovernanceCompare UC grants/identities
Data exists but query failsCatalog/storageValidate metastore and locations
Recovery meets RTO but loses too much dataRPO designIncrease replication frequency
Failover works but failback failsRunbookTest reverse synchronization

28. Common architect mistakes​

Mistake 1 — Treating HA as DR​

AZ resilience does not protect against a full-region failure.

Mistake 2 — Replicating only the data​

The workspace, identity, network and downstream ecosystem matter.

Mistake 3 — Assuming DR prevents corruption​

Replication can reproduce bad data.

Mistake 4 — Building active-active unnecessarily​

More architecture is not automatically more resilience.

Mistake 5 — No failback plan​

A one-way failover is incomplete recovery engineering.

Mistake 6 — No testing​

Untested RTO is a hypothesis.

Mistake 7 — Manual reconstruction​

If production depends on undocumented clicks, recovery depends on tribal knowledge.

Mistake 8 — Ignoring regional feature availability​

Databricks services and capabilities can vary by region. Validate supported services before selecting a DR region.

29. Production readiness checklist​

Before declaring the platform DR-ready:

  • Business services classified.
  • RPO defined.
  • RTO defined.
  • RLO defined.
  • Failure domains documented.
  • Native HA understood.
  • Primary and secondary regions selected.
  • Regional Unity Catalog architecture documented.
  • Data replication validated.
  • Workspace assets recoverable.
  • IaC and CI/CD tested.
  • Secrets and identities recoverable.
  • Streaming checkpoints recoverable.
  • BI and downstream dependencies tested.
  • Split-brain controls implemented.
  • Failover runbook approved.
  • Failback runbook approved.
  • DR test schedule established.
  • Actual RPO/RTO measured.
  • Recovery evidence retained.
  • Cost reviewed.
  • Business owner sign-off obtained.

30. Interview questions​

Foundation​

  1. What is the difference between HA and DR?
  2. Define RPO and RTO.
  3. Why is backup different from DR?
  4. What is active-passive?

Advanced​

  1. Why can DR fail to protect against data corruption?
  2. How do you recover a streaming pipeline?
  3. How does Unity Catalog affect multi-region architecture?
  4. Why should the same table not simply be registered independently in multiple metastores?
  5. When would active-active be justified?
  6. How do you prevent split-brain?

Architect​

  1. Design a multi-region Databricks platform for a 15-minute RPO.
  2. How would you prove a 60-minute RTO?
  3. What would you automate with IaC?
  4. What belongs in the runbook versus the deployment pipeline?
  5. How would you calculate the cost of resilience?
  6. How would you test a regional outage without waiting for a real outage?

31. Final mental model​

The progression is:

HIGH AVAILABILITY
      ↓
FAILURE-DOMAIN DESIGN
      ↓
RPO / RTO / RLO
      ↓
DATA + METADATA RECOVERY
      ↓
MULTI-REGION ARCHITECTURE
      ↓
FAILOVER / FAILBACK
      ↓
TESTING + CHAOS
      ↓
BUSINESS CONTINUITY
      ↓
ENTERPRISE RESILIENCE

The enterprise architect's final question is not:

"Do we have a DR workspace?"

It is:

"Can the business continue operating, with known data loss and recovery time, when the primary platform disappears — and can we prove that claim through repeatable testing?"

That is the standard for production resilience.

Official references​