22. Enterprise Architect Capstone
Mission: design, justify, implement, operate, recover, optimize, and evolve a production-grade Databricks platform for a global enterprise.
This is the final phase of the Databricks Masterclass.
You are no longer learning isolated features. You are being evaluated as an enterprise data and AI architect.
The capstone integrates:
- Databricks foundations and platform architecture
- Spark and PySpark
- Databricks SQL
- Delta Lake
- Unity Catalog
- data engineering, Auto Loader, streaming and Lakeflow
- CI/CD and Declarative Automation Bundles
- security, observability and SRE
- performance engineering and FinOps
- HA/DR
- MLflow, MLOps and model serving
- Generative AI, RAG, agents and Unity Gateway
22.1 What makes this an architect-level exercise
A developer asks:
How do I implement this pipeline?
A senior engineer asks:
How do I make this pipeline reliable and fast?
An architect asks:
Why should this pipeline exist, where should it run, what contract should it expose, who owns it, what security boundary protects it, what will it cost, how will it recover, and how will the design change at 10x scale?
The architect owns decisions and trade-offs, not merely diagrams.
A defensible architecture explains:
- business outcome,
- requirements,
- constraints,
- architecture,
- alternatives,
- rejected alternatives,
- security,
- operations,
- cost,
- resilience,
- migration,
- risks,
- exit conditions.
22.2 Capstone scenario
You are the enterprise data and AI architect for Global Industrial Group, a multinational manufacturer operating across North America, Europe, and Asia-Pacific.
The organization has:
- approximately 30,000 employees,
- 12 major business domains,
- multiple SAP S/4HANA landscapes,
- legacy SAP ECC,
- procurement platforms,
- manufacturing systems,
- CRM platforms,
- IoT/event streams,
- SaaS applications,
- partner data.
The company wants one governed data and AI platform.
Business objectives
The platform must support:
- finance reporting,
- procurement analytics,
- supply-chain analytics,
- manufacturing analytics,
- customer analytics,
- operational dashboards,
- machine learning,
- generative AI,
- governed data sharing,
- regulatory reporting,
- self-service analytics.
Non-functional requirements
| Requirement | Target |
|---|---|
| Critical finance freshness | less than 2 hours |
| Operational events | near real time |
| Critical production availability | 99.9% service target |
| Finance reporting | before 06:00 local time |
| Critical RPO | 15 minutes |
| Critical RTO | 2 hours |
| Sensitive data | least privilege |
| Auditability | critical access traceable |
| Deployment | automated from source control |
| Cost | allocated by domain and workload |
| DR | tested, not merely documented |
| AI | governed and attributable |
| Data products | explicit owners and contracts |
Treat these as architecture inputs, not implementation details.
22.3 Step 1 — Start with business capabilities
Do not start with Databricks products.
Start with business capabilities.
GLOBAL INDUSTRIAL GROUP
|
+-- Finance
| +-- General Ledger
| +-- Accounts Payable
| +-- Accounts Receivable
| +-- Management Reporting
|
+-- Procurement
| +-- Purchase Orders
| +-- Supplier
| +-- Spend
| +-- Contracts
|
+-- Supply Chain
| +-- Inventory
| +-- Demand
| +-- Logistics
| +-- Fulfillment
|
+-- Manufacturing
| +-- Production
| +-- Quality
| +-- Maintenance
| +-- IoT
|
+-- Customer
| +-- Orders
| +-- Service
| +-- Customer 360
|
+-- Data & AI Platform
+-- Data products
+-- ML
+-- GenAI
+-- Governance
+-- Operations
For every capability identify:
- owner,
- consumers,
- sources,
- criticality,
- latency,
- classification,
- regulatory constraints,
- SLO,
- recovery tier,
- growth expectation.
If these are unknown, the architecture is premature.
22.4 Step 2 — Architecture principles
Write principles before selecting components.
- Business ownership — every critical data product has an accountable owner.
- Governance by default — governed assets are the normal path.
- Least privilege — access follows business need.
- Automation first — production changes are version controlled.
- Observable by design — critical services expose operational signals.
- Recovery is a feature — critical services have RPO, RTO and tested recovery.
- Cost is measurable — workloads have attributable economics.
- Data products have contracts — consumers depend on published semantics.
- Prefer managed capabilities — avoid unnecessary operational burden.
- Reversibility — major decisions document their exit conditions.
22.5 Step 3 — Account and identity architecture
The Databricks account is the enterprise management boundary. Current Databricks guidance places identity, workspace management, Unity Catalog metastore management, and usage management at the account level.
Mental model:
ENTERPRISE
|
v
DATABRICKS ACCOUNT
|
+-- Identity
+-- Groups
+-- Service principals
+-- Workspaces
+-- Regional governance
+-- Billing / usage
+-- Enterprise policies
Separate identity classes:
- human users,
- service principals,
- CI/CD identities,
- integration identities.
Prefer groups over individual grants and service principals for automated workloads.
Do not give every engineer account-wide administrative privileges.
22.6 Step 4 — Workspace topology
There is no universal "one workspace per team" rule.
Evaluate:
| Driver | More isolation | More consolidation |
|---|---|---|
| Regulatory isolation | strong | weak |
| Network isolation | strong | weak |
| Independent lifecycle | strong | weak |
| Small team count | weak | strong |
| Shared governance | weak | strong |
| Collaboration | weak | strong |
| Administrative simplicity | weak | strong |
A mature enterprise often uses a hybrid topology:
ACCOUNT
|
+-- Region A
| +-- Metastore
| +-- Shared engineering
| +-- Critical production
| +-- Special workloads
|
+-- Region B
| +-- Metastore
| +-- Development
| +-- Production
|
+-- Region C
+-- Metastore
+-- Regional workloads
Choose boundaries because of real security, network, regulatory, operational, or blast-radius requirements.
22.7 Step 5 — Regional architecture
Unity Catalog metastores are regional. A multi-region architecture therefore requires deliberate regional governance and cross-region data movement or sharing decisions.
For every cross-region dataset ask:
- Must it move?
- Can consumers use a regional copy?
- Can governed sharing satisfy the requirement?
- Does regulation prohibit movement?
- What is the latency requirement?
- What is the transfer cost?
- What is the recovery implication?
Never treat cross-region access as free.
22.8 Step 6 — Unity Catalog architecture
Unity Catalog is the unified governance layer for Databricks data and AI assets.
Use the three-level namespace:
catalog.schema.object
A possible enterprise design:
enterprise_shared
+-- reference
+-- master_data
+-- shared_products
finance
+-- bronze
+-- silver
+-- gold
procurement
+-- bronze
+-- silver
+-- gold
supply_chain
+-- bronze
+-- silver
+-- gold
This is illustrative, not mandatory.
Create a meaningful catalog boundary when ownership, security, storage, lifecycle, regulatory requirements, or operating responsibility changes.
Do not create catalogs merely because a diagram looks cleaner.
22.9 Step 7 — Data products
A data product is a governed business capability.
DATA PRODUCT
|
+-- Owner
+-- Business definition
+-- Grain
+-- Schema
+-- Keys
+-- Quality rules
+-- Freshness SLO
+-- Classification
+-- Lineage
+-- Retention
+-- Consumers
+-- Support model
+-- Change policy
Example:
Supplier Spend
- grain: supplier x company x fiscal period
- owner: Procurement Data Product Owner
- freshness: less than 2 hours
- completeness: at least 99.5%
- consumers: Finance, Procurement, SAC, ML
- classification: Confidential
The physical table is an implementation detail.
22.10 Step 8 — Data contracts
A contract protects consumers from uncontrolled producer changes.
At minimum define:
- schema,
- semantics,
- grain,
- keys,
- nullability,
- quality,
- freshness,
- compatibility rules,
- classification,
- retention,
- ownership,
- support.
PRODUCER
|
v
DATA CONTRACT
|
+-- compatible change --> consumers continue
|
+-- breaking change --> version / migration
A consumer should depend on the contract, not internal notebook logic.
22.11 Step 9 — Source and ingestion architecture
Classify sources before choosing ingestion.
| Source | Candidate pattern |
|---|---|
| SAP S/4HANA | extraction, CDC, APIs |
| SAP ECC | extraction, CDC, legacy integration |
| Database | CDC / incremental |
| File | object storage + incremental ingestion |
| API | managed/API ingestion |
| IoT | streaming |
| SaaS | connector/API |
| Partner | secure file/API exchange |
For each source document:
- initial load,
- incremental strategy,
- delete handling,
- late data,
- replay,
- backfill,
- schema evolution,
- error handling,
- recovery.
22.12 Step 10 — SAP integration
SAP experience is an advantage, but avoid false 1:1 mappings.
| SAP capability | Lakehouse concern |
|---|---|
| ODP | extraction/change-data contract |
| SLT | CDC/replication |
| BW ADSO | governed persistent structure |
| Process chain | orchestration |
| CompositeProvider | semantic/serving model |
| SAC | analytics consumption |
| Datasphere | integration/semantic/governed capability |
Preserve:
- business semantics,
- reconciliation,
- history,
- keys,
- currencies,
- units,
- fiscal logic,
- organizational structures.
SAP S/4HANA / ECC
|
+-- ODP / extraction
+-- SLT / CDC
+-- APIs
|
v
Enterprise ingestion
|
v
Bronze -> Silver -> Data Products
|
+-------+-------+
| | |
SAC ML GenAI
The mapping should be capability-to-capability, not object-to-object.
22.13 Step 11 — Medallion architecture
Use Bronze/Silver/Gold when each layer provides measurable value.
Bronze
Preserve source fidelity and replayability.
Silver
Standardize, validate, deduplicate, conform and enrich.
Gold
Publish business-facing products optimized for consumption.
Do not force three layers when:
- the source is already curated,
- duplication adds no value,
- latency makes extra hops harmful,
- the data product has a simpler lifecycle.
Architecture maturity includes knowing when not to use a pattern.
22.14 Step 12 — Processing architecture
Choose batch, streaming, or event-driven execution from business latency.
| Requirement | Typical pattern |
|---|---|
| Daily reporting | batch |
| Hourly analytics | scheduled / micro-batch |
| Near-real-time dashboard | streaming |
| Event-triggered action | event-driven |
| Historical backfill | batch |
| Continuous events | streaming |
Define:
- freshness SLO,
- completeness,
- late-arrival policy,
- replay,
- state,
- checkpoints,
- recovery,
- cost.
22.15 Step 13 — Orchestration
Orchestration should express business dependencies.
SAP ingestion
|
v
Bronze validation
|
+--+--+
| |
Supplier PO
| |
+--+--+
|
v
Spend product
|
+--+----+
| | |
SAC ML GenAI
The orchestration layer should own:
- dependencies,
- retries,
- scheduling,
- parameters,
- alerting,
- recovery,
- operational metadata.
Do not hide enterprise dependency logic inside notebooks.
22.16 Step 14 — Compute architecture
Classify workloads:
| Workload | Primary concern |
|---|---|
| ETL | throughput |
| Streaming | sustained latency |
| Ad hoc SQL | concurrency |
| BI | predictable serving |
| ML training | distributed compute |
| Inference | latency/cost |
| GenAI | model/token economics |
| Data quality | predictable execution |
Ask:
- interactive or automated?
- batch or continuous?
- latency sensitive?
- CPU or memory heavy?
- shuffle heavy?
- concurrency heavy?
- cost sensitive?
Select compute from evidence, not habit.
22.17 Step 15 — Serving architecture
Consumers should not need ingestion internals.
Governed Data Products
|
+-- SQL analytics
+-- BI / SAC
+-- APIs
+-- Sharing
+-- ML
+-- AI
Optimize the serving layer independently from transformation.
A pipeline can be fast while its consumer experience is slow.
22.18 Step 16 — Performance architecture
Use a feedback loop:
Measure
|
v
Locate bottleneck
|
v
Form hypothesis
|
v
Change one variable
|
v
Benchmark
|
v
Validate
|
v
Standardize
Investigate:
- file layout,
- data skipping,
- clustering,
- joins,
- shuffle,
- skew,
- query plans,
- compute,
- concurrency,
- materialization.
Do not optimize without a measured bottleneck.
22.19 Step 17 — FinOps architecture
Cost should be attributable.
Account
|
Region
|
Workspace
|
Domain
|
Workload
|
Pipeline / Query / Model / AI request
Track:
- compute,
- storage,
- network,
- data transfer,
- serving,
- ML,
- GenAI,
- operational tooling.
Prefer unit economics:
- cost per successful pipeline,
- cost per TB processed,
- cost per data product,
- cost per dashboard refresh,
- cost per million events,
- cost per inference,
- cost per AI request.
A platform that cannot explain its cost cannot be governed economically.
22.20 Step 18 — Security architecture
Security is layered.
Identity
|
Account controls
|
Workspace controls
|
Network controls
|
Unity Catalog authorization
|
Fine-grained data controls
|
Cloud/storage permissions
|
Audit
Prefer:
- groups,
- service principals,
- least privilege,
- separation of duties,
- automated entitlement lifecycle.
Current Databricks administration guidance distinguishes account administrators, workspace administrators, and optional metastore administrators by scope.
22.21 Step 19 — Zero trust
Assume no implicit trust between:
- users,
- workloads,
- domains,
- regions,
- services,
- cloud resources.
For each access path establish:
Identity
+
Authorization
+
Network policy
+
Data policy
+
Audit
A network control does not replace authorization. Authorization does not replace network control.
22.22 Step 20 — CI/CD and Declarative Automation Bundles
Production changes should flow through source control.
Developer
|
Git
|
Pull Request
|
Validation + tests
|
DEV
|
TEST
|
Approval
|
PROD
Declarative Automation Bundles are the current name for Databricks Asset Bundles and support source-controlled definitions of project files and Databricks resources.
Current Databricks guidance says new CLI versions use the direct deployment engine by default. The old Terraform deployment engine was removed in CLI 1.20.0, so do not design a new platform around the removed engine.
Environment-specific values should be configuration, not duplicated source code.
22.23 Step 21 — Observability and SRE
Measure:
Platform
- workspace health,
- compute,
- network,
- capacity.
Data pipelines
- success,
- duration,
- retries,
- backlog,
- freshness.
Data
- completeness,
- validity,
- volume anomalies,
- schema changes.
Serving
- latency,
- concurrency,
- errors.
AI
- requests,
- latency,
- token usage,
- provider usage,
- failures,
- safety signals.
Cost
- spend,
- budgets,
- anomalies,
- unit economics.
Define SLI, SLO, alert threshold, owner, and runbook.
22.24 Step 22 — HA/DR
Recovery must cover more than data.
Identity
+
Network
+
Workspace configuration
+
Unity Catalog
+
Storage
+
Pipelines
+
Schedules
+
Secrets/configuration
+
Applications
+
Runbooks
For every critical service define:
- RPO,
- RTO,
- recovery owner,
- detection,
- failover,
- validation,
- failback.
Do not call a system resilient until recovery has been tested.
22.25 Step 23 — Failure-domain analysis
For each critical service identify:
- failure domain,
- blast radius,
- detection,
- containment,
- recovery,
- validation.
| Failure | Typical blast radius | First response |
|---|---|---|
| Task failure | one workload | retry / investigate |
| Workspace failure | multiple workloads | recovery |
| Region failure | regional platform | DR |
| Identity outage | enterprise | identity recovery |
| Source outage | dependent products | graceful degradation |
| Contract break | consumers | quarantine/version |
| Model outage | ML/AI workload | fallback |
22.26 Step 24 — ML architecture
ML belongs inside the same governance and operating model.
Data Products
|
Training Data
|
MLflow
|
Experiments / Runs / Models / Evaluation
|
Governed Registry
|
Serving
|
Predictions
|
Monitoring
|
Retraining
Control:
- reproducibility,
- lineage,
- versioning,
- promotion,
- rollback,
- drift,
- monitoring,
- cost.
22.27 Step 25 — MLOps
A production ML lifecycle is:
Data
|
Training
|
Evaluation
|
Approval
|
Deployment
|
Monitoring
|
Retraining
A successful training job is not a production approval.
Evaluate:
- model quality,
- data quality,
- drift,
- latency,
- cost,
- business impact,
- security.
22.28 Step 26 — Generative AI and RAG
Treat GenAI as an enterprise workload.
Enterprise Data
|
Governed ingestion
|
AI Search / retrieval
|
RAG / Agent
|
Unity Gateway
|
Models / providers
|
Application
Production RAG needs:
- source governance,
- chunking,
- metadata,
- embeddings,
- retrieval,
- filtering,
- reranking,
- generation,
- evaluation,
- monitoring.
Evaluate retrieval quality separately from generation quality.
22.29 Step 27 — Agents and tool security
Separate read and write tools.
Read tools
- SQL query,
- search,
- metadata lookup.
Write tools
- update master data,
- trigger workflow,
- create transaction.
Write actions require stronger authorization.
User
|
Identity
|
Agent
|
Tool authorization
|
Data authorization
|
Action
|
Audit
Never assume the model can enforce enterprise permissions.
22.30 Step 28 — Unity Gateway
Unity Gateway is Databricks' enterprise AI governance control plane for model and MCP access, routing, rate limits, cost controls, policies, and usage monitoring.
Enterprise pattern:
AI applications
|
v
Unity Gateway
|
+----+-----------+
| | |
Models MCP AI services
| |
Providers / tools
Architectural value:
- centralized access control,
- provider portability,
- usage visibility,
- cost controls,
- policy enforcement,
- auditability.
22.31 Step 29 — Enterprise reference architecture
ENTERPRISE
|
Identity / Security / Architecture
|
DATABRICKS ACCOUNT
|
+---------------------+---------------------+
| | |
REGION A REGION B REGION C
| | |
Metastore Metastore Metastore
| | |
Workspaces Workspaces Workspaces
| | |
+------------- Governed Data Products -----+
|
+--------------+--------------+
| | |
Analytics ML GenAI
| | |
SAC MLflow Unity Gateway
| | |
+--------------+--------------+
|
BUSINESS OUTCOMES
This is a conceptual architecture. Adapt it to geography, regulation, workload criticality, network, cost, and team maturity.
22.32 Step 30 — Architecture decision records
Every major decision should have an ADR.
Required ADRs:
- workspace topology,
- regional topology,
- Unity Catalog strategy,
- managed vs external storage,
- batch vs streaming,
- ingestion strategy,
- data-product ownership,
- CI/CD strategy,
- DR strategy,
- AI governance strategy.
Template:
Context
Decision
Alternatives
Decision criteria
Consequences
Risks
Exit condition
The exit condition is essential.
It answers:
Under what future condition would we reconsider this decision?
22.33 Step 31 — Risk register
Create a living risk register.
| Risk | Probability | Impact | Mitigation | Owner |
|---|---|---|---|---|
| Workspace sprawl | Medium | High | architecture standards | Platform |
| Data leakage | Low | Critical | zero trust + UC | Security |
| Cost explosion | Medium | High | FinOps | Platform |
| Poor quality | High | High | contracts + gates | Domain |
| DR failure | Medium | Critical | regular tests | SRE |
| AI misuse | Medium | High | gateway + policies | AI Platform |
| Vendor lock-in | Medium | Medium | open interfaces | Architecture |
| Source outage | Medium | High | replay + resilience | Integration |
Risks must have owners.
22.34 Step 32 — Migration strategy
Do not migrate everything at once.
Discover
|
Assess
|
Classify
|
Prioritize
|
Pilot
|
Industrialize
|
Migrate by domain
|
Decommission
|
Optimize
Classify workloads by:
- business value,
- complexity,
- dependencies,
- data volume,
- latency,
- regulatory risk,
- current cost,
- migration effort.
For SAP-heavy environments, success means business reconciliation and controlled cutover, not merely moving tables.
22.35 Step 33 — Parallel-run and cutover
For critical finance workloads:
Source
|
+-------+-------+
| |
Legacy Databricks
| |
Result A Result B
| |
+-------+-------+
|
Reconcile
|
+-------+-------+
| |
Match Investigate
Before cutover:
- final load,
- reconciliation,
- security validation,
- consumer validation,
- monitoring,
- rollback point,
- business approval.
22.36 Step 34 — Operating model
Technology does not operate itself.
Platform team
Owns:
- workspaces,
- shared infrastructure,
- governance patterns,
- CI/CD,
- observability,
- FinOps.
Domain teams
Own:
- data products,
- business semantics,
- quality,
- consumer support.
SRE / operations
Own:
- reliability,
- incidents,
- runbooks,
- DR testing.
Architecture
Owns:
- principles,
- standards,
- exceptions,
- strategic evolution.
22.37 Step 35 — RACI
Example:
| Capability | Platform | Domain | Security | Architecture |
|---|---|---|---|---|
| Workspace | A/R | C | C | C |
| Catalog strategy | R | A/R | C | A |
| Data product | C | A/R | C | C |
| Security policy | R | C | A/R | C |
| Pipeline | C | A/R | C | C |
| DR | A/R | C | C | A |
| FinOps | A/R | R | C | C |
| Standards | C | C | C | A/R |
Adapt the RACI to the actual organization.
22.38 Step 36 — Production runbooks
Every critical service should have a runbook containing:
- symptom,
- impact,
- detection,
- immediate containment,
- diagnostic steps,
- recovery,
- validation,
- escalation,
- post-incident action.
Example: finance pipeline delayed.
1. Check orchestration.
2. Check source freshness.
3. Check compute.
4. Check data quality.
5. Check dependencies.
6. Determine whether retry is safe.
7. Replay from known state.
8. Reconcile output.
9. Notify consumers.
10. Record incident.
A runbook is part of production architecture.
22.39 Step 37 — Architecture maturity
Level 1 — Project
One team builds pipelines.
Level 2 — Standardized
Reusable patterns and governance appear.
Level 3 — Platform
Central identity, CI/CD, observability and FinOps.
Level 4 — Enterprise
Data products, domain ownership, SLOs, DR and architecture governance.
Level 5 — Adaptive
Measured cost, reliability, performance and business value continuously drive architecture evolution.
Target Level 4+.
22.40 Step 38 — Architecture scorecard
Score the platform from 1 to 5.
| Dimension | Score |
|---|---|
| Business alignment | /5 |
| Data architecture | /5 |
| Governance | /5 |
| Security | /5 |
| Reliability | /5 |
| Performance | /5 |
| FinOps | /5 |
| CI/CD | /5 |
| Observability | /5 |
| ML/MLOps | /5 |
| GenAI governance | /5 |
| Operating model | /5 |
| Migration readiness | /5 |
The goal is not a perfect score.
The goal is explicit trade-offs.
22.41 Architect defense framework
When challenged, answer in this order:
1. Requirement
What problem are we solving?
2. Constraint
What limits the solution?
3. Options
What alternatives exist?
4. Decision
What did we choose?
5. Rationale
Why?
6. Consequence
What do we give up?
7. Evidence
What measurements support the decision?
8. Exit condition
When would we change it?
This is the core of architecture reasoning.
22.42 Mandatory architect exercises
Exercise A — Region failure
Primary production region becomes unavailable during finance reporting.
Answer:
- What detects failure?
- Who declares the incident?
- What is RPO?
- What is RTO?
- How is the recovery environment activated?
- How are identities recovered?
- How are data products recovered?
- How are schedules restored?
- How is data reconciled?
- How do consumers reconnect?
- What happens if RTO is exceeded?
Exercise B — Security breach
An analyst attempts to access confidential supplier information outside their domain.
Explain:
- identity verification,
- workspace access,
- Unity Catalog authorization,
- fine-grained controls,
- storage boundary,
- audit,
- incident response.
Exercise C — Cost spike
Monthly platform spend increases 60%.
Investigate:
- workspace changes,
- workload changes,
- compute changes,
- query changes,
- data growth,
- retries,
- streaming state,
- cross-region transfer,
- AI usage,
- storage retention.
Exercise D — Data discrepancy
Supplier spend is 8% higher than the finance source.
Investigate:
Source reconciliation
|
Record counts
|
Keys / duplicates
|
Currency
|
Fiscal calendar
|
Filters
|
Late data
|
CDC deletes
|
Transformation
|
Serving aggregation
Exercise E — Performance regression
A dashboard moves from 5 seconds to 40 seconds.
Use:
Confirm regression
|
Check workload change
|
Inspect query history
|
Inspect plan
|
Check data growth
|
Check layout
|
Check joins / shuffle
|
Check compute / concurrency
|
Change one variable
|
Benchmark
Exercise F — AI safety
A procurement agent retrieves confidential supplier information and attempts a purchase action.
Separate:
- user authorization,
- retrieval authorization,
- tool authorization,
- write-action authorization,
- model behavior,
- audit,
- human approval.
22.43 Required capstone deliverables
Produce:
A. Executive architecture
One page covering objective, major decisions, risks, and outcomes.
B. Context diagram
Business domains, sources, consumers and external systems.
C. Deployment diagram
Account, regions, metastores, workspaces and network boundaries.
D. Data architecture
Ingestion, processing, data products and serving.
E. Security architecture
Identity, authorization, network, sensitive data and audit.
F. Operations architecture
Observability, SLOs, alerts, runbooks and incident management.
G. DR architecture
Failure domains, RPO/RTO, recovery and validation.
H. ML architecture
Training, MLflow, governance, serving and monitoring.
I. GenAI architecture
RAG, agents, tools, Unity Gateway and governance.
J. FinOps model
Allocation, unit economics, budgets and optimization.
K. Migration roadmap
Discovery through decommissioning.
L. Ten ADRs
Use the ADR template above.
22.44 The 10x test
Assume:
- data volume increases 10x,
- users increase 5x,
- domains double,
- AI usage increases 20x,
- regional footprint doubles.
Answer:
- What breaks first?
- What cost grows fastest?
- Which governance boundary becomes stressed?
- Which workspace strategy changes?
- Which data products need redesign?
- Which workloads should become streaming?
- Which AI controls need strengthening?
- Which operating teams need to scale?
Then repeat for 100x.
If the architecture itself must change, identify the phase transition rather than simply adding compute.
22.45 Production readiness checklist
Business
- Business capabilities mapped
- Owners identified
- Consumers identified
- Outcomes measurable
Platform
- Account strategy defined
- Workspace topology justified
- Regional strategy defined
- Network boundaries documented
Governance
- Unity Catalog strategy defined
- Catalog boundaries justified
- Ownership defined
- Lineage available
- Audit strategy defined
Data
- Data products defined
- Contracts defined
- Quality gates defined
- Retention defined
- Schema policy defined
Engineering
- Ingestion justified
- Batch/streaming justified
- Orchestration defined
- CI/CD automated
- Configuration versioned
Security
- Least privilege
- Workload identities
- Sensitive-data controls
- Network controls
- Audit
- Separation of duties
Operations
- SLOs
- SLIs
- Alerts
- Runbooks
- Incident process
- On-call ownership
FinOps
- Cost allocation
- Budgets
- Anomaly detection
- Unit economics
- Optimization process
Resilience
- Failure domains
- RPO
- RTO
- Recovery runbooks
- DR tests
- Failback
ML and AI
- MLflow lifecycle
- Model governance
- Serving
- Monitoring
- RAG controls
- Agent tool authorization
- Unity Gateway
- AI cost controls
Architecture governance
- ADRs
- Risk register
- Standards
- Exception process
- Review cadence
22.46 Final architect examination
You have completed the masterclass only when you can defend:
Platform
- Why this account structure?
- Why these workspaces?
- Why these regional boundaries?
Data
- Why these data products?
- Why these contracts?
- Why this ingestion architecture?
Governance
- Why these catalogs?
- Why these privileges?
- How is sensitive data protected?
Engineering
- Why batch here?
- Why streaming there?
- Why this orchestration model?
Performance
- What is the bottleneck?
- What evidence proves it?
- What is the optimization trade-off?
Cost
- What does the platform cost?
- Who owns the cost?
- What is the unit economics?
Reliability
- What happens when a dependency fails?
- What happens when a region fails?
- How do you prove recovery?
ML
- How do you reproduce a model?
- How do you promote it?
- How do you roll it back?
GenAI
- How do you control model access?
- How do you secure tools?
- How do you evaluate grounding?
- How do you control AI spend?
Enterprise
- How does the architecture evolve at 10x?
- What changes at 100x?
- Which decision would you reverse?
- Which assumption is most dangerous?
22.47 Final mental model
BUSINESS OUTCOMES
|
v
DOMAINS + OWNERSHIP
|
v
DATA PRODUCTS + CONTRACTS
|
v
GOVERNANCE + SECURITY
|
v
ACCOUNT + REGION + WORKSPACE
|
v
INGESTION + PROCESSING
|
v
SERVING + CONSUMPTION
|
v
SLO + OBSERVABILITY
|
v
FINOPS
|
v
HA / DR
|
v
ML + MLOPS
|
v
GENAI + AI GOVERNANCE
|
v
OPERATING MODEL
|
v
CONTINUOUS ARCHITECTURE EVOLUTION
If you start with:
Which Databricks feature should I use?
you are starting too low.
Start with:
What business capability are we building, who owns it, what contract does it expose, what reliability does it require, who may access it, what will it cost, and how will it evolve?
Then choose the Databricks architecture.
Official references
- https://docs.databricks.com/aws/en/getting-started/high-level-architecture/
- https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/account-setup
- https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/unity-catalog
- https://docs.databricks.com/aws/en/data-governance/unity-catalog
- https://docs.databricks.com/aws/en/dev-tools/bundles
- https://docs.databricks.com/aws/en/dev-tools/bundles/direct
- https://docs.databricks.com/aws/en/ai-gateway/ai-governance
Current-platform note: Databricks evolves rapidly. Validate production implementation details against the official documentation for your cloud, region, account configuration, and feature availability before deployment.