18. Enterprise Lakehouse Architecture
Learning objective: design an enterprise Databricks lakehouse as a coherent platform—covering account, workspace, Unity Catalog, data domains, ingestion, processing, serving, security, networking, observability, FinOps, resilience, and operating model—and defend the resulting architecture with explicit trade-offs.
Phase 17 taught you how to make the platform recoverable.
Phase 18 asks a broader question:
How should an enterprise design the lakehouse so that hundreds of users, many domains, multiple environments, and critical workloads can scale without turning the platform into a collection of isolated projects?
This is the first phase where the primary unit of thinking is no longer a pipeline or workspace. It is the enterprise platform.
18.1 What an enterprise lakehouse actually is
A lakehouse is not simply:
- object storage + Delta tables,
- a Databricks workspace,
- a medallion diagram,
- or a collection of ETL jobs.
An enterprise lakehouse is a socio-technical system consisting of:
- Business domains — the producers and consumers of data.
- Data products — governed datasets, tables, views, files, and analytical interfaces.
- Platform services — compute, orchestration, ingestion, serving, governance, observability, and security.
- Control plane — identity, policies, metadata, deployment, configuration, and operating standards.
- Data plane — cloud storage, compute execution, network paths, and workload data.
- Operating model — ownership, SLOs, support, FinOps, change management, and architecture governance.
The architecture must make these concerns explicit.
18.2 Start with architecture boundaries
Before choosing a Databricks feature, establish the boundaries.
Business boundary
Who owns the data?
Examples:
- Finance
- Procurement
- Supply Chain
- Customer
- Marketing
- Operations
- Data & AI Platform
A domain should have accountable owners for:
- data definition,
- quality,
- access decisions,
- lifecycle,
- service expectations,
- and consumer support.
Environment boundary
Development, test, staging, and production exist for different reasons.
Do not assume that every organization needs one workspace per environment. Environment isolation can be implemented through a combination of:
- workspaces,
- Unity Catalog catalogs,
- storage locations,
- identities,
- deployment controls,
- network boundaries,
- and data access policies.
The correct design depends on the required isolation level, regulatory constraints, team topology, and operational model.
Regional boundary
A region is a real architectural boundary for Unity Catalog.
Databricks documents a regional metastore model: a metastore is scoped to one cloud region, and workspaces in that region can be attached to it. Multi-region designs therefore require deliberate cross-region sharing or synchronization rather than pretending that one metastore is globally authoritative.
Security boundary
Separate:
- human identities,
- workload identities,
- administrative privileges,
- data permissions,
- network controls,
- and cloud-resource permissions.
Never use a data-layer permission as a substitute for a network boundary, or a network boundary as a substitute for data authorization.
18.3 The enterprise architecture stack
Think of the platform as layers.
| Layer | Primary responsibility | Architect question |
|---|---|---|
| Business | Domains, capabilities, ownership | Who owns the outcome? |
| Data products | Trusted datasets and interfaces | What contract do consumers depend on? |
| Governance | Unity Catalog, policies, lineage | Who may discover, read, change, or share it? |
| Processing | Batch, streaming, transformations | Where should computation occur? |
| Orchestration | Pipelines, jobs, dependencies | How is work scheduled and recovered? |
| Serving | SQL, BI, APIs, downstream sharing | How will consumers access the result? |
| Platform | Workspaces, compute, networking | How is the runtime operated safely? |
| Cloud | Storage, IAM, network, KMS | What must the cloud provider enforce? |
| Operating model | SLOs, support, FinOps, standards | Who operates the platform after go-live? |
A common architecture mistake is to optimize only the processing layer while leaving governance, ownership, and operations undefined.
18.4 Account → workspace → metastore
The Databricks account is the enterprise management boundary. Databricks describes account-level responsibilities such as identity and access, workspace management, Unity Catalog metastore management, and usage management. Workspaces are execution and collaboration environments; Unity Catalog metastores provide regional governance for data and AI assets.
A useful mental model is:
Enterprise / Databricks Account
│
├── Identity + groups + service principals
├── Billing + usage + policies
├── Workspace A
│ └── attached to regional Metastore
├── Workspace B
│ └── attached to regional Metastore
└── Workspace C
└── attached to another regional Metastore
Do not confuse these scopes:
- Account = enterprise administration.
- Workspace = execution and collaboration boundary.
- Metastore = regional Unity Catalog governance boundary.
- Catalog = organizational/data isolation boundary.
- Schema = logical grouping inside a catalog.
- Table/view/volume/function = governed data or code asset.
A single metastore can be attached to multiple workspaces in its region, and permissions on the metastore apply across those attached workspaces.
18.5 Workspace topology: choose isolation deliberately
There is no universal "one workspace per team" rule.
Evaluate workspace boundaries against:
| Driver | Favors more isolation | Favors consolidation |
|---|---|---|
| Regulatory isolation | Strong | Weak |
| Network isolation | Strong | Weak |
| Independent platform lifecycle | Strong | Weak |
| Small team count | Weak | Strong |
| Shared governance | Weak | Strong |
| Cross-team collaboration | Weak | Strong |
| Cost / administration simplicity | Weak | Strong |
Common enterprise patterns include:
Pattern A — Shared development + isolated production
Shared DEV
│
├── Sales
├── Finance
└── Engineering
Dedicated PROD workspaces
├── Sales PROD
├── Finance PROD
└── Engineering PROD
Useful when teams need collaboration but production requires stronger isolation.
Pattern B — Environment-oriented
DEV ── TEST ── STAGE ── PROD
Useful where release controls and centralized operations dominate.
Pattern C — Domain-oriented
Sales Workspace
Finance Workspace
Supply Chain Workspace
Engineering Workspace
Useful when domains have materially different operating, security, or network requirements.
Pattern D — Hybrid
Most mature enterprises use a hybrid model.
For example:
- shared engineering workspace(s),
- controlled development workspaces,
- dedicated production workspaces for critical domains,
- regional workspaces where required,
- centralized governance through Unity Catalog.
Databricks' current workspace strategy guidance explicitly presents single-region, multi-region, multi-business-unit, and environment-based patterns.
18.6 Unity Catalog: the enterprise governance backbone
Unity Catalog should be treated as an architectural control plane for governed data and AI assets—not merely as a replacement for an old metastore.
Databricks recommends one metastore per region. Catalogs then become an important isolation and organization mechanism within that regional boundary.
Three-level namespace
catalog.schema.object
Example:
finance_prod.gl.gold_profitability
finance_prod.slv.vendor_invoice
sales_prod.gold.customer_revenue
The naming convention is less important than the ownership and isolation model behind it.
Catalog design patterns
Environment-first
dev
test
prod
Simple and effective when environments are the primary isolation concern.
Domain-first
finance
sales
supply_chain
customer
Useful when domains own their data products.
Hybrid
finance_dev
finance_prod
sales_dev
sales_prod
Useful when both domain ownership and environment isolation must be obvious.
Databricks' current guidance also describes hub-and-spoke designs in which shared enterprise data lives in hub catalogs and domain-owned data lives in spoke catalogs.
Architect rule
Do not create catalogs simply because a diagram looks cleaner.
Create a catalog when there is a meaningful boundary involving:
- ownership,
- security,
- storage,
- lifecycle,
- environment,
- regulatory requirements,
- or operating responsibility.
18.7 Managed tables, external tables, and volumes
Modern lakehouse architecture should start with governed storage patterns rather than legacy filesystem habits.
For structured lakehouse data, managed tables are generally the preferred starting point. Volumes provide a governed mechanism for file-oriented access. External locations and external tables remain useful where data must remain in existing cloud storage or be shared with systems that require direct storage access.
Architectural question:
Who owns the physical storage lifecycle?
If the answer is "the lakehouse platform," managed storage is often simpler.
If the answer is "an external enterprise storage domain," external locations may be appropriate.
Avoid designing new production platforms around legacy DBFS-root storage patterns. Current Databricks guidance explicitly recommends against storing production data on DBFS and against using the root bucket for customer data.
18.8 Data domains and data products
A domain is not merely a folder.
A mature domain has:
- accountable owner,
- business glossary,
- source systems,
- data products,
- quality rules,
- access policy,
- lineage,
- SLOs,
- consumers,
- support model.
Example:
PROCUREMENT DOMAIN
│
├── Source systems
│ ├── SAP S/4HANA
│ ├── Ariba
│ └── Supplier platforms
│
├── Data products
│ ├── Purchase Order
│ ├── Supplier
│ ├── Spend
│ └── Contract
│
├── Quality
│ ├── completeness
│ ├── uniqueness
│ └── referential integrity
│
└── Consumers
├── BI
├── Finance
├── Supply Chain
└── ML
This is where your SAP background becomes valuable: think beyond tables and pipelines. A source object becomes useful only when its business semantics, ownership, history, quality, and consumer contract are understood.
18.9 Medallion architecture: useful pattern, not religion
Bronze/Silver/Gold is a powerful teaching and implementation pattern, but it is not a mandatory architecture for every workload.
Bronze
Purpose:
- preserve source fidelity,
- absorb ingestion variability,
- provide replayable raw-ish state.
Silver
Purpose:
- standardize,
- validate,
- deduplicate,
- conform,
- enrich.
Gold
Purpose:
- publish business-ready products,
- optimize consumption,
- encode business semantics.
When not to force three layers
A three-layer design may be excessive when:
- the source is already curated,
- the workload is transient,
- a low-latency serving path is more appropriate,
- the data product has a simpler lifecycle,
- or duplication creates more cost than value.
Architectural maturity means knowing why a layer exists.
18.10 Batch, streaming, and event-driven architecture
Choose the processing pattern from business latency—not technology fashion.
| Requirement | Typical pattern |
|---|---|
| Daily reporting | Batch |
| Hourly operational analytics | Micro-batch / scheduled |
| Near-real-time dashboards | Streaming |
| Event-triggered workflows | Event-driven |
| Historical backfill | Batch |
| Continuous sensor/event processing | Streaming |
Do not build a streaming pipeline simply because streaming is available.
The architect should define:
- freshness SLO,
- completeness expectation,
- late-arrival policy,
- replay strategy,
- state management,
- checkpointing,
- recovery behavior,
- cost envelope.
18.11 Ingestion architecture
Enterprise ingestion normally falls into several patterns:
Pattern 1 — File ingestion
Source
↓
Cloud object storage
↓
Auto Loader / ingestion pipeline
↓
Bronze
Pattern 2 — Database CDC
Operational DB
↓
CDC
↓
Landing / ingestion
↓
Delta
↓
Conformed data
Pattern 3 — Application/API ingestion
API
↓
Landing
↓
Validation
↓
Raw / Bronze
↓
Conformed
Pattern 4 — SAP integration
A practical enterprise pattern may look like:
SAP S/4HANA / ECC
│
├── ODP / extractors
├── SLT / CDC
├── APIs
└── files / integration services
│
▼
Enterprise ingestion
│
▼
Delta / Lakehouse
│
┌───────┴────────┐
▼ ▼
Data products Analytics
│ │
▼ ▼
BI / SAC ML / AI
This is not a 1:1 replacement map such as "BW ADSO = Delta table." Instead, map capabilities:
| SAP concept | Lakehouse architectural concern |
|---|---|
| BW data model | Data product / analytical model |
| ADSO | Persistent governed data structure |
| Process chain | Orchestration / dependency graph |
| ODP | Extraction / change-data contract |
| SLT | CDC / replication pattern |
| CompositeProvider | Semantic serving / federated analytical model |
| SAC | BI / analytical consumption |
| Datasphere | Data integration / semantic / governed data layer, depending on scope |
The goal is to transfer architectural thinking, not terminology.
18.12 Data contracts
A data contract defines what a producer promises to consumers.
At minimum:
- schema,
- semantic definitions,
- keys,
- nullability,
- freshness,
- quality thresholds,
- change policy,
- ownership,
- classification,
- retention,
- support contact.
Example:
Supplier Spend Product
│
├── grain: supplier × fiscal period
├── primary business key: supplier_id + fiscal_period
├── freshness: < 2 hours
├── completeness: ≥ 99.5%
├── schema policy: additive changes allowed
├── owner: Procurement Data Product Owner
└── consumer SLA: 08:00 business reporting
The contract is more important than the physical table.
18.13 Data quality as an architecture capability
Quality should be measurable and observable.
Classify rules into:
- structural,
- completeness,
- uniqueness,
- validity,
- consistency,
- referential integrity,
- timeliness,
- reconciliation.
A production quality design should answer:
- Where is the rule evaluated?
- What happens when it fails?
- Is bad data quarantined?
- Can the record be replayed?
- Who is notified?
- Does the failure block publication?
- How is quality reported over time?
A quality rule without an operational response is only documentation.
18.14 Serving architecture
Consumers should not need to understand ingestion internals.
Typical serving paths include:
Curated data products
│
├── SQL analytics
├── BI / semantic layer
├── APIs / applications
├── data sharing
└── ML / AI features
Use the serving interface that matches the consumer.
For example:
- analysts may need SQL,
- dashboards may need a SQL warehouse,
- applications may need an API,
- another organization may need governed sharing,
- ML workloads may need feature/model interfaces.
18.15 Security architecture
Treat security as multiple layers.
Identity
↓
Workspace access
↓
Network controls
↓
Unity Catalog privileges
↓
Row / column controls where required
↓
Storage permissions
↓
Audit + monitoring
A strong architecture avoids relying on one control.
Privilege model
Prefer:
- groups over individual grants,
- service principals for workloads,
- least privilege,
- separation of duties,
- controlled administrative roles,
- automated entitlement management.
Current Databricks guidance distinguishes account administrators, workspace administrators, and optional metastore administrators by scope.
18.16 Network architecture
Network design is not an afterthought.
Evaluate:
- private connectivity,
- inbound/outbound paths,
- cloud-native firewall controls,
- DNS,
- egress,
- data exfiltration risk,
- access to source systems,
- access to object storage,
- access to external services.
For every critical data path, draw:
Consumer
↓
Databricks service
↓
Network boundary
↓
Cloud resource
↓
Data
Then ask:
What prevents an unauthorized path from bypassing the intended boundary?
18.17 Observability and SLOs
An enterprise platform needs measurable objectives.
Examples:
| Service | SLO |
|---|---|
| Daily finance product | Available by 06:00 |
| Supplier spend | Fresh within 2 hours |
| Critical pipeline | ≥ 99.5% successful runs |
| Data quality | ≥ 99% critical checks pass |
| SQL serving | p95 query latency target |
| Recovery | RTO/RPO aligned with business tier |
Measure:
- pipeline success,
- latency,
- freshness,
- backlog,
- data quality,
- compute utilization,
- query performance,
- cost,
- security events.
18.18 FinOps is part of architecture
Every architecture decision has an economic consequence.
Evaluate:
Business value
÷
Total cost of ownership
TCO includes more than DBUs:
- compute,
- storage,
- cloud networking,
- data transfer,
- operational overhead,
- support,
- engineering time,
- platform tooling,
- DR,
- security controls.
Use workload-level unit economics where possible:
Cost per successful pipeline run
Cost per TB processed
Cost per dashboard refresh
Cost per million events
Cost per data product
If the platform cannot explain where cost comes from, the architecture is incomplete.
18.19 Enterprise reference architecture
A robust enterprise pattern can be visualized as:
ENTERPRISE
│
┌───────────────┴───────────────┐
│ │
Identity / IAM Architecture / FinOps
│ │
└───────────────┬───────────────┘
│
DATABRICKS ACCOUNT
│
┌─────────────────────┼─────────────────────┐
│ │ │
Region A Region B Region C
│ │ │
Metastore Metastore Metastore
│ │ │
┌─────┼─────┐ ┌─────┼─────┐ │
│ │ │ │ │ │ │
DEV TEST PROD DEV PROD SPECIAL ...
│ │
│ Data products
│ │
└───────┬───┘
│
┌─────┴──────────────────────────────────┐
│ │
Ingestion / CDC / Files Streaming
│ │
└────────────────┬───────────────────────┘
▼
Delta / Lakehouse
│
┌────────────┼────────────┐
▼ ▼ ▼
Analytics BI/SAC ML/AI
│
Business outcomes
The exact topology must be adapted to:
- geography,
- regulations,
- domain ownership,
- network requirements,
- workload criticality,
- team maturity,
- cost,
- recovery objectives.
18.20 Architecture decision record
For every major platform decision, record:
Decision
What are we choosing?
Context
What business and technical constraints exist?
Options
What alternatives were evaluated?
Decision criteria
What matters?
- security,
- latency,
- cost,
- reliability,
- operability,
- scalability,
- developer experience.
Consequences
What becomes easier?
What becomes harder?
Exit condition
Under what future condition should this decision be reconsidered?
This turns architecture from opinion into an auditable engineering practice.
18.21 Common enterprise anti-patterns
Anti-pattern 1 — Workspace sprawl
Creating a workspace for every team without a governance reason.
Why it fails: operational overhead, fragmented standards, duplicated configuration.
Anti-pattern 2 — Catalog explosion
Creating hundreds of catalogs for cosmetic organization.
Why it fails: permissions and ownership become difficult to reason about.
Anti-pattern 3 — One giant shared workspace
Putting every workload and team into one workspace despite materially different security or network requirements.
Why it fails: weak isolation and noisy-neighbor risk.
Anti-pattern 4 — Medallion by ritual
Creating Bronze/Silver/Gold for every dataset.
Why it fails: unnecessary copies and latency.
Anti-pattern 5 — Pipeline-centric architecture
Optimizing individual jobs without defining data products, owners, SLOs, or consumers.
Why it fails: local optimization produces enterprise inconsistency.
Anti-pattern 6 — Cloud storage as the security model
Assuming bucket permissions alone solve data governance.
Why it fails: governance needs identity, metadata, policy, lineage, and audit.
Anti-pattern 7 — DR as backup
Assuming replicated files automatically mean the platform is recoverable.
Why it fails: identity, configuration, dependencies, orchestration, networking, and operational procedures also need recovery.
18.22 Enterprise migration strategy
Do not migrate everything at once.
Use waves:
Assess
↓
Classify
↓
Prioritize
↓
Design target
↓
Pilot
↓
Industrialize
↓
Migrate by domain
↓
Decommission legacy
↓
Optimize
Classify workloads by:
- business criticality,
- data volume,
- latency,
- complexity,
- regulatory requirements,
- integration dependencies,
- current cost,
- migration risk.
For SAP-heavy organizations, migration should preserve business semantics and reconciliation—not simply move tables.
18.23 Architect lab — design a global lakehouse
Scenario
A global manufacturer has:
- SAP S/4HANA in multiple regions,
- procurement and finance domains,
- global BI,
- regional regulatory requirements,
- near-real-time operational analytics,
- batch financial reporting,
- ML workloads,
- SAC consumers,
- existing cloud storage,
- a central data platform team.
Your task
Design:
- account structure,
- workspace topology,
- regional Unity Catalog model,
- catalog strategy,
- data domain boundaries,
- SAP ingestion patterns,
- batch and streaming architecture,
- data contracts,
- security model,
- network model,
- observability,
- FinOps,
- HA/DR,
- operating model.
Deliverables
Produce:
- one context diagram,
- one deployment diagram,
- one data-flow diagram,
- five ADRs,
- three SLOs,
- one cost model,
- one failure scenario,
- one migration plan.
Architect defense
You must be able to answer:
Why did you choose this topology instead of a single shared workspace?
Why are these catalog boundaries correct?
What happens when a region fails?
What happens when a data contract changes?
What is the most expensive workload?
Which controls prevent unauthorized cross-domain access?
What would you change if the organization doubled in size?
18.24 Troubleshooting matrix
| Symptom | Likely architectural cause | First investigation |
|---|---|---|
| Teams cannot share data | Incorrect workspace/metastore/catalog model | Check regional metastore and sharing design |
| Too many access requests | Poor ownership / group model | Review data-product ownership |
| High cross-region cost | Excessive remote consumption | Measure access paths and sharing pattern |
| Slow dashboards | Serving layer not designed for consumption | Inspect warehouse/query workload |
| Pipeline succeeds but data is wrong | Weak contract/quality model | Inspect quality gates and reconciliation |
| Platform costs grow unexpectedly | No workload unit economics | Analyze usage by workload/domain |
| DR test fails | Only data was recovered | Test identities, config, orchestration, network, and dependencies |
18.25 Production readiness checklist
Architecture
- Business domains identified
- Data products defined
- Workspace topology justified
- Regional boundaries documented
- Unity Catalog architecture documented
- Data contracts defined
Security
- Identity model defined
- Service principals used for workloads
- Least privilege enforced
- Network boundaries documented
- Sensitive-data controls implemented
- Auditability verified
Data engineering
- Batch/streaming decisions justified
- Replay and backfill strategy defined
- Quality gates implemented
- Schema evolution policy defined
- Lineage available
Operations
- SLOs defined
- Monitoring implemented
- On-call ownership defined
- Runbooks available
- DR tested
FinOps
- Cost allocation implemented
- Workload classes defined
- Budgets/anomaly detection implemented
- Unit economics tracked
Architecture governance
- ADRs stored
- Standards published
- Exceptions documented
- Review cadence established
18.26 Interview questions
- What is the difference between a Databricks account, workspace, metastore, catalog, and schema?
- Why is Unity Catalog regional?
- When would you use multiple workspaces?
- When would you avoid creating another workspace?
- How would you design catalogs for a large enterprise?
- How would you design multi-region governance?
- How do managed and external storage patterns differ?
- When should Bronze/Silver/Gold not be used?
- How would you design a data product?
- How would you define a data contract?
- How would you integrate SAP with a lakehouse without forcing a false 1:1 mapping?
- How do you separate platform security from data authorization?
- How would you control cross-region data access?
- How do you measure lakehouse TCO?
- How would you migrate an enterprise from legacy analytics platforms?
- How would you prove the architecture is production-ready?
18.27 Final mental model
The enterprise architect should think in this order:
Business outcomes
↓
Domains + ownership
↓
Data products + contracts
↓
Governance + security
↓
Regional/account/workspace topology
↓
Ingestion + processing
↓
Serving + consumption
↓
SLO + observability
↓
FinOps
↓
HA / DR
↓
Operating model
↓
Continuous architecture evolution
If you start with "Which Databricks feature should I use?", you are starting too low.
Start with:
What business capability are we building, who owns it, what contract does it expose, what reliability does it require, who may access it, and what will it cost to operate?
Then choose the Databricks architecture.