Skip to main content

02. Lakehouse Architecture

A lakehouse combines the flexible, low-cost storage model of a data lake with capabilities expected from analytical data warehouses.

Four architectural layers​

1. Storage​

Cloud object storage such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage provides durable storage.

2. Table and transaction layer​

Delta Lake adds ACID transactions, schema enforcement, and reliable table semantics on top of files.

3. Compute​

Spark, Photon, SQL warehouses, streaming engines, and ML compute execute workloads independently of durable storage.

4. Governance​

Unity Catalog provides centralized governance, access control, discovery, auditing, and lineage.

Architecture​

                    USERS / APPLICATIONS
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
          BI / SQL      Data Eng       ML / AI
             │             │             │
             └─────────────┼─────────────┘
                           ▼
                 Databricks Platform
                           │
                 ┌─────────┴─────────┐
                 │                   │
             Compute             Governance
            Spark/Photon       Unity Catalog
                 │
                 ▼
              Delta Lake
                 │
                 ▼
          Cloud Object Storage

Medallion architecture​

Sources
   │
   ▼
Bronze ── raw / auditable
   │
   ▼
Silver ── cleaned / standardized / integrated
   │
   ▼
Gold ── business-ready / consumption optimized

Bronze preserves source information, Silver applies quality and integration rules, and Gold publishes consumer-oriented data products.

Why storage and compute separation matters​

  • Storage remains durable.
  • Batch jobs can scale independently.
  • SQL users can use SQL warehouses.
  • Streaming workloads can have dedicated compute.
  • ML workloads can use specialized compute.
  • Cost can be managed per workload.

Architect's rule​

Do not design a lakehouse by starting with folders. Start with business domains, ownership, workloads, security boundaries, SLAs, governance, and cost model.

Next​

Continue with the next topic in 01. Databricks Foundations.