08.7 Medallion Architecture — Bronze, Silver & Gold
1. The core idea
The medallion architecture is a logical data-design pattern in which data becomes progressively more reliable, structured, and business-ready as it moves through Bronze → Silver → Gold.
Databricks describes:
- Bronze — raw ingestion and source fidelity
- Silver — validated, cleaned, deduplicated and enriched data
- Gold — business-ready, semantically meaningful datasets for analytics and consumption
It is a recommended pattern, not a mandatory platform feature. The objective is to create progressively higher-quality data products rather than merely creating three schemas. Databricks medallion architecture
2. Why layers exist
Without clear boundaries, a lakehouse can become a collection of pipelines with unclear ownership:
Source A ---> random table
Source B ---> random table ---> BI
Source C ---> another transformation
A layered architecture introduces explicit contracts:
SOURCE
|
v
BRONZE
raw + replayable
|
v
SILVER
trusted + standardized
|
v
GOLD
business-ready
Each layer answers a different question.
| Layer | Question |
|---|---|
| Bronze | What did the source send us? |
| Silver | What is the trusted representation of that data? |
| Gold | What does the business need to consume? |
3. Bronze — preserve source fidelity
Bronze is the landing and raw-data layer.
Typical characteristics:
- minimal transformation;
- source fidelity;
- append-oriented ingestion;
- ingestion metadata;
- replayability;
- auditability;
- schema-change resilience;
- batch, streaming or CDC inputs.
Bronze is not supposed to be the polished analytical model.
Databricks recommends preserving raw source data so downstream layers can be rebuilt when transformation requirements change. Databricks reliability guidance
Example
bronze.sap_customer_cdc
bronze.salesforce_account_raw
bronze.orders_raw
bronze.kafka_orders
A Bronze record may include operational metadata such as:
_source_system
_ingest_timestamp
_batch_id
_file_name
_event_id
_sequence_number
Do not discard information in Bronze merely because Silver does not currently need it.
4. Silver — establish trust
Silver is where raw records become reliable data assets.
Typical operations include:
- schema enforcement;
- type conversion;
- null handling;
- deduplication;
- CDC application;
- normalization;
- parsing;
- data-quality rules;
- late/out-of-order handling;
- joins;
- standardization;
- enrichment.
Databricks recommends that Silver contain at least one validated, non-aggregated representation of each record so downstream workloads retain useful detail. Databricks medallion architecture
Example
silver.customer
silver.order
silver.product
silver.customer_order
Silver is frequently the best layer for reusable enterprise data engineering assets.
5. Gold — serve business meaning
Gold is designed around business consumption.
Examples:
gold.customer_sales
gold.daily_revenue
gold.procurement_spend
gold.finance_pnl
gold.inventory_snapshot
Gold commonly contains:
- dimensional models;
- aggregates;
- KPIs;
- business rules;
- semantic measures;
- data marts;
- application-facing datasets.
Gold should not become a second uncontrolled copy of Silver. Its purpose is to make business consumption easier, faster, and more consistent.
6. The most important principle: layer contracts
A mature medallion architecture defines a contract for each boundary.
Bronze contract
Preserve source evidence.
Silver contract
Produce trusted, standardized data.
Gold contract
Produce business-ready, consumption-oriented data.
This makes ownership and troubleshooting dramatically easier.
7. Medallion does not mean “three copies of everything”
A common beginner mistake is:
Bronze = full copy
Silver = full copy
Gold = full copy
That is not the architectural goal.
Instead:
Bronze: source evidence
Silver: reusable trusted entities
Gold: purpose-specific products
Some Silver tables may be wide and detailed. Gold may contain only the columns and grain needed by a particular data product.
8. Bronze design patterns
Batch ingestion
S3 / files
|
v
Bronze Delta
Streaming
Kafka / Kinesis / events
|
v
Bronze streaming
CDC
S/4HANA / database
|
v
CDC feed
|
v
Bronze
Federation
Not every source needs to be physically ingested. Lakehouse Federation can provide governed access to external systems where federation is the right trade-off.
This is an architectural decision, not a reason to bypass medallion thinking.
9. Silver processing patterns
Silver is where complexity becomes controlled.
Typical pipeline:
Bronze
|
+--> schema validation
+--> type casting
+--> deduplication
+--> CDC ordering
+--> data quality
+--> normalization
+--> enrichment
|
v
Silver
This is also where SCD1 and SCD2 commonly become reusable enterprise dimensions.
10. Gold modeling patterns
Aggregate
silver.sales
|
v
gold.daily_sales
Dimensional
silver.customer ---> gold.dim_customer
silver.product ---> gold.dim_product
silver.sales ---> gold.fact_sales
Semantic KPI
silver.finance
|
v
gold.revenue
gold.gross_margin
gold.working_capital
The Gold layer should expose stable business semantics rather than forcing every dashboard to recreate calculations.
11. Streaming vs batch by layer
The workload should determine the processing mode.
A common pattern is:
| Layer | Typical processing |
|---|---|
| Bronze | Streaming/incremental ingestion |
| Silver | Streaming or incremental transformation |
| Gold | Incremental materialization or batch-style aggregation |
Databricks specifically recommends streaming for many Bronze ingestion workloads, while Silver and Gold often benefit from incremental materialized views or batch-style processing depending on latency and stateful-processing requirements. Batch vs streaming guidance
Do not make every layer streaming merely because the source is streaming.
12. Separate ingestion from transformation
A production architecture should often decouple Bronze ingestion from Silver/Gold transformation.
Pipeline A
Sources ---> Bronze
|
v
Pipeline B
Silver ---> Gold
Benefits:
- ingestion continues during downstream failures;
- independent schedules;
- clearer ownership;
- easier replay;
- isolated monitoring;
- safer deployments.
Databricks recommends separating ingestion and transformation pipelines when practical. Lakeflow pipeline best practices
13. Data quality gates
Quality should increase as data moves forward.
Example:
Bronze
|
| schema / completeness
v
Silver
|
| business rules / referential checks
v
Gold
A failed Silver quality rule should not silently contaminate Gold.
Use expectations or equivalent quality controls appropriate to the pipeline technology. Lakeflow expectations can report or enforce row-level quality behavior. Lakeflow pipeline guidance
14. Quarantine pattern
Do not always discard invalid data.
Bronze
|
+---- valid ----> Silver
|
+---- invalid --> quarantine
A quarantine dataset should preserve enough information to investigate and replay rejected records.
Example:
silver.customer_valid
silver.customer_quarantine
15. Schema evolution
Schema changes are inevitable.
Bronze
Prefer resilience and preservation.
Silver
Apply controlled schema contracts.
Gold
Protect consumer-facing contracts.
Example:
source adds column
|
v
Bronze accepts
|
v
Silver evaluates
|
v
Gold changes only through governed release
This prevents an upstream source change from unexpectedly breaking dashboards.
16. Idempotency and replay
A mature medallion design assumes that pipelines will fail.
If Silver can be rebuilt from Bronze, recovery becomes simpler:
Bronze retained
|
+--> rebuild Silver
|
+--> rebuild Gold
This is one of the strongest reasons to preserve Bronze correctly.
17. Medallion + CDC
For CDC:
Source
|
CDC events
|
v
Bronze raw events
|
v
Silver SCD1/SCD2
|
v
Gold facts / dimensions
Bronze should preserve event evidence. Silver interprets change semantics. Gold exposes business meaning.
18. Medallion + SCD1/SCD2
A practical pattern:
| Layer | Example |
|---|---|
| Bronze | raw customer CDC |
| Silver | customer SCD2 |
| Gold | current customer mart |
| Gold | customer historical analytics |
Do not force SCD2 into Gold simply because it is “historical.” SCD2 is a modeling technique whose correct layer depends on its role in the enterprise data contract.
19. Medallion + Unity Catalog
A common namespace:
enterprise
|
+-- sales
| +-- bronze
| +-- silver
| +-- gold
|
+-- finance
+-- bronze
+-- silver
+-- gold
Or a platform-wide catalog model:
catalog
+-- bronze
+-- silver
+-- gold
There is no universal naming scheme. Choose based on domain ownership, governance, workspace strategy, and data-product boundaries.
20. Domain-oriented medallion
At enterprise scale, one giant Bronze/Silver/Gold namespace can become difficult to govern.
A stronger model is often:
Finance
Bronze -> Silver -> Gold
Procurement
Bronze -> Silver -> Gold
Sales
Bronze -> Silver -> Gold
Each domain owns its data products while platform governance remains centralized through Unity Catalog.
This connects medallion architecture with data-mesh and data-product principles.
21. Enterprise reference architecture
ENTERPRISE SOURCES
|
+---------------+---------------+
| | |
SAP CRM Events
| | |
+---------------+---------------+
|
v
BRONZE / RAW
source-fidelity + replay
|
v
SILVER / TRUSTED
validation + CDC + standardization
|
+------------+------------+
| | |
v v v
Gold BI Gold ML Data Products
| | |
+------------+------------+
|
Consumers / Apps
Unity Catalog governs access, discovery, lineage, and data-product boundaries across the architecture.
22. SAP/BW perspective
For someone coming from SAP BW, a useful mental mapping is:
| Traditional BW concept | Lakehouse analogue |
|---|---|
| PSA/raw acquisition | Bronze |
| Harmonization / staging | Silver |
| CompositeProvider / reusable semantic layer | Silver/Gold depending on purpose |
| Queries / reporting models | Gold |
| Process chains | Lakeflow Jobs / orchestration |
| InfoObjects | Governed business attributes |
| ADSO | Delta tables / managed tables |
This is a conceptual mapping, not a one-to-one product equivalence.
The important architectural shift is that the lakehouse separates raw evidence, trusted reusable data, and business consumption models.
23. Performance strategy
Do not optimize all layers identically.
Bronze
Optimize for ingestion reliability and replay.
Silver
Optimize for transformation efficiency, incremental processing, joins, deduplication, and data quality.
Gold
Optimize for consumer query latency, concurrency, and predictable business workloads.
Use Delta layout, clustering, materialization, Photon, caching, and other capabilities according to workload evidence rather than applying every optimization everywhere.
24. Cost strategy
Cost increases when:
- every layer is recomputed unnecessarily;
- Bronze is repeatedly re-read without incremental logic;
- Gold tables duplicate large Silver datasets;
- streaming is used where batch is sufficient;
- excessive historical versions are generated;
- unnecessary columns are carried through every layer.
A strong architecture minimizes repeated work while preserving replayability.
25. Anti-patterns
Anti-pattern 1 — Business logic in Bronze
Bronze should not become a hidden transformation layer.
Anti-pattern 2 — Raw data directly to Gold
This creates brittle pipelines and weak replayability.
Anti-pattern 3 — Gold as a dumping ground
Gold should contain intentional business products.
Anti-pattern 4 — Three copies of every table
Layering is about responsibility and quality, not mandatory duplication.
Anti-pattern 5 — One pipeline for everything
Coupling ingestion, transformation, and serving increases blast radius.
Anti-pattern 6 — No data-quality boundary
Bad Silver data eventually becomes bad Gold data.
Anti-pattern 7 — Treating every source as CDC
Use the ingestion pattern that matches the source capabilities and business SLA.
26. Lab 1 — Build a three-layer pipeline
Create:
bronze.orders_raw
silver.orders
gold.daily_sales
Tasks:
- Ingest raw order records.
- Validate and standardize Silver.
- Deduplicate orders.
- Create a daily Gold aggregate.
- Add data-quality checks.
- Document the contract of every layer.
27. Lab 2 — Failure and replay
Intentionally break the Silver transformation.
Then:
- keep Bronze ingestion running;
- repair the Silver logic;
- rebuild Silver from Bronze;
- rebuild Gold;
- reconcile results.
The goal is to demonstrate that Bronze is an operational recovery boundary.
28. Lab 3 — CDC medallion
Implement:
CDC source
-> Bronze events
-> Silver SCD1
-> Gold current customer analytics
Then extend it to SCD2 and create a historical Gold report.
29. Lab 4 — Enterprise domain design
Design Finance and Procurement domains.
Each must contain:
- ingestion;
- Bronze;
- Silver;
- Gold;
- ownership;
- quality;
- governance;
- lineage;
- publishing rules.
Explain where shared customer and supplier data belongs and how domains consume it without uncontrolled duplication.
30. Architect decision framework
Ask these questions:
- What must be preserved exactly as received?
- What data becomes trusted and reusable?
- What business questions must Gold answer?
- Which transformations are reusable?
- Which outputs are domain-specific?
- What is the freshness SLA?
- What must be replayable?
- Where should quality failures stop?
- Who owns each dataset?
- What is the governance boundary?
If these questions cannot be answered, the architecture is not mature enough.
31. Production checklist
- Bronze preserves source evidence.
- Bronze is replayable.
- Silver has explicit quality contracts.
- Silver contains reusable trusted data.
- Gold has defined business consumers.
- Layer ownership is documented.
- Data quality failures have explicit handling.
- Ingestion is decoupled where appropriate.
- Batch vs streaming is workload-driven.
- CDC semantics are explicit.
- SCD semantics are explicit.
- Schema evolution is governed.
- Unity Catalog permissions are defined.
- Lineage is available.
- Replay/recovery is tested.
- Cost and performance are measured.
- Gold products have stable semantic definitions.
32. Architect mental model
BRONZE
"What arrived?"
|
v
SILVER
"What can we trust?"
|
v
GOLD
"What does the business need?"
Medallion architecture is a set of data contracts and responsibilities, not simply three folders named Bronze, Silver, and Gold.
When designed correctly, it gives an enterprise a reliable path from raw source evidence to governed data products.