07.7 Unity Catalog Data Lineage & Metadata Governance
A governed lakehouse needs more than permissions. It must answer where data came from, how it was transformed, where it is consumed, who owns it, whether it is trusted, and what breaks when it changes.
Unity Catalog combines metadata, discovery, lineage, auditing, classification, and governance signals. Current Databricks lineage can capture relationships across tables, columns, jobs, notebooks, dashboards, pipelines, models, and other governed assets. It can also aggregate lineage across workspaces attached to the same metastore. Official lineage documentation
1. Metadata vs lineage vs governance
| Concept | Main question |
|---|---|
| Metadata | What is this asset? |
| Lineage | Where did it come from and where does it go? |
| Governance | Who may use it and under what rules? |
| Audit | What actually happened? |
| Classification | What kind of sensitive/business data is it? |
| Discovery | Can people find and understand it? |
Think of the lifecycle as:
Metadata ↓ Discovery ↓ Lineage ↓ Impact analysis ↓ Governance decision ↓ Audit evidence
Lineage without metadata tells you relationships but not meaning. Metadata without lineage tells you meaning but not dependencies.
2. What Unity Catalog lineage can capture
Current Unity Catalog lineage can include:
- tables and views
- columns
- file paths
- materialized views
- streaming tables
- jobs
- notebooks
- pipelines
- SQL queries
- dashboards
- ML model versions
- model services and providers
- UDF relationships
- registered external assets
Lineage is captured automatically for supported Databricks workloads and is available through Catalog Explorer and lineage system tables.
The key architectural idea is:
Lineage is evidence generated from actual workload execution and governed object relationships, not merely a manually maintained dependency spreadsheet.
3. Enterprise lineage architecture
A simplified data flow is:
Source ↓ Bronze ↓ Silver ↓ Gold ↓ Dashboard / ML / API
↓ Unity Catalog ↓ Lineage graph ↓ Impact / governance / audit
Example:
SAP S/4HANA ↓ purchase_orders_bronze ↓ purchase_orders_silver ↓ supplier_spend_gold ↓ Procurement dashboard
An architect should be able to traverse this entire chain.
4. Column-level lineage
Table lineage answers:
Which tables feed this table?
Column lineage answers:
Which source columns contributed to this column?
Example:
supplier_spend_gold.total_spend ↑ purchase_orders_silver.net_value ↑ SAP purchase_order.net_value
This is important for:
- PII
- financial measures
- regulatory reporting
- derived KPIs
- sensitive customer attributes
- impact analysis
Databricks captures column lineage where the relationship can be inferred. It has documented limitations, so lineage should be treated as a high-value governance signal rather than proof of every transformation.
5. Why lineage matters to an architect
Lineage enables operational decisions.
Impact analysis
Before changing a column:
supplier_id ↓ Which tables? Which jobs? Which dashboards? Which ML assets? Which consumers?
Root-cause analysis
If a dashboard is wrong:
Dashboard ↓ Gold ↓ Silver ↓ Bronze ↓ Source
Trace backward until the divergence is found.
Compliance
For sensitive data:
Sensitive source ↓ Transformations ↓ Gold assets ↓ Reports ↓ External consumers
Lineage provides evidence of propagation.
6. Catalog Explorer lineage
Typical workflow:
- Open Catalog.
- Find the governed table or view.
- Open the Lineage tab.
- Inspect upstream and downstream relationships.
- Expand the graph.
- Inspect lineage details.
- Select a column when column lineage is available.
- Follow associated jobs, notebooks, pipelines, or dashboards.
The graph initially presents a local neighborhood and can be expanded.
Lineage viewing is permission-aware. Users need appropriate catalog/object access, including BROWSE where applicable, and workspace-object permissions for notebooks, jobs, and dashboards.
This prevents lineage from becoming an information-disclosure back door.
7. Cross-workspace lineage
Suppose an enterprise has:
DEV workspace TEST workspace PROD workspace BI workspace ML workspace
and they share a Unity Catalog metastore.
The lineage graph can aggregate relationships across those workspaces.
Unity Catalog Metastore | +-----+-----+-----+ | | | | DEV TEST PROD BI | | | | pipelines tables dashboards +-----+-----+-----+ ↓ lineage graph
Workspace-specific details remain subject to permissions.
Architectural implication:
Unity Catalog should be viewed as an account-level governance plane, not merely a workspace feature.
8. Lineage system tables
For programmatic governance, use:
system.access.table_lineage system.access.column_lineage
These can support:
- governance dashboards
- dependency reports
- automated impact analysis
- audit workflows
- lineage coverage metrics
- data-platform observability
The lineage system tables currently retain a rolling 395-day window by default. Catalog Explorer and the lineage API provide longer historical availability for captured lineage after September 1, 2024.
Do not confuse system-table retention with the historical lineage available through Catalog Explorer/API.
9. Querying table lineage
Example:
SELECT event_time, source_table_full_name, target_table_full_name, entity_type, created_by FROM system.access.table_lineage WHERE target_table_full_name = 'prod.gold.supplier_spend';
This can become the foundation for a governance report:
Target table ↓ Upstream sources ↓ Transformation workload ↓ Owner ↓ Last observed lineage event
Always validate the current system-table schema before production automation because Databricks evolves system-table schemas.
10. Column lineage for sensitive data
Consider:
SAP customer.tax_id ↓ customer_silver.tax_id ↓ customer_gold.tax_identifier ↓ executive_dashboard
Governance can ask:
- Where did the sensitive column originate?
- Where was it transformed?
- Where is it exposed?
- Which dashboards use it?
- Which teams own those assets?
- Which policy applies?
This is substantially stronger than tagging only the source table.
11. Metadata: the context around data
Lineage describes relationships.
Metadata provides meaning.
Useful metadata includes:
- table name
- schema
- owner
- descriptions
- column definitions
- data types
- tags
- certification status
- deprecation status
- comments
- usage information
- lineage
- quality information
- business definitions
A production dataset should be understandable without requiring a meeting with its creator.
12. Information Schema
Unity Catalog provides SQL-accessible metadata through Information Schema.
The system-level Information Schema exposes metadata across catalogs in the metastore, while each Unity Catalog catalog also has an information_schema describing objects within that catalog.
Information Schema is privilege-aware and filters results to objects the querying principal can access.
Example:
SELECT * FROM system.information_schema.tables;
Use it for:
- asset inventory
- ownership reports
- schema audits
- table counts
- governance coverage
- documentation generation
13. Metadata quality is an architecture concern
This:
t001
may be technically valid but is poor enterprise metadata.
Prefer:
supplier_invoice_payment_status
with:
- business description
- owner
- domain
- refresh expectation
- sensitive-data classification
- certification status
- source system
The target is:
Technical metadata + Business metadata
Usable data product
14. Comments and documentation
A production table should answer:
What does this table represent?
A production column should answer:
What exactly does this column mean?
Example:
COMMENT ON TABLE prod.gold.supplier_spend IS 'Daily supplier spend aggregated from approved procurement transactions';
COMMENT ON COLUMN prod.gold.supplier_spend.net_spend IS 'Net purchase value after applicable discounts and adjustments';
Standardize documentation requirements rather than leaving them to individual developers.
15. Certification and deprecation
Discovery becomes dangerous when every dataset appears equally trustworthy.
A useful enterprise vocabulary is:
CERTIFIED ↓ Approved enterprise dataset
DEPRECATED ↓ Do not build new dependencies
UNVERIFIED ↓ Use with caution
Databricks supports certification/deprecation signals in Catalog Explorer using governance tags.
16. External lineage
Enterprise data flows often extend beyond Databricks.
Example:
Salesforce ↓ Fivetran ↓ Databricks ↓ Power BI
Runtime lineage can describe the Databricks portion. External lineage can represent upstream and downstream systems outside Databricks.
This produces:
External source ↓ External lineage ↓ Unity Catalog ↓ Runtime lineage ↓ External lineage ↓ External consumer
This is essential when Databricks is the middle of a larger enterprise ecosystem.
17. SAP → Databricks lineage
For an SAP-oriented architecture:
SAP S/4HANA ↓ ODP / SLT / CDC ↓ Bronze ↓ Silver ↓ Gold ↓ SAC / Power BI / Tableau
An architect should be able to answer:
Which SAP field ultimately feeds this KPI?
Example:
S/4HANA EKPO-NETWR ↓ purchase_item_bronze.net_value ↓ purchase_item_silver.net_value ↓ supplier_spend_gold.total_spend ↓ Procurement KPI
This is especially valuable when modernizing or migrating BW/4HANA landscapes.
18. Lineage and schema-change impact analysis
Suppose an upstream team proposes:
supplier_id ↓ vendor_id
Do not execute the change immediately.
First identify:
Downstream tables ↓ Views ↓ Jobs ↓ Dashboards ↓ ML features/models ↓ External consumers
Then classify the dependency:
| Dependency | Typical action |
|---|---|
| Critical production report | Formal change review |
| Internal development job | Coordinate |
| Deprecated asset | Migrate/remove |
| Certified data product | Formal owner approval |
| External consumer | Contract/change notification |
Lineage therefore becomes part of enterprise change management.
19. Lineage does not equal data quality
A table can have complete lineage and still contain incorrect data.
Example:
SAP ↓ Bronze ↓ Silver ↓ Gold
The path is known.
But a faulty transformation can still calculate:
Revenue = Quantity × Wrong Price
Therefore:
Lineage + Data quality + Observability
Trust
Lineage tells you the path. Quality tells you whether the result is reliable.
20. Lineage + audit
Lineage tells you:
How are assets connected?
Audit tells you:
What actions occurred?
Databricks exposes audit information through system audit tables, and lineage tracking events can appear in the audit system with service_name = lineageTracking.
A mature governance platform combines:
Lineage + Audit + Access control + Metadata + Quality
This provides both structural and behavioral evidence.
21. Governance architecture
A practical enterprise model is:
Unity Catalog | +-----+-----+-----+ | | | Security Metadata Lineage | | | privileges owners dependencies groups tags columns policies docs workloads +-----------+-----+ | Governance | +--------+--------+ | | | Discovery Impact Audit analysis
This is much broader than permissions alone.
22. Metadata operating model
Define clear responsibilities.
Data owner
Accountable for:
- business meaning
- access decisions
- certification
- retention/business policy
Data steward
Responsible for:
- definitions
- metadata completeness
- classification
- quality coordination
Data engineer
Responsible for:
- pipeline implementation
- technical metadata
- lineage-producing workloads
- operational correctness
Platform team
Responsible for:
- Unity Catalog
- governance infrastructure
- system tables
- monitoring
- policy enforcement
This separates business accountability from platform administration.
23. Metadata quality score
Create a measurable governance score.
For example:
Metadata Score = description present
- owner present
- classification present
- certification status
- lineage available
- quality monitoring
Example governance table:
| Asset | Owner | Description | Lineage | Classification | Certified |
|---|---|---|---|---|---|
| supplier_spend | Yes | Yes | Yes | Yes | Yes |
| raw_orders | Yes | No | Yes | No | No |
| customer_gold | Yes | Yes | Yes | Yes | Yes |
This converts governance from a policy document into measurable operations.
24. Governance data mart
System metadata can be transformed into a governance data mart:
system.access.table_lineage system.access.column_lineage system.access.audit Information Schema ↓ Governance ETL ↓ Governance Gold ↓ Steward / Executive BI
Possible KPIs:
- assets without owners
- assets without descriptions
- sensitive assets without classification
- certified assets
- deprecated assets still queried
- highly connected datasets
- unused datasets
- lineage coverage
- recent access
- top downstream dependencies
This is an architect-level pattern because governance becomes observable and measurable.
25. Critical asset analysis
Lineage can identify highly connected assets.
Example:
customer_master ↓ 37 Silver tables ↓ 18 Gold tables ↓ 42 dashboards
This is a critical enterprise dependency.
Treat it differently from:
one_off_test_table
Possible additional controls:
- stronger change management
- higher availability
- more monitoring
- explicit certification
- stronger ownership
- recovery planning
- dependency review
Lineage can therefore influence reliability architecture.
26. Important lineage limitations
Never assume lineage captures every transformation.
Current documented limitations include scenarios involving:
- RDDs
- global temporary views
- certain job submission modes
- checkpointing
- some private Lakeflow pipeline scenarios
- renamed objects
- some temporary-table write patterns
- path-based references for certain column-lineage views
- UDFs where column mappings cannot be inferred
Architectural rule:
Absence of lineage is not proof that no dependency exists.
Treat lineage as strong evidence with known coverage boundaries.
27. Temporary tables and lineage
Temporary objects do not become permanent Catalog objects.
When supported temporary staging objects sit between permanent datasets, lineage can connect the permanent source directly to the permanent target.
This is useful because a temporary staging implementation does not need to become a permanent governance node.
However, documented limitations apply to certain UPDATE, DELETE, and MERGE patterns involving temporary objects.
28. Performance and operational design
Governance should not become a poorly designed workload.
For enterprise reporting:
Raw system metadata ↓ Incremental governance pipeline ↓ Curated governance tables ↓ Dashboard
Avoid repeatedly scanning large raw system-table datasets for every dashboard refresh.
Use incremental processing, appropriate retention, and curated governance marts.
29. Hands-on Lab 1 — Explore lineage
Create:
bronze_orders ↓ silver_orders ↓ gold_customer_sales
Run the transformations using Unity Catalog tables.
Then:
- Open Catalog Explorer.
- Open gold_customer_sales.
- Open Lineage.
- Inspect upstream assets.
- Expand downstream relationships.
- Inspect column lineage.
Goal: understand table-level versus column-level lineage.
30. Hands-on Lab 2 — Query lineage system tables
Run:
SELECT event_time, source_table_full_name, target_table_full_name, entity_type, created_by FROM system.access.table_lineage ORDER BY event_time DESC;
Then inspect:
SELECT * FROM system.access.column_lineage LIMIT 100;
Goal: move from visual lineage to programmatic governance.
31. Hands-on Lab 3 — Build impact analysis
Choose:
prod.gold.supplier_spend
Create an impact report containing:
- upstream tables
- downstream tables
- jobs
- dashboards
- important columns
- owners
- recent lineage events
Then classify each dependency as critical, normal, deprecated, or external.
This simulates a real enterprise change-review process.
32. Hands-on Lab 4 — Metadata quality audit
Use Information Schema to build an inventory containing:
- table
- schema
- owner
- comments
- column count
Calculate:
Documentation Coverage = documented assets / total assets
Set an enterprise target such as greater than 95 percent for certified production domains.
33. Hands-on Lab 5 — SAP impact analysis
Model:
S/4HANA ↓ Bronze ↓ Silver ↓ Gold ↓ BI
Select a business field such as Purchase Order Net Value.
Document:
- source field
- ingestion object
- Silver transformation
- Gold KPI
- dashboard
- owner
- downstream consumers
This is a practical bridge between SAP/BW architecture and Databricks governance.
34. Architect decision framework
When evaluating lineage, ask:
- What is the source-of-truth system?
- Which transformations occur inside Databricks?
- Which occur outside Databricks?
- Are external assets represented?
- Is table lineage sufficient?
- Do we need column lineage?
- Who owns the asset?
- Is it certified?
- Is sensitive data classified?
- What is the downstream blast radius?
- How long must lineage be retained?
- Should system-table data be materialized into a governance mart?
- What lineage gaps exist?
- How will schema changes be impact-analyzed?
- How will lineage evidence be used during audit?
35. Enterprise metadata architecture
A strong enterprise platform can look like:
Source systems | External lineage | Databricks | +----+----+ | | Data Workloads assets | | +----+----+ | Unity Catalog | +----+----+----+ | | | Metadata Lineage Security | | | +----+----+----+ | Governance data mart | +----+----+ | | Steward Executive BI
This is the difference between using Unity Catalog as a registry and using it as an enterprise governance platform.
36. Interview questions
Q1. What is data lineage?
The traceable relationship showing where data comes from, how it is transformed, and where it is consumed.
Q2. Does Unity Catalog capture column lineage?
Yes, where the relationship can be inferred, with documented limitations.
Q3. Is lineage available across workspaces?
Yes, lineage can be aggregated across workspaces attached to the same Unity Catalog metastore, subject to permissions.
Q4. Where can lineage be queried?
Using lineage system tables such as system.access.table_lineage and system.access.column_lineage.
Q5. Is lineage the same as audit?
No. Lineage describes relationships; audit records activity.
Q6. Is lineage the same as data quality?
No. Lineage describes flow; quality evaluates correctness and health.
Q7. What is external lineage?
Metadata representing upstream or downstream relationships outside Databricks.
Q8. Why is column lineage important?
It enables fine-grained impact analysis and sensitive-data propagation analysis.
Q9. Can absence of lineage prove there is no dependency?
No. Lineage has documented coverage limitations.
Q10. How would you use lineage during a schema migration?
Identify dependencies, classify critical consumers, coordinate owners, migrate consumers, and validate the resulting lineage.
37. Production checklist
- Every production domain has an accountable owner.
- Production tables have useful descriptions.
- Important columns have business definitions.
- Sensitive data is classified.
- Certified datasets are clearly identified.
- Deprecated datasets are clearly marked.
- Lineage is monitored for critical data products.
- External relationships are represented where required.
- System lineage tables are available to governance teams.
- Governance reporting uses curated metadata marts where appropriate.
- Lineage limitations are documented.
- Schema changes use lineage for impact analysis.
- Audit and lineage are treated as complementary.
- Critical dependencies have stronger operational controls.
- Metadata completeness is measured.
38. Final mental model
METADATA What is this? ↓ LINEAGE Where did it come from? Where does it go? ↓ IMPACT What breaks if it changes? ↓ GOVERNANCE Who owns it? Who can use it? Is it sensitive? Is it trusted? ↓ AUDIT What actually happened?
The enterprise architect should be able to move from a dashboard KPI all the way back to its source system, understand important transformations and dependencies, identify the accountable owner, and determine the impact of a proposed change.
That is the real purpose of Unity Catalog lineage.