07.10 Audit, Access Monitoring & Governance Observability

Enterprise governance does not stop at "who can access what?"

A mature Unity Catalog implementation must also answer:

• Who accessed the data?
• What action did they perform?
• From which workspace?
• Which identity actually authorized the action?
• Which sensitive assets were accessed?
• What changed in governance configuration?
• Which assets are heavily used?
• Which policies are generating risk?
• Can security and compliance reconstruct an incident?

The core observability building blocks are:

Audit logs + system tables + lineage + classification + usage metadata + governance metrics

Databricks system tables provide a Databricks-hosted analytical store for operational data, including audit/security events, lineage, data classification results, data quality results, compute, billing and workload information. The audit system table is currently Public Preview, and availability varies by cloud, region and feature state.

Official references:

• Audit log reference
• Audit log system table
• System tables reference
• Unity Catalog governance
• Unity Catalog

1. Audit vs lineage vs monitoring

Capability	Core question
Audit	What action happened, and who performed it?
Lineage	Where did data come from and where did it go?
Classification	What sensitive data exists?
Access control	Who is allowed to access it?
Usage monitoring	How is the asset being used?
Data quality	Is the data healthy?
Governance observability	Is the whole control system working?

A security incident often requires all of them.

Example:

User accesses customer.email ↓ Audit → who / when / action ↓ Classification → email is sensitive ↓ ABAC → should be masked ↓ Lineage → downstream consumers ↓ Usage → normal or anomalous pattern ↓ Investigation / remediation

---

2. The system catalog

System tables live in the system catalog.

Current examples include:

• system.access.audit
• system.access.table_lineage
• system.access.column_lineage
• system.data_classification.results
• system.data_quality_monitoring.table_results
• system.billing.usage
• compute and warehouse history
• other feature-specific operational tables

Availability and lifecycle status vary by table. Treat the current system-table reference as the source of truth.

System tables are Databricks-managed operational data exposed through Unity Catalog. Do not casually export everything to another platform because system-table data can contain sensitive operational information.

---

3. Audit events

The audit system table records audit events.

Important fields include:

• event_time
• event_date
• user_identity
• service_name
• action_name
• request_id
• request_params
• response
• audit_level
• event_id

The exact fields vary by service and action. Some request parameters containing SQL definitions are restricted to highly privileged identities.

A useful model is:

Identity + Service + Action + Target / request parameters + Result + Time ↓ Audit event

---

4. Audit is not real-time telemetry

Databricks documents that audit system-table data is updated throughout the day and does not support real-time monitoring.

Therefore:

Audit analytics are excellent for historical investigation and governance analytics, but should not automatically be treated as a sub-second SIEM stream.

For real-time security detection, architect an appropriate security-event pipeline in addition to historical audit analytics.

---

5. Unity Catalog events

Unity Catalog activity is logged under the unityCatalog service for relevant events.

Examples include operations involving:

• metastores;
• catalogs;
• schemas;
• tables;
• volumes;
• storage governance;
• permissions;
• other governed objects.

The Databricks audit-log reference is the authoritative event catalog because event coverage evolves.

---

6. Lineage audit events

Lineage events appear under the lineageTracking service in the audit system table.

Although lineage is associated with a workspace, relevant lineage events are logged at account level. The originating workspace can be identified through request parameters.

Example:

SELECT * FROM system.access.audit WHERE service_name = 'lineageTracking' AND request_params['workspace_id'] = 'workspace_id';

This is an important multi-workspace governance pattern.

---

7. Table and column lineage system tables

Unity Catalog provides:

• system.access.table_lineage
• system.access.column_lineage

These tables provide programmatic access to lineage information and currently retain a rolling 395-day window by default. Catalog Explorer and the lineage API retain captured lineage beyond that system-table window for supported historical lineage.

This creates two useful layers:

Interactive lineage → Catalog Explorer

Analytical lineage → system tables

---

8. Governance observability architecture

A practical architecture is:

Unity Catalog | +-----+-----+-----+ | | | Access Classification Lineage | | | +-----+-----+-----+ | System Tables | +-----+-----+-----+ | | | Audit Usage Quality | | | +-----+-----+-----+ | Governance Data Mart | +-----+-----+-----+ | | | Security Compliance Platform Dashboard Reports Operations

The governance data mart should answer business questions rather than simply copy raw system tables.

---

9. Access monitoring

A useful access-monitoring model contains:

Dimension	Example
Who	user/service principal
What	catalog/schema/table/column
Action	SELECT / CREATE / ALTER / DROP
When	event timestamp
Where	workspace
How	SQL / notebook / job / API
Result	success/failure
Sensitivity	PII / financial / confidential
Business owner	Finance / Procurement

This allows security teams to prioritize meaningful events.

---

10. Identity attribution

Identity attribution becomes critical with service principals and run-as identities.

Databricks documents identity metadata such as run_by and run_as. For workload execution, the identity initiating an operation can differ from the identity whose permissions authorize it.

Example:

Human developer ↓ starts job ↓ Job service principal ↓ run_as identity ↓ reads production table

A compliance report that records only the final service principal can lose important attribution.

---

11. Workspace attribution

Multi-workspace enterprises need to distinguish:

• development;
• test;
• production;
• business-unit workspaces;
• regional workspaces.

For account-level events, workspace_id can be zero while the originating workspace appears in request parameters for relevant events.

Normalize workspace identity in your governance mart. Do not assume every event has identical workspace fields.

---

12. Security event classification

Not every audit event has equal importance.

Critical

• permission escalation;
• ownership changes;
• storage credential changes;
• external location changes;
• security-policy changes;
• sensitive-data access anomalies.

High

• DROP or ALTER of critical assets;
• bulk access to restricted data;
• unexpected service-principal activity;
• production governance changes.

Medium

• normal metadata changes;
• development object creation;
• routine queries.

Informational

• catalog discovery;
• expected metadata reads;
• normal dashboard usage.

This turns a huge audit stream into an actionable security program.

---

13. Sensitive-data access monitoring

Connect this module to 07.9.

Classification ↓ Sensitive asset inventory ↓ Audit events ↓ Sensitive-data access mart ↓ Risk dashboard

Example question:

Which users accessed restricted supplier bank data during the last 30 days?

Answering it requires:

1. sensitive-data classification;
2. asset mapping;
3. audit events;
4. identity resolution;
5. time filtering.

---

14. Governance metrics

Create metrics that measure governance effectiveness.

Classification coverage

classified sensitive columns

total relevant columns

Policy coverage

sensitive assets with policy

sensitive assets

Unmasked exposure

sensitive accesses without expected masking

sensitive accesses

Ownership coverage

assets with valid owner

governed assets

Also measure:

• policies not reviewed recently;
• inactive owners;
• unused assets;
• stale external locations;
• old service principals.

---

15. Governance scorecard

A useful executive dashboard:

KPI	Meaning
Sensitive-data coverage	How much sensitive data is classified
Policy coverage	How much sensitive data is protected
Access anomalies	Suspicious access patterns
Ownership coverage	Governance accountability
Lineage coverage	Traceability
Quality incidents	Data health
Orphaned assets	Governance debt
Privilege exceptions	Security debt

The goal is not to maximize every number. The goal is to make risk visible.

---

16. Example audit query

Start with a narrow, time-bounded query:

SELECT event_time, user_identity, service_name, action_name, request_id, audit_level FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 7 DAYS ORDER BY event_time DESC;

Then narrow it further by workspace, user, service, action and target object.

Avoid building an enterprise dashboard with unbounded scans of raw audit data.

---

17. Security investigation query

A conceptual investigation:

SELECT event_time, user_identity.email, service_name, action_name, request_params, response FROM system.access.audit WHERE event_date BETWEEN DATE '2026-11-01' AND DATE '2026-11-07' AND service_name = 'unityCatalog' AND action_name IN ( 'getTable', 'updateTable', 'deleteTable' ) ORDER BY event_time DESC;

Always validate action names against the current audit-log reference before deploying a production detector.

---

18. System-table permissions

System tables are themselves governed.

Users need the appropriate Unity Catalog permissions, commonly including:

• USE CATALOG on system;
• USE SCHEMA on the relevant system schema;
• SELECT on the relevant system table/schema;
• query-compute permission.

This produces an important recursive principle:

the governance data itself must be governed.

---

19. Retention architecture

Current documented default retention for many system tables is 395 days, while some tables have different retention models.

If compliance requires seven years:

Databricks system table ↓ controlled extraction ↓ long-term governance archive ↓ immutable / access-controlled storage

Do not assume Databricks default retention automatically satisfies every regulatory requirement.

---

20. Governance data mart

Do not expose raw system tables to every business user.

Build curated governance datasets such as:

• governance.access_events;
• governance.sensitive_access;
• governance.policy_changes;
• governance.asset_inventory;
• governance.lineage_edges;
• governance.ownership;
• governance.risk_scores.

Then expose:

• Security dashboard;
• Compliance dashboard;
• Data-owner dashboard;
• Platform-operations dashboard.

This creates a clean abstraction.

---

21. Detection pipeline

For advanced enterprises:

system.access.audit | v Normalize identities | v Enrich assets | +----+----+----+ | | | Classification Lineage Ownership | | | +----+----+----+ | Risk rules | Alerts / cases / dashboards

Every derived alert should preserve the original audit event ID and timestamp so it can be traced back to source evidence.

---

22. Historical vs near-real-time architecture

Use two lanes.

Historical governance lane

System tables ↓ Governance mart ↓ BI / compliance

Near-real-time security lane

Security-relevant events ↓ Event pipeline / SIEM ↓ Detection ↓ Alert / response

Do not force one mechanism to solve both use cases.

---

23. Hands-on Lab 1 — Build an audit explorer

Create a SQL dashboard containing:

1. events by service;
2. events by action;
3. events by user;
4. events by workspace;
5. failed operations;
6. security-sensitive operations.

Start with the last seven days.

Add filters rather than querying the entire history.

---

24. Hands-on Lab 2 — Sensitive-data access report

Combine:

• classification results;
• audit events;
• asset inventory.

Produce:

User	Asset	Sensitive class	Action	Time	Risk
analyst1	customer.email	email	SELECT	timestamp	Medium
svc_finance	supplier.bank	bank	SELECT	timestamp	High

Then create a risk rule for restricted assets.

---

25. Hands-on Lab 3 — Governance scorecard

Build KPIs for:

• classification coverage;
• policy coverage;
• owner coverage;
• lineage coverage;
• sensitive access;
• stale assets.

Create an executive dashboard.

The dashboard should show risk and action, not merely event counts.

---

26. Hands-on Lab 4 — Incident investigation

Simulate:

A service principal accessed a restricted production table outside its normal workload window.

Investigate:

1. audit event;
2. identity;
3. run-as identity;
4. workspace;
5. target object;
6. sensitive classification;
7. lineage;
8. historical usage;
9. policy;
10. remediation.

Write a one-page incident timeline.

---

27. Hands-on Lab 5 — Governance retention

Design a seven-year audit architecture.

Decide:

• what remains in Databricks;
• what is archived;
• encryption;
• access control;
• retention;
• legal hold;
• retrieval;
• deletion policy.

Document the architecture decision.

---

28. Troubleshooting

Audit event missing

Check:

• event generation time;
• event_date;
• region;
• workspace;
• service name;
• action name;
• system-table availability;
• permissions.

Audit system-table data is not real-time, so a recently generated event may not immediately appear.

User cannot query system tables

Check:

• USE CATALOG system;
• USE SCHEMA;
• SELECT;
• query compute access;
• workspace/region.

Account-level event appears without workspace

Check whether the event is account-scoped.

For relevant Unity Catalog and lineage events, inspect request_params.workspace_id.

Lineage and audit disagree

They answer different questions.

Audit records an action.

Lineage records inferred data relationships.

Lineage is not guaranteed for every operation; Databricks documents several lineage limitations.

---

29. Common anti-patterns

1. Treating audit logs as SIEM

Audit analytics are not automatically a real-time security detector.

2. Giving everyone raw system-table access

Governance metadata can itself contain sensitive operational information.

3. Keeping only user email

Service principals, run-as identities and delegated execution require richer identity modeling.

4. Building dashboards directly on raw events

Curate and aggregate first.

5. Ignoring region

System tables contain regional/global distinctions.

6. Ignoring retention

Default retention may not satisfy enterprise requirements.

7. Alerting on everything

High-volume low-value alerts create security fatigue.

8. No evidence chain

Every alert should preserve source event IDs and timestamps.

---

30. Architect decision framework

Need to answer "who did what?"

Use audit.

Need to answer "where did the data go?"

Use lineage.

Need to answer "what data is sensitive?"

Use classification.

Need to answer "who should see it?"

Use RBAC/ABAC/row filters/masks.

Need to answer "is governance working?"

Use governance metrics.

Need long-term compliance evidence?

Use a controlled retention/archive architecture.

Need real-time threat detection?

Use an event/SIEM lane in addition to historical system-table analytics.

---

31. Enterprise governance operating model

A mature organization creates three layers.

Layer 1 — Platform controls

• Unity Catalog;
• privileges;
• ABAC;
• classification;
• lineage;
• audit.

Layer 2 — Governance analytics

• asset inventory;
• sensitive-data inventory;
• access mart;
• lineage mart;
• policy coverage;
• risk scores.

Layer 3 — Organizational response

• security operations;
• data stewards;
• compliance;
• data owners;
• platform engineering.

This prevents the platform from becoming a collection of disconnected features.

---

32. Interview questions

What is the audit system table?

A Unity Catalog system table containing audit events for Databricks activity.

What is the difference between audit and lineage?

Audit records actions; lineage records data relationships and flow.

Why are system tables important?

They provide queryable historical operational data for security, cost, performance and governance analytics.

How do you monitor sensitive-data access?

Join sensitive-data classification/inventory with audit events and identity metadata, then apply risk rules.

How do you handle seven-year compliance retention?

Use Databricks system tables for operational history and a controlled long-term archive for the required retention period.

Why are run_by and run_as important?

They distinguish the identity initiating an operation from the identity used for authorization in workload scenarios.

Can audit replace a SIEM?

Not generally. Audit system tables are excellent for historical analysis but are not a real-time monitoring mechanism.

---

33. Production checklist

• System-table access is explicitly governed.
• Audit events are normalized.
• User and service-principal identities are modeled.
• run_by and run_as are preserved where applicable.
• Workspace attribution is normalized.
• Sensitive-data classification is joined to access data.
• Lineage is available for impact analysis.
• Security-sensitive events have risk tiers.
• Dashboards use curated governance marts.
• Raw system-table access is restricted.
• Retention requirements are documented.
• Archive architecture exists for long-term compliance.
• Real-time security monitoring is separated from historical audit analytics.
• Every alert preserves source event IDs.
• Governance KPIs have accountable owners.
• Audit gaps and lineage limitations are documented.

---

34. Final mental model

Remember the complete governance loop:

CONTROL ↓ CLASSIFY ↓ ACCESS ↓ AUDIT ↓ LINEAGE ↓ MEASURE ↓ DETECT ↓ RESPOND ↓ IMPROVE

The architect's key principle is:

Governance is not complete when access is configured; it is complete when access can be observed, explained, measured and acted upon.

That is the difference between a Databricks security configuration and an enterprise governance operating model.