02.2 Unity Catalog Architecture
Phase 2 — Platform Architecture
Unity Catalog (UC) is the unified governance layer for Databricks data and AI assets. It provides a common object model, permissions, lineage, auditing and fine-grained access controls across workspaces attached to a metastore. citeturn609970search0
1. The architecture mental model
ACCOUNT
│
├── WORKSPACES → operational boundary
│
└── UNITY CATALOG → governance boundary
│
METASTORE
│
CATALOG → SCHEMA → OBJECT
│
TABLE / VIEW / VOLUME
│
GOVERNED STORAGE
Think of the five layers as:
| Layer | Architectural responsibility |
|---|---|
| Account | Enterprise administration and identity |
| Workspace | Collaboration, workload and operational boundary |
| Metastore | Regional Unity Catalog governance container |
| Catalog / Schema / Object | Data organization and authorization hierarchy |
| Cloud storage | Durable physical data layer |
2. Metastore
A Unity Catalog metastore is the top-level regional container for metadata and governance. It stores metadata and permissions for securable objects such as catalogs, tables, volumes, external locations and storage credentials. Current Databricks architecture guidance describes one metastore per region and requires each workspace to be assigned to a metastore. citeturn609970search2turn609970search8
REGIONAL METASTORE
│
┌─────────────┼─────────────┐
▼ ▼ ▼
DEV TEST PROD
workspace workspace workspace
The key enterprise benefit is centralized governance across multiple operational workspaces.
3. Catalog
A catalog is the primary unit of data organization and a major governance boundary in Unity Catalog. It contains schemas and is a good place to express domain, environment or access-isolation intent. citeturn609970search14
Example:
finance
├── raw
├── curated
└── reporting
sales
├── raw
├── curated
└── reporting
Architect question:
What meaningful data-isolation or governance boundary does this catalog represent?
Do not create a catalog for every person or small team. Use catalogs when the access, ownership, lifecycle or compliance boundary is meaningful.
4. Schema
A schema is the second layer below a catalog and groups tables, views, volumes and other objects.
finance
└── gold
├── gl_revenue
├── gl_cost
└── gl_margin
Schema boundaries are useful for lifecycle, ownership, workload and privilege organization.
5. Three-level namespace
The core data namespace is:
catalog.schema.object
Example:
SELECT * FROM finance.gold.gl_revenue;
This hierarchy is also the foundation for scalable authorization. Unity Catalog represents governed assets as securable objects on which privileges can be granted. citeturn609970search0turn609970search9
6. Securable objects
Common securable objects include:
- Metastore
- Catalog
- Schema
- Table
- View
- Volume
- Function
- Model
- External location
- Storage credential
- Connection and sharing objects
The security relationship is:
Principal → Privilege → Securable Object
Principals can include users, groups and service principals.
7. Ownership and privileges
Every securable object has an owner. Owners have the capabilities needed to manage the object and can delegate management through supported privileges. citeturn609970search1turn609970search11
Typical privileges include:
USE CATALOG
USE SCHEMA
SELECT
MODIFY
CREATE TABLE
CREATE VOLUME
MANAGE
READ METADATA
Do not confuse ownership, data privileges, workspace access and cloud IAM. They are different control layers.
8. Privilege inheritance
Unity Catalog's hierarchical model allows privileges to be organized at higher levels and inherited by supported child objects.
Catalog
│
└── Schema
│
└── Table
Architect rule:
Design the object hierarchy first; design the grants second.
A clean catalog/schema structure can remove hundreds of unnecessary table-specific grants.
9. Managed vs external assets
Unity Catalog distinguishes managed and external storage patterns. Managed objects are governed and have their storage lifecycle managed by Databricks. External objects remain in customer-managed cloud storage while Unity Catalog provides governance. citeturn609970search0turn609970search7
MANAGED
Unity Catalog → Managed table → UC-managed storage
EXTERNAL
Unity Catalog → External table → External location → S3
Prefer managed storage when there is no concrete requirement to control the physical storage path yourself. Use external objects when existing storage, direct file interoperability, residency or other requirements make physical ownership important.
10. Storage credentials
A storage credential is a Unity Catalog securable object used to authenticate to customer-managed cloud storage. It prevents developers from needing long-lived cloud credentials for ordinary data access. citeturn609970search2
On AWS, the common architecture is:
Databricks
│
▼
Storage Credential
│
▼
AWS IAM Role
│
▼
S3
Unity Catalog and AWS IAM therefore work together rather than replacing one another.
11. External locations
An external location combines a cloud storage path with a storage credential. citeturn609970search13
External Location
├── s3://enterprise-data-prod/finance/
└── finance-prod IAM-backed credential
Good external-location boundaries usually correspond to meaningful catalog, domain or environment boundaries. Current Databricks guidance recommends avoiding unnecessary proliferation and using the highest useful common path prefix. citeturn609970search2
12. Workspace bindings
Workspace-catalog binding restricts a catalog to specific workspaces. This is important when a user may have a privilege on an object but that object must still be unavailable from a particular workspace. citeturn609970search3
METASTORE
│
┌─────────┴─────────┐
▼ ▼
finance_prod sandbox
│
PROD only
Workspace binding can also apply to external locations and storage credentials.
Think:
WHO? → privileges and policies
WHERE? → workspace bindings
PHYSICAL ACCESS? → storage credentials + cloud IAM
13. Fine-grained security
Unity Catalog provides complementary access-control mechanisms:
| Mechanism | Best use |
|---|---|
| Privileges | Baseline object access and delegation |
| ABAC | Centralized tag-driven policies at scale |
| Row filters | Table-specific row-level restrictions |
| Column masks | Table-specific sensitive-column protection |
| Workspace bindings | Restrict where governed objects can be used |
Databricks currently recommends ABAC for consistent, scalable tag-driven policies; row filters and column masks remain useful for table-specific logic. citeturn609970search4turn609970search10
Example: row-level security
User = SOUTH_REGION
│
▼
Row filter
│
▼
Only SOUTH rows visible
Example: column masking
customer_email
│
├── authorized user → real value
└── other user → masked value
14. Unity Catalog + workspace architecture
Connect this topic to 02.1:
ACCOUNT
│
┌──────────┴──────────┐
▼ ▼
DEV WORKSPACE PROD WORKSPACE
│ │
└──────────┬──────────┘
▼
METASTORE
│
┌──────────┼──────────┐
▼ ▼ ▼
sandbox finance sales
The workspace answers where the workload runs. Unity Catalog answers what governed assets exist, who can use them and under what rules.
15. Multi-workspace governance
A major enterprise advantage is separating operational workspaces from centralized data governance.
UC METASTORE
│
┌──────────────┼──────────────┐
▼ ▼ ▼
DEV TEST PROD
│ │ │
└──────────────┼──────────────┘
│
shared governance
Shared metastore does not mean unrestricted access. Privileges, ownership, ABAC, filters, masks, bindings and storage controls still apply.
16. Data lineage
Unity Catalog also participates in lineage tracking, allowing teams to understand upstream and downstream relationships between data and AI assets. citeturn609970search0
Source → Bronze → Silver → Gold → BI / ML
Architecturally, lineage is useful for impact analysis, troubleshooting, governance and change management.
17. Regional enterprise architecture
For a multinational AWS deployment:
ACCOUNT
│
┌──────────────┴──────────────┐
│ │
INDIA REGION EUROPE REGION
│ │
UC METASTORE UC METASTORE
│ │
DEV / PROD DEV / PROD
│ │
domain catalogs domain catalogs
Current Databricks guidance treats the metastore as regional and recommends designing governance, storage and workspace boundaries around residency, security and operating-model requirements. citeturn609970search2
18. Governance operating models
Centralized
Central Governance
│
┌─────┼─────┐
Fin Sales HR
Federated
Central Platform
│
┌─────┼─────┐
▼ ▼ ▼
Fin Sales HR
│ │ │
Own Own Own
data data data
Hybrid
The platform team owns identity, security standards, automation and guardrails while domain teams own their data products. This is often the most practical enterprise operating model.
19. Enterprise catalog strategy
Common patterns include:
Environment: dev / test / prod
Domain: finance / sales / procurement
Hybrid: finance_prod / finance_dev / sales_prod
Choose based on access boundaries, ownership, compliance, SDLC and cross-domain sharing. There is no universal naming scheme.
20. Architecture anti-patterns
Anti-pattern 1 — One catalog per person
Creates administrative noise and weak governance boundaries.
Anti-pattern 2 — One workspace per team
Use workspaces for meaningful operational isolation, not folders.
Anti-pattern 3 — External location for everything
Managed storage is simpler when physical-path control is not required.
Anti-pattern 4 — Direct cloud credentials for developers
Use governed storage credentials and IAM delegation.
Anti-pattern 5 — Unity Catalog replaces IAM
It does not. Cloud authorization remains a separate layer.
Anti-pattern 6 — Permissions designed before hierarchy
Design catalogs and schemas around governance first.
21. Hands-on implementation
Run this only in a development environment where you have the required privileges.
Create a catalog
CREATE CATALOG IF NOT EXISTS training;
Create schemas
CREATE SCHEMA IF NOT EXISTS training.bronze;
CREATE SCHEMA IF NOT EXISTS training.silver;
CREATE SCHEMA IF NOT EXISTS training.gold;
Create a managed table
CREATE TABLE IF NOT EXISTS training.gold.customer_sales (
customer_id BIGINT,
sales_amount DECIMAL(18,2),
region STRING
);
Inspect the hierarchy
SHOW CATALOGS;
SHOW SCHEMAS IN training;
SHOW TABLES IN training.gold;
Grant access
GRANT USE CATALOG ON CATALOG training TO data_analysts;
GRANT USE SCHEMA ON SCHEMA training.gold TO data_analysts;
GRANT SELECT ON TABLE training.gold.customer_sales TO data_analysts;
Inspect grants
SHOW GRANTS ON CATALOG training;
SHOW GRANTS ON SCHEMA training.gold;
SHOW GRANTS ON TABLE training.gold.customer_sales;
Your resulting model:
Metastore
│
training
│
├── bronze
├── silver
└── gold
└── customer_sales
│
SELECT
│
data_analysts
22. Architect checklist
Before approving a Unity Catalog design, ask:
- What is the governance operating model?
- What does each catalog represent?
- Which schemas represent meaningful ownership or lifecycle boundaries?
- Who owns each data domain?
- Which workspaces can access production catalogs?
- Which data is managed and which must remain external?
- What storage credentials exist?
- What external locations exist and why?
- Where is ABAC required?
- Where are row filters or column masks required?
- How does Unity Catalog interact with AWS IAM?
- How are grants deployed and audited?
- How will onboarding be automated?
23. Interview questions
Q1. What is Unity Catalog?
A unified governance layer for Databricks data and AI assets providing a common object model, access control, lineage and auditing.
Q2. What is the three-level namespace?
catalog.schema.object.
Q3. What is a metastore?
The top-level regional Unity Catalog governance container.
Q4. Managed vs external table?
Managed objects have storage lifecycle managed by Databricks; external objects remain in customer-managed storage while Unity Catalog governs them.
Q5. How do you isolate production data from development?
Use workspace isolation where operational separation is required, then enforce Unity Catalog privileges, workspace bindings, storage controls and environment-specific credentials.
Q6. ABAC vs row filters?
ABAC is preferred for consistent centralized tag-driven policies across many objects. Row filters and column masks are useful when the logic is table-specific. citeturn609970search4turn609970search10
Q7. Does Unity Catalog replace AWS IAM?
No. Unity Catalog governs Databricks access to governed assets; AWS IAM controls cloud-resource authorization for the delegated identity.
24. Architect mental model
Memorize:
ACCOUNT
│
├── WORKSPACES
│ └── COMPUTE
│
└── UNITY CATALOG
└── METASTORE
└── CATALOG
└── SCHEMA
└── TABLE / VIEW / VOLUME
└── CLOUD STORAGE
And the security chain:
WHO? Identity / group / service principal
WHAT? Privileges / ABAC / row / column controls
WHERE? Workspace binding
PHYSICAL ACCESS? Storage credential / cloud IAM
The enterprise architect does not start with GRANT SELECT. The architect starts by defining the governance boundaries, then chooses the object hierarchy and access mechanisms that enforce them.
What you should be able to explain now
- Metastore architecture
- Catalog vs schema
- Three-level namespace
- Securable objects
- Ownership and privileges
- Managed vs external assets
- Storage credentials
- External locations
- Workspace bindings
- ABAC, row filters and column masks
- Lineage
- Regional and multi-workspace governance
- Unity Catalog + AWS IAM
- Centralized, federated and hybrid governance
- Production catalog strategy
- Common architecture anti-patterns
Official references
- Databricks — What is Unity Catalog
- Databricks — Design Unity Catalog architecture
- Databricks — Permissions model
- Databricks — Manage privileges
- Databricks — Workspace-catalog binding
- Databricks — Row filters and column masks
- Databricks — Storage architecture
Next
02.3 Storage Architecture — managed storage, external storage, S3 integration, storage credentials, external locations, catalog/schema storage isolation, data layout, cross-account access and enterprise storage patterns.