07.15 External Data Governance & Foreign Catalog Security Patterns

A foreign catalog makes an external system look native inside Unity Catalog, but it does not make the external system disappear.

The enterprise security model therefore has two layers:

Source system security + Unity Catalog governance

Effective federated security

Databricks documents a foreign catalog as a Unity Catalog securable object that mirrors an external database and provides read-only querying with Unity Catalog access controls.

1. The five security boundaries

Treat these as separate controls:

1. Identity — who is the Databricks user or service principal?
2. Unity Catalog — which foreign catalog/schema/table may they see and query?
3. Connection — who may create or inspect the external connection?
4. Network — which compute resources may reach the source?
5. Source database — what identity and privileges exist in the external system?

A failure or over-permission at any layer changes the effective security posture.

2. Foreign catalog does not mean data ownership

A foreign catalog is a governed mirror of external metadata.

Conceptually:

External database | Connection | Foreign catalog | Unity Catalog ACLs | Consumer

The data remains in the external system.

This is fundamentally different from:

External source | Ingestion | Delta table

where Databricks owns the analytical copy.

3. Connection is a security boundary

A Lakehouse Federation connection is a Unity Catalog securable object containing the path and credentials needed to reach the external system.

Do not treat the connection as a shared technical detail.

Treat it as a privileged security object.

Recommended ownership:

Platform / Data Platform | Connection owner | Domain access | Foreign catalog

This prevents every data engineer from creating independent source identities.

4. Separate connection privileges from data privileges

This distinction is critical.

A user may need access to a foreign table without being allowed to administer the connection.

Databricks exposes connection privileges including:

• CREATE FOREIGN CATALOG
• USE CONNECTION
• MANAGE
• READ METADATA

The exact privilege required depends on the operation.

Architectural principle:

Data access should not imply connection administration.

5. CREATE FOREIGN CATALOG vs USE CONNECTION

These privileges solve different problems.

CREATE FOREIGN CATALOG

Allows a principal to create foreign catalogs using the connection.

USE CONNECTION

Allows a principal to use/view the connection according to the documented semantics.

Do not grant both automatically.

Grant only the privilege required by the operating model.

6. Foreign catalog permissions

Once the foreign catalog exists, govern it like a Unity Catalog data asset.

Example:

foreign_catalog | schema | tables

Apply:

• catalog access;
• schema access;
• table SELECT;
• ownership;
• BROWSE where appropriate;
• tags and metadata governance.

Databricks supports table-level access controls for foreign catalog data.

7. The access chain

For a user to query a foreign table, reason through:

User / Group | Workspace access | Catalog privilege | Schema privilege | Table privilege | Foreign connection | External database authorization | Data

A grant at one layer does not automatically mean the user has permission at every other layer.

8. Dedicated compute nuance

Compute mode matters.

Databricks documents an important distinction: on dedicated access-mode compute, querying a foreign catalog requires ownership of the underlying connection; CREATE FOREIGN CATALOG alone is not sufficient. Standard access mode and serverless use standard catalog privilege semantics.

Architect lesson:

Always include compute access mode in a federation security test.

9. Source-side authorization

Unity Catalog does not replace source authorization.

The external database still has its own identity and permissions.

For example:

Databricks group | Foreign catalog SELECT | Connection credential | Source database user | Source SELECT

If the source credential has broad privileges, the source side may still expose more data than intended.

10. The credential blast-radius problem

Imagine one connection uses a source account with:

SELECT on every schema

while Unity Catalog grants users access to only one foreign schema.

The user may be governed at the Databricks layer, but the underlying credential still has a large blast radius.

Therefore:

• minimize source credentials;
• separate sensitive domains;
• use read-only identities;
• isolate high-risk systems;
• rotate credentials centrally.

11. One connection per domain?

There is no universal answer.

Shared connection

Pros:

• fewer objects;
• easier operations;
• centralized credential rotation.

Cons:

• larger credential blast radius;
• harder source-side isolation.

Domain-specific connections

Pros:

• smaller blast radius;
• stronger source isolation;
• easier domain accountability.

Cons:

• more operational objects;
• more credential lifecycle management.

Choose based on:

sensitivity + source authorization model + number of domains + operational maturity

12. High-sensitivity pattern

For highly sensitive sources:

HR Database | HR read-only identity | HR connection | HR foreign catalog | HR security group

Avoid:

Enterprise DB | One super-user connection | Every domain

13. Data minimization pattern

Do not expose the complete operational schema when consumers need only a subset.

Prefer a source-controlled curated interface where possible:

Operational database | curated view | connection | foreign catalog | consumers

This reduces accidental discovery and unnecessary source load.

14. Sensitive columns

Foreign catalog access is not a substitute for masking requirements.

For sensitive attributes such as:

• national identifiers;
• bank information;
• employee data;
• customer contact data;

design an explicit protection model.

Possible strategies include:

1. source-side protected views;
2. Unity Catalog column masking where supported;
3. row filters where supported;
4. separate sensitive and non-sensitive interfaces;
5. tokenization before federation.

The correct choice depends on where policy must be enforced.

15. Row-level security architecture

Consider supplier data:

supplier_region

IN EU US

The security requirement may be:

India users → IN rows Europe users → EU rows US users → US rows

Do not assume a source-wide connection automatically provides this isolation.

Model row-level policy explicitly and test it with real identities.

16. Column masking architecture

A common pattern:

customer_email | authorized role → full value normal role → masked value

Security design should answer:

• where masking occurs;
• which identity evaluates the policy;
• whether the source can bypass it;
• whether exports inherit the protection;
• whether downstream tables preserve the policy.

17. ABAC and tags

For larger estates, attribute-based access control can reduce the number of individual grants.

Conceptually:

Data tag + User attribute | Policy | Access decision

Use governed tags to classify sensitive assets and align access policies with business attributes.

Do not create hundreds of manually maintained table grants when a stable policy can express the same rule.

18. External metadata is still metadata

A foreign catalog may expose:

• table names;
• schema names;
• column names;
• comments;
• data types;
• lineage context.

Metadata can itself be sensitive.

Decide:

• who can browse;
• who can discover;
• who can query;
• who can administer.

These are separate questions.

19. BROWSE vs data access

A user may need to discover that an asset exists without being able to query its contents.

That leads to a useful governance pattern:

Discovery ↓ Metadata ↓ Request ↓ Approval ↓ Data access

Use BROWSE and catalog discovery capabilities deliberately rather than equating discoverability with SELECT.

20. Workspace bindings

In multi-workspace environments, control which workspaces can access sensitive catalogs and supporting objects.

Example:

Production workspace | Finance foreign catalog

while:

Development workspace | denied

This reduces accidental access from lower-trust environments.

21. Development vs production

A strong enterprise pattern:

DEV | masked / synthetic source

TEST | controlled subset

PROD | governed foreign catalog

Avoid pointing development notebooks at unrestricted production operational data.

22. Service principals

Machine-to-machine access should use service principals rather than personal credentials.

Typical pattern:

Job / BI / automation | service principal | Unity Catalog | foreign catalog | external source

This improves:

• auditability;
• rotation;
• lifecycle management;
• offboarding.

23. Credential rotation

Credential rotation should be:

centrally owned | automated | tested in lower env | production | monitored

The application should not contain the password.

Databricks recommends secure credential handling for connection options rather than embedding credentials directly in SQL.

24. Network policy is separate from connection policy

This is an easy architectural mistake.

Unity Catalog connections govern credentials and connection configuration.

Network policies govern whether outbound traffic is allowed.

Databricks explicitly distinguishes these layers.

Therefore:

Connection exists ≠ Network path is permitted

Both must be configured.

25. Audit model

A production federation security model should answer:

• who accessed the catalog?
• who queried the table?
• when?
• from which workspace?
• using which identity?
• what source was involved?
• did the query succeed?
• did the source reject it?

Correlate Databricks audit/query information with source database audit logs where possible.

26. Security incident investigation

Example:

Sensitive table accessed | Databricks audit | User identity | Foreign catalog | Connection | Source identity | Source audit | Network logs

The goal is end-to-end attribution.

27. Foreign catalog vs ingestion security

Federation

Source data | Remote query | Consumer

Security is primarily about:

• access;
• network;
• credentials;
• source authorization;
• query controls.

Ingestion

Source | Pipeline | Delta | Consumer

Security additionally includes:

• landing zone;
• copied data;
• storage;
• checkpoints;
• historical retention;
• downstream sharing.

Federation can reduce data-copy footprint but increases runtime dependency on the source.

28. Foreign catalog vs JDBC

JDBC gives application-level connectivity and can support read/write patterns.

Lakehouse Federation provides a governed foreign-catalog abstraction with Unity Catalog integration.

Databricks currently distinguishes query federation, catalog federation and JDBC connections as different connection types with different governance and read/write characteristics.

For governed analytical access, prefer the abstraction that matches the requirement rather than embedding JDBC connection logic in notebooks.

29. Security decision matrix

Requirement	Preferred pattern
Read-only live analytics	Foreign catalog
Strong table-level governance	Foreign catalog + UC privileges
Highly sensitive source	Dedicated connection + least privilege
Source must remain private	Private connectivity
Large recurring analytical workload	Ingest to Delta
Need historical ownership	Delta
Need source-native writes	JDBC/application integration
Development access	Masked/synthetic data
Mission-critical live + fallback	Federation + Delta

30. Hands-on Lab 1 — Build a least-privilege model

Create:

• platform-admin group;
• finance-analyst group;
• finance-reader group;
• service principal.

Design permissions for:

connection foreign catalog schema table

Prove that the analyst can query approved data but cannot administer the connection.

31. Hands-on Lab 2 — Blast-radius analysis

Take one shared connection.

Document the source privileges of its credential.

Then answer:

• what is the maximum data a compromised credential can read?
• which catalogs depend on it?
• which domains share it?
• what happens if the credential leaks?

Redesign the connection boundaries.

32. Hands-on Lab 3 — Environment isolation

Create a security design for:

DEV TEST PROD

Use different identities and data exposure levels.

Prove that development cannot access unrestricted production data.

33. Hands-on Lab 4 — Sensitive-data policy

Choose a customer table containing:

customer_id email phone region

Design:

• classification;
• row policy;
• masking;
• groups;
• service principal access;
• audit requirements.

34. Hands-on Lab 5 — Incident investigation

Simulate:

A user accessed a sensitive foreign table unexpectedly.

Trace:

identity ↓ UC permission ↓ catalog ↓ connection ↓ source identity ↓ source audit

Produce a root-cause report.

35. Common mistakes

Mistake 1 — One super-user source credential

Huge blast radius.

Mistake 2 — Giving analysts connection administration

Data consumers rarely need connection management.

Mistake 3 — Assuming UC replaces source security

It does not.

Mistake 4 — Ignoring metadata exposure

Table and column names can be sensitive.

Mistake 5 — Treating network access as authorization

Network reachability is not data permission.

Mistake 6 — Production source in development

Creates unnecessary compliance and operational risk.

Mistake 7 — No credential rotation

Creates long-lived security exposure.

Mistake 8 — No source-side audit correlation

Leaves investigations incomplete.

36. Enterprise reference architecture

+-----------------------------------------------------+ | Unity Catalog | | | | Identity → Catalog → Schema → Table | | | | | | | +---------- Policy / Tags ----------------+ | | | | | Foreign Catalog | | +--------------------------|----------------------+ | | Connection | Network Policy | Private Path | +---------v----------+ | Read-only Source | | Identity | +---------+----------+ | External DB | Source ACLs | Source Audit

The security boundary is intentionally layered.

37. Architect mental model

Remember:

UC governs the Databricks access plane. Source DB governs the source access plane. Network controls reachability. Connection controls how the systems authenticate. Audit connects the evidence.

Therefore:

A secure foreign catalog is not one permission. It is a chain of controls.

38. Production checklist

• Connection owner defined.
• Source credential owner defined.
• Source credential is least privilege.
• CREATE FOREIGN CATALOG restricted.
• USE CONNECTION restricted.
• Foreign catalog access reviewed.
• Schema/table grants reviewed.
• Sensitive data classified.
• Row policies evaluated.
• Column masking evaluated.
• Metadata discovery reviewed.
• Workspace bindings reviewed.
• DEV/TEST/PROD isolation validated.
• Service principals used for automation.
• Credential rotation tested.
• Network policy documented separately.
• Source audit correlation implemented.
• Incident runbook tested.
• Blast radius documented.
• Federation vs ingestion decision recorded.

39. Interview questions

Does a foreign catalog copy the source data?

No. It provides a governed read-only mirror of the external database metadata and queries the source.

Does Unity Catalog replace source authorization?

No. Source-side authorization remains part of the security boundary.

Why separate connection privileges from table privileges?

Because connection administration and data consumption are different responsibilities.

Why is a shared super-user connection dangerous?

A compromised credential can potentially expose every source object available to that identity.

Can users browse metadata without querying data?

Governance can separate discovery/metadata access from data access; use discovery privileges deliberately.

Why is network policy separate from connection security?

A valid connection object does not automatically authorize network traffic.

What is the strongest pattern for highly sensitive data?

Dedicated least-privilege source identity, tightly governed foreign catalog, private connectivity where required, explicit data policies and end-to-end auditing.

When should you stop federating?

When source load, recurring workload, resilience, ownership or cost requirements make a Delta copy the better architecture.

40. Final rule

The one-line enterprise rule is:

Never evaluate foreign-catalog security at only the Databricks layer; evaluate identity, Unity Catalog, connection, network and source authorization as one security chain.