Skip to main content

07.14 Lakehouse Federation — Advanced Networking, Security & Performance

07.13 established the core mental model:

Federation is an access pattern; Delta ingestion is an ownership pattern.

This module takes that idea to enterprise-architect level. The difficult part of federation is rarely creating the foreign catalog. The difficult part is making the architecture secure, predictable, performant, observable and supportable when the source is on-premises, behind a firewall, in another cloud, protected by private connectivity, using OAuth, or subject to strict source-system capacity limits.

Official references:

Advanced Lakehouse Federation architecture

1. The enterprise federation problem​

A simple proof of concept looks like:

Databricks | Internet | PostgreSQL

An enterprise design looks more like:

User | Unity Catalog | Foreign Catalog | Databricks SQL / Serverless | Egress / Private connectivity | Firewall / Network boundary | External DB | Source authorization | Source workload controls

Every layer can fail independently.

The architect therefore needs to reason about identity, authorization, network reachability, authentication, query pushdown, source capacity, data transfer, observability, failure handling and migration strategy.

2. The most important network rule​

For Lakehouse Federation, the federated query data path is between Databricks compute and the external database.

Unity Catalog and the Databricks control plane are not normally on that data path.

Databricks Compute | network | External Database

Do not design the firewall assuming every query passes through the control plane.

Networking recommendations

3. OAuth creates a second network path​

OAuth changes the architecture.

Data path:

Databricks Compute | v External Database

OAuth token path:

Databricks Control Plane | v OAuth Authorization Server

Therefore, allowlisting only the compute network can still produce authentication failures.

4. Network topology options​

Public endpoint with allowlist​

Databricks | Internet | Firewall | Database

Use when public connectivity is acceptable and the source supports reliable allowlisting.

Private connectivity​

Databricks | Private connectivity | VPC / VNet | Database

Use when compliance, sensitivity or network policy requires private paths. Validate connector-specific capabilities before implementation.

On-premises database​

Databricks cloud | Enterprise network | VPN / private route | Data center | Database

The key question is whether the Databricks compute plane has a stable and permitted route to the source.

5. Stable egress​

For classic compute, a stable egress design can provide a predictable source IP.

Databricks subnet | NAT / gateway | Stable public IP | Source firewall

For serverless compute, Databricks publishes outbound IP information for supported firewall patterns. Production allowlists should be automated rather than manually copied once.

Serverless firewall configuration

6. Serverless private connectivity​

Serverless compute does not use VPC peering in the same way classic customer-managed compute does.

When private connectivity is required, use supported serverless private endpoints / Private Link patterns.

Serverless | Private endpoint | VPC resource | External service

Serverless compute networking

7. DNS is part of the architecture​

A network path can exist while the connection still fails because DNS is wrong.

Check the complete chain:

DNS → route → firewall → TLS → authentication → database authorization.

For customer-managed networks with custom DNS, the external database hostname must resolve correctly.

8. TLS and certificates​

Production federation should use encrypted transport.

Validate:

  • TLS version;
  • server certificate;
  • certificate chain;
  • hostname verification;
  • trusted CA;
  • certificate expiration;
  • mutual TLS requirements where supported.

A successful TCP connection does not prove that TLS and authentication are correct.

9. Authentication vs authorization​

Think of authentication separately from authorization.

Authentication asks: Who are you?

Authorization asks: What may you access?

Architecture:

Identity | Authentication | Connection | External DB identity | Source authorization

Independently:

Databricks user/group | Unity Catalog privileges | Foreign catalog access

Both control planes must be documented.

10. Connection ownership​

A Unity Catalog connection is a securable object.

Do not let every analyst create connections.

A stronger pattern is:

Platform team | Owns connection | Controlled privilege | Domain teams | Foreign catalog access

Databricks documents connection privileges such as CREATE FOREIGN CATALOG and USE CONNECTION.

Connection permissions

11. Credential architecture​

Never embed credentials in notebooks.

Prefer:

  • centralized connection objects;
  • supported OAuth mechanisms;
  • managed identity patterns where supported;
  • secure secret handling;
  • short-lived credentials where available;
  • rotation policies.

Credential rotation should not require editing hundreds of notebooks.

12. Query execution architecture​

A federated query is a distributed execution problem:

SQL | Databricks optimizer | Pushdown analysis | Remote SQL | External optimizer | Source execution | Result transfer | Databricks processing | Final result

Some operations can execute remotely while others may execute in Databricks. Exact pushdown capability depends on the connector.

13. Pushdown is the first performance lever​

Prefer:

Filter ↓ Projection ↓ Aggregation ↓ Result transfer

over:

Full remote scan ↓ Massive transfer ↓ Databricks filtering

Example:

SELECT supplier_id, SUM(spend) FROM procurement.orders WHERE fiscal_year = 2026 GROUP BY supplier_id;

The objective is to reduce bytes crossing the network.

14. Why SELECT * is dangerous​

Consider a multi-terabyte source table queried with SELECT * when the final report needs only a few columns.

Prefer explicit projection:

SELECT supplier_id, order_date, amount FROM procurement.orders;

Column pruning is both a performance and security practice.

15. Fetch size and round trips​

JDBC-based connectors can fetch rows in batches.

A fetch size that is too small can create excessive round trips. A fetch size that is too large can increase memory pressure.

Databricks documents fetchSize for several JDBC-based connectors. A large value such as 100,000 can be a useful benchmark starting point for supported environments, but it should be tested against row width, source capacity and network characteristics.

Performance recommendations

16. Source database capacity is a hard boundary​

Suppose Databricks can run hundreds of concurrent queries while the Oracle database can safely support only a fraction of them.

The real system capacity is approximately:

min( Databricks capacity, network capacity, source database capacity )

Federation couples the analytics workload to the source platform.

17. BI workload risk​

Federation becomes dangerous when BI tools generate many concurrent queries, repeated scans, dashboard refresh storms, high-cardinality joins or unrestricted filters.

A dashboard can become an operational database workload generator.

For recurring BI:

Source DB | Federation | BI

may eventually need to become:

Source DB | Ingestion | Delta | SQL Warehouse | BI

18. Cross-system joins​

A difficult pattern is:

Oracle customers + Snowflake transactions + PostgreSQL products | JOIN

The architect must determine where the data should meet.

For large recurring joins, a common analytical storage layer is usually more predictable than repeated remote joins.

Federation is strongest for selective access, not as a replacement for distributed analytical storage.

19. Source-side performance​

Federated performance is partly controlled by the external database.

Inspect:

  • indexes;
  • partitioning;
  • clustering;
  • statistics;
  • materialized views;
  • source query plans.

If the source query plan is poor, Databricks cannot magically fix the remote database.

20. Query-plan thinking​

For every slow query ask:

  1. What executed remotely?
  2. What executed in Databricks?
  3. How many rows were read remotely?
  4. How many rows crossed the network?
  5. Where did the largest join occur?
  6. Where did aggregation occur?
  7. Did the filter push down?
  8. Did the source use an efficient access path?

Do not optimize only the Databricks side.

21. Observability architecture​

Create a federation monitoring layer:

Query History | Federation Metrics | +----+-----------+ | | Databricks Source DB metrics metrics | | +-------+--------+ | Governance Mart | Capacity / Risk

Track query duration, source execution time, bytes transferred, row counts, failures, timeouts, connection errors, concurrency, top users, top tables, source CPU and source wait events.

22. Security architecture​

Federation has at least four security boundaries:

Identity ↓ Unity Catalog ↓ Network ↓ Source database

A secure Unity Catalog configuration cannot compensate for an exposed source credential.

Likewise, a private database cannot compensate for overly broad Unity Catalog privileges.

23. Data minimization​

Apply minimization before data leaves the source.

Prefer:

Curated view ↓ Foreign catalog ↓ Consumer

over:

Complete operational schema ↓ Foreign catalog ↓ Every analyst

This improves security, performance and governance.

24. Source workload isolation​

If possible, expose a reporting replica or read-only analytical replica.

Primary DB | Replication | Read Replica | Federation | Databricks

This protects transactional workloads while preserving near-live access.

25. Network segmentation​

A strong enterprise topology is:

Users | Databricks | Analytics egress | Network security boundary | Federation endpoint | Read replica | Source network

Avoid broad east-west access between analytics and operational networks.

26. Private connectivity decision​

Use private connectivity when:

  • data is highly sensitive;
  • public exposure is prohibited;
  • compliance requires private transport;
  • source network policy demands it.

Use IP allowlisting when:

  • public connectivity is acceptable;
  • the source supports reliable allowlists;
  • operational simplicity matters more than private routing.

The choice is a risk decision.

27. Failure domains​

Federation introduces more failure domains than a local Delta table.

User ↓ Databricks ↓ Network ↓ DNS ↓ TLS ↓ Authentication ↓ External DB ↓ Source workload

A production runbook should identify the owner and test for every layer.

28. Timeout and retry strategy​

Timeouts should be intentional.

Define:

  • connection timeout;
  • query timeout where supported;
  • retry behavior;
  • concurrency limits;
  • cancellation behavior;
  • source-side workload controls.

Do not blindly increase timeouts when a query is slow. A timeout may be protecting the source system.

Retries can multiply source workload. For expensive reads, retry behavior must be designed with source capacity in mind.

29. Federation and disaster recovery​

Ask:

  • What happens if the source is unavailable?
  • Is there a replica?
  • Can the workload fail over?
  • Can critical reporting run from Delta?
  • Is historical data already ingested?
  • What are the RTO and RPO?

For critical analytics, federation should rarely be the only execution path.

30. Hybrid fallback architecture​

For critical data:

External DB | +------ Federation ------→ live access | +------ Ingestion --------→ Delta history | ↓ BI / recovery

Normal operation can use federation. If the source is unavailable, critical reporting can fall back to the lakehouse copy.

This is a powerful enterprise pattern.

31. Governance​

Use Unity Catalog for:

  • catalog/schema/table permissions;
  • ownership;
  • discovery;
  • lineage;
  • audit;
  • data classification;
  • domain ownership.

Use the source platform for:

  • source-native authorization;
  • operational controls;
  • source workload protection;
  • database auditing.

Do not try to collapse both systems into one authorization model.

32. Benchmark framework​

Before production, benchmark:

Query classes​

  • point lookup;
  • selective filter;
  • aggregation;
  • wide scan;
  • join;
  • BI dashboard;
  • concurrent workload.

Metrics​

  • p50 latency;
  • p95 latency;
  • remote CPU;
  • Databricks compute;
  • bytes transferred;
  • rows transferred;
  • source concurrency;
  • failure rate.

Test levels​

1 user ↓ 10 users ↓ 50 users ↓ expected peak

The goal is to discover the system bottleneck, not just the fastest single query.

33. Cost model​

Estimate:

Databricks compute + external DB compute + network transfer + private connectivity + operational support + failure / recovery cost

Compare with:

ingestion + Delta storage + transformation + SQL warehouse

The cheapest architecture for one query may be the most expensive architecture for thousands of daily queries.

34. Architect decision tree​

Is the source operational?​

Yes → prefer selective federation or a read replica.

Is workload recurring and high volume?​

Yes → evaluate ingestion.

Is latency near-real-time?​

Yes → federation may be attractive.

Is source capacity limited?​

Yes → isolate or ingest.

Is data highly sensitive?​

Yes → private connectivity and minimization become stronger requirements.

Is BI generating repeated scans?​

Yes → strongly consider Delta materialization.

Is the workload mission-critical?​

Yes → evaluate a lakehouse fallback.

35. Hands-on Lab 1 — Network design​

Choose an external PostgreSQL database.

Document:

  • Databricks compute type;
  • source location;
  • DNS;
  • routing;
  • firewall;
  • TLS;
  • authentication;
  • source authorization;
  • private vs public connectivity.

Produce a one-page network diagram.

36. Hands-on Lab 2 — Pushdown benchmark​

Create three queries:

  1. selective filter;
  2. aggregation;
  3. wide scan.

Measure execution time, returned rows, source execution and network transfer.

Then rewrite the queries to minimize remote data transfer.

37. Hands-on Lab 3 — BI concurrency​

Simulate 1, 10 and 50 concurrent dashboard-style queries.

Record:

  • p50;
  • p95;
  • source CPU;
  • connection count;
  • failures.

Determine the point at which federation stops being appropriate.

38. Hands-on Lab 4 — Federation vs Delta​

Build:

Source | +---- Federation | +---- Delta ingestion

Run the same workload against both.

Compare freshness, latency, cost, source impact, resilience, governance and operational complexity.

Write the final architecture decision.

39. Hands-on Lab 5 — Failure drill​

Break one layer at a time:

  • DNS;
  • firewall;
  • credentials;
  • source availability;
  • permission;
  • timeout.

For every failure document:

  1. symptom;
  2. evidence;
  3. responsible layer;
  4. owner;
  5. remediation;
  6. prevention.

40. Common architect mistakes​

Mistake 1 — Treating federation as free​

Remote compute and network still cost money.

Mistake 2 — Ignoring source capacity​

The external database becomes part of your analytics platform.

Mistake 3 — Allowlisting only compute for OAuth​

OAuth may require control-plane connectivity.

Mistake 4 — Using public access for highly sensitive workloads​

Evaluate private connectivity.

Mistake 5 — Federating directly against OLTP primary​

Use a replica or ingestion when appropriate.

Mistake 6 — Measuring only Databricks latency​

Measure source execution and network transfer too.

Mistake 7 — No fallback​

Critical analytics should have a resilience strategy.

Mistake 8 — Giving every analyst connection privileges​

Centralize connection ownership.

41. Enterprise reference architecture​

+----------------------------------------------------+ | Enterprise Network | | | | +--------------+ +-----------------------+ | | | Operational | | Read / Analytics | | | | Database |------>| Replica | | | +--------------+ +-----------+-----------+ | +--------------------------------------|-------------+ | Private / Controlled Connectivity | Databricks Compute | Unity Catalog | +-----------------+----------------+ | | v v Foreign Catalog Delta Lake | | Live access Owned analytics | | +-----------------+----------------+ | BI / ML / AI

The strongest architecture is often federation for freshness plus Delta for ownership and resilience.

42. Interview questions​

What is the network path for a federated query?

Databricks compute to the external database. Unity Catalog and the control plane are not normally on the data path.

Why can OAuth still require control-plane connectivity?

The OAuth token exchange can originate from the Databricks control plane.

What is the first performance lever?

Reduce remote data movement through filter, projection and aggregation pushdown.

Why can federation overload an operational database?

Analytical queries execute against remote source compute and consume source CPU, memory, connections and I/O.

When would you use a read replica?

When near-live federated access is needed but the primary transactional workload must be protected.

When should federation become ingestion?

When workload volume, latency, repeated scans, historical ownership, source protection or resilience requirements justify a local analytical copy.

What should be measured in a federation benchmark?

Databricks latency, remote execution time, rows/bytes transferred, source resource usage, concurrency and failure rate.

What is the enterprise fallback pattern?

Use federation for live access while maintaining a governed Delta copy for historical analytics and resilience.

43. Production checklist​

  • Network path documented.
  • DNS tested.
  • Firewall rules documented.
  • Private connectivity evaluated.
  • OAuth control-plane path evaluated where applicable.
  • TLS validated.
  • Connection ownership centralized.
  • Credentials securely managed.
  • Source authorization documented.
  • Unity Catalog permissions defined.
  • Pushdown benchmark completed.
  • Source capacity tested.
  • BI concurrency tested.
  • Fetch-size tuning evaluated where supported.
  • Cross-system joins assessed.
  • Source replica considered.
  • Audit and lineage validated.
  • Timeout policy defined.
  • Retry policy defined.
  • Failure runbook created.
  • DR/fallback architecture defined for critical workloads.
  • Cost comparison with ingestion completed.

44. Final mental model​

IDENTITY ↓ UNITY CATALOG ↓ CONNECTION ↓ NETWORK ↓ TLS / AUTH ↓ EXTERNAL DATABASE ↓ PUSH DOWN ↓ REMOTE EXECUTION ↓ DATA TRANSFER ↓ DATABRICKS PROCESSING ↓ GOVERNANCE / OBSERVABILITY

The enterprise rule:

Federation is successful only when governance, network, source capacity, query execution, cost and resilience are designed together.

That is the difference between a federation demo and an enterprise architecture.