Skip to main content

12. Lakeflow Connect — Enterprise Data Ingestion

Learning objective: start with the idea of moving data from another system into Databricks, then learn how to select, secure, deploy, monitor and operate the correct enterprise ingestion architecture.

Lakeflow Connect is Databricks' connector and managed-ingestion layer for bringing data from enterprise databases, SaaS applications, enterprise file services, message sources and other supported systems into the lakehouse.

It is not one universal connector engine. Different source types use different capture mechanisms, runtime components and reliability models.

The central architect question is:

How can we ingest the required source data with the right correctness, latency, source impact, governance and operational cost?

1. Where Lakeflow Connect fits​

A complete data platform has several distinct responsibilities:

SOURCE SYSTEMS
      |
      v
INGESTION
      |
      v
DATA ENGINEERING
      |
      v
GOVERNANCE
      |
      v
ANALYTICS / AI / APPLICATIONS

Lakeflow Connect belongs primarily in the ingestion layer.

It can create and operate managed ingestion pipelines. Those pipelines use Lakeflow pipeline capabilities, are governed through Unity Catalog, and integrate with Lakeflow Jobs and the wider Databricks platform.

Think of the platform layers as:

LayerMain responsibility
Lakeflow ConnectCapture data from supported source systems
Lakeflow pipelinesDefine and maintain destination datasets
Delta LakeTransactional table storage
Unity CatalogGovernance, access and discovery
Lakeflow JobsCoordinate multi-task workflows
SQL / BI / MLConsume the resulting data products

Lakeflow Connect does not replace the entire data platform. It standardizes source ingestion.

2. Animated enterprise architecture​

Animated Lakeflow Connect architecture

How to read it: the source systems enter through source-specific connector paths. Database CDC has a separate gateway and staging path. The result is written into Unity Catalog-governed destination tables for downstream data products.

The key lesson:

A managed ingestion experience does not mean every connector has the same internal architecture.

3. Connector families​

Lakeflow Connect organizes connectors by source type.

Connector familyExample sourcesTypical ingestion pattern
Database CDCMySQL, PostgreSQL, SQL Server, OracleInitial snapshot + change capture
SaaSSalesforce, HubSpot, Jira, WorkdayAPI reads, cursors, incremental extraction
Enterprise file servicesSharePoint, Google DriveFile and object ingestion
Query-based databaseSupported relational databasesScheduled source queries using cursor columns
Managed streamingRabbitMQ in the current AWS managed-connector documentationContinuous message ingestion
Community connectorsCommunity-maintained sourcesConnector interface backed by Lakeflow pipelines
Custom connectorsSources without an available connectorCustom implementation and testing

Connector availability, cloud support and release state can differ. A connector listed in documentation is not automatically GA or enabled in every workspace.

Production rule: verify the exact connector, cloud, region, authentication method, feature release state, supported tables and limitations before committing to a design.

4. Managed connectors vs standard ingestion​

Databricks provides different levels of abstraction.

MORE CONTROL
    |
    v
STRUCTURED STREAMING
    |
    v
LAKEFLOW PIPELINES + STANDARD SOURCES
    |
    v
MANAGED LAKEFLOW CONNECTORS
    |
    v
MORE AUTOMATION

Managed connector​

The connector handles much of the source-specific work:

  • authentication integration;
  • source metadata discovery;
  • source-specific extraction;
  • incremental reads where supported;
  • CDC where supported;
  • schema evolution behavior;
  • automated retry and lifecycle handling.

Standard connector / custom pipeline​

You gain more control over:

  • source-specific options;
  • transformations;
  • processing logic;
  • custom state;
  • custom error handling;
  • specialized network or runtime requirements.

Architect decision​

Start with the most managed option that meets the requirements. Move down to a more customizable layer only when the managed connector does not support the required source behavior, semantics or controls.

5. Connection, pipeline and destination​

A managed connector commonly exposes three key concepts:

  1. Connection — a governed object that stores source endpoint and authentication information.
  2. Ingestion pipeline — the process that extracts and incrementally loads source data.
  3. Destination tables — the tables receiving the ingested data.

For database CDC connectors, the architecture adds an ingestion gateway and staging storage.

Do not confuse a connection with a pipeline. A connection describes how to reach a source; the pipeline performs the ingestion.

6. Unity Catalog connections​

A connection is a Unity Catalog securable object.

It centralizes source connectivity information rather than embedding secrets throughout notebook code.

A connection may represent:

  • endpoint or host;
  • authentication configuration;
  • credential references;
  • source-specific connection parameters;
  • governed access to the source.

Conceptually:

WORKLOAD IDENTITY
       |
       v
UNITY CATALOG CONNECTION
       |
       v
SOURCE AUTHENTICATION
       |
       v
SOURCE SYSTEM

Security principle​

Do not put passwords, API keys, OAuth refresh tokens or cloud access keys directly in notebooks, pipeline source files, job parameters or Git.

Use governed connection objects and supported secret/identity integrations.

7. Database CDC architecture​

Database CDC connectors are especially important for enterprise operational systems.

A CDC architecture typically has:

SOURCE DATABASE
      |
      v
INITIAL SNAPSHOT + CHANGE LOG
      |
      v
INGESTION GATEWAY
      |
      v
STAGING STORAGE
      |
      v
SERVERLESS INGESTION PIPELINE
      |
      v
DESTINATION STREAMING TABLES

There are distinct runtime components:

ComponentResponsibility
ConnectionSource authentication
Ingestion gatewayCaptures source snapshots, logs and metadata
Staging storageTemporarily buffers extracted data
Ingestion pipelineApplies staged changes to destination tables
DestinationGoverned streaming tables backed by Delta

This architecture matters for reliability and cost.

8. Why the database gateway exists​

A database CDC connector must capture source changes reliably.

A continuous gateway can read the source change stream before the source database truncates or expires the relevant transaction-log records.

The gateway is separate from the scheduled ingestion pipeline.

That separation enables:

  • continuous capture of source changes;
  • scheduled application of staged data;
  • buffering while the destination pipeline is idle;
  • recovery from temporary destination pipeline failures.

Architect caution​

The database gateway can continue consuming compute even when the downstream ingestion pipeline is not currently running.

This is an important cost difference compared with a purely scheduled query-based connector.

9. Staging storage​

Database CDC connectors use staging storage as an intermediate durability layer.

Conceptually:

SOURCE CHANGE LOG
       |
       v
CONTINUOUS GATEWAY
       |
       v
UNITY CATALOG VOLUME
       |
       v
INGESTION PIPELINE
       |
       v
DESTINATION TABLE

The staging volume decouples the capture process from the destination update schedule.

It can help with:

  • buffering;
  • recovery;
  • temporary ingestion delays;
  • separating source capture from target materialization.

The current Databricks AWS managed database connector documentation describes a staging volume that is automatically created for the gateway and has a 30-day automatic purge policy. Confirm current behavior and retention for the specific connector and cloud before designing compliance or replay guarantees.

10. Initial snapshot vs ongoing CDC​

A CDC ingestion pipeline generally needs to establish the initial source state before it can maintain the destination incrementally.

Initial snapshot​

The connector reads the selected source tables to establish a baseline.

SOURCE TABLES
      |
      v
INITIAL SNAPSHOT
      |
      v
BASELINE DESTINATION

Ongoing capture​

After the baseline is established, changes are captured and applied.

INSERT / UPDATE / DELETE
          |
          v
      CHANGE LOG
          |
          v
      CDC CAPTURE
          |
          v
    INCREMENTAL APPLY

The exact process differs by source database and connector.

Architect decision​

The initial snapshot is often the largest and most resource-intensive phase. Size the gateway, understand source impact and plan the first load separately from steady-state CDC.

11. Database-specific CDC mechanisms​

Different databases expose changes differently.

Source databaseExample capture mechanism
MySQLBinary log based capture
PostgreSQLLogical replication / publication and replication slot model
SQL ServerCDC or supported snapshot/change tracking approach
OracleLogMiner-based change capture

The actual prerequisites vary by source and connector version.

Before enabling production replication, verify:

  • supported database version;
  • source permissions;
  • log/CDC configuration;
  • primary-key requirements;
  • supported data types;
  • transaction-log retention;
  • network connectivity;
  • initial load sizing;
  • source workload impact.

Never apply a generic CDC setup checklist without checking the source-specific connector requirements.

12. Transaction ordering and change correctness​

CDC is not simply a stream of arbitrary rows.

It represents changes to source records.

A robust ingestion architecture must preserve the intended change semantics:

  • source table identity;
  • row key;
  • operation type;
  • source transaction ordering;
  • commit or sequence metadata;
  • delete semantics;
  • schema version where applicable.

Suppose a source emits:

CUSTOMER 100 — INSERT — sequence 10
CUSTOMER 100 — UPDATE — sequence 12
CUSTOMER 100 — DELETE — sequence 15

The destination must apply changes according to the source ordering semantics.

Architect rule​

Do not assume that the order in which an engineer sees records in a destination query is sufficient evidence of source transaction order. Use the connector's documented ordering and consistency model.

13. Source log retention is an architectural dependency​

A CDC gateway must capture source changes before the database removes the relevant log history.

If the gateway is offline for too long, the source may no longer retain all changes needed to catch up.

Therefore monitor:

  • gateway availability;
  • source log retention;
  • replication slot health;
  • CDC capture lag;
  • storage growth;
  • source-side replication errors.

For critical systems, agree on a recovery procedure with the database operations team before go-live.

14. Managed SaaS connectors​

SaaS applications commonly expose APIs rather than database transaction logs.

Examples include:

  • Salesforce;
  • HubSpot;
  • Jira;
  • Workday;
  • ServiceNow;
  • other supported enterprise applications.

The managed connector handles source-specific authentication and extraction behavior.

Typical architecture:

SAAS APPLICATION
       |
       v
API AUTHENTICATION
       |
       v
MANAGED CONNECTOR
       |
       v
INCREMENTAL EXTRACTION
       |
       v
INGESTION PIPELINE
       |
       v
STREAMING TABLES

The connector may use a supported cursor field or application-specific incremental mechanism.

15. SaaS incremental ingestion​

A SaaS API may expose a field such as:

  • updated timestamp;
  • modified date;
  • event cursor;
  • continuation token;
  • API-specific change marker.

Conceptually:

LAST SUCCESSFUL CURSOR
          |
          v
REQUEST NEW / CHANGED RECORDS
          |
          v
WRITE DESTINATION
          |
          v
ADVANCE CURSOR

This reduces repeated full extraction.

However, not every SaaS object or endpoint supports reliable incremental extraction.

Architect questions​

  1. Which objects support incremental reads?
  2. Which cursor field is used?
  3. Can updates arrive late?
  4. How are deletes represented?
  5. Are API rate limits enforced?
  6. What is the backfill strategy?
  7. What happens when a token expires?
  8. Does the API return a stable ordering?

Do not assume that every object in a SaaS application supports the same incremental semantics.

16. SaaS API limits and source protection​

Managed connectors simplify API integration, but they cannot remove the source application's own limits.

A production design must understand:

  • API quota;
  • per-user or per-application request limits;
  • pagination;
  • concurrency limits;
  • rate-limit retry behavior;
  • token expiration;
  • source maintenance windows;
  • object-level access restrictions.

Enterprise principle​

A successful ingestion design must protect both the destination platform and the source application.

Do not optimize Databricks throughput by generating an unacceptable API load on Salesforce, Jira or another SaaS platform.

17. Enterprise file-service connectors​

Enterprise file repositories contain more than simple CSV drops.

Examples:

  • SharePoint documents;
  • Google Drive files;
  • enterprise document repositories;
  • metadata and folder structures;
  • structured exports;
  • unstructured files.

File-service connectors can provide an ingestion path into governed lakehouse storage.

But distinguish them from Auto Loader:

CapabilityFile-service connectorAuto Loader
SourceEnterprise content serviceCloud object storage
Source authenticationConnector-managed connectionCloud storage identity / governed access
DiscoveryService-specific API semanticsFile listing or file events
Data shapeFiles and service metadataFiles in object storage
Typical rolePull content from enterprise repositoriesIncrementally ingest landed files

A practical architecture may use both:

SHAREPOINT
    |
    v
LAKEFLOW CONNECT
    |
    v
LANDING / BRONZE
    |
    v
AUTO LOADER OR LAKEFLOW PIPELINE
    |
    v
SILVER / GOLD

Do not add a second ingestion layer when the connector already lands the data in the required destination.

18. Query-based database connectors​

Not every source database can or should expose CDC.

Lakeflow Connect also offers query-based ingestion for supported database sources.

The connector queries the source database directly using a cursor column.

Typical cursor:

  • monotonically increasing timestamp;
  • increasing integer;
  • suitable change/update marker.

Conceptually:

SOURCE TABLE
     |
     v
QUERY WHERE cursor > LAST HIGH-WATER MARK
     |
     v
DESTINATION STREAMING TABLE
     |
     v
STORE NEW HIGH-WATER MARK

A query-based connector does not require the database CDC gateway and staging volume used by the managed database CDC architecture.

It typically runs on a defined schedule.

19. Query-based high-water marks​

Suppose the source table contains:

Order IDModified timestamp
100109:00
100209:04
100309:07

The last successful high-water mark is 09:04.

The next extraction requests rows newer than that marker.

Conceptually:

SELECT *
FROM source_orders
WHERE modified_at > :last_successful_cursor;

This is a conceptual example, not a connector-specific SQL command.

The connector tracks the high-water mark after a successful run.

Architect caution​

A cursor must be suitable for detecting all relevant changes. If a source allows updates without changing the cursor, the connector can miss them.

20. Query-based ingestion vs CDC​

DimensionDatabase CDCQuery-based ingestion
Change sourceTransaction log / CDC mechanismSource table query
GatewayRequired for managed database CDC architectureNot required
Staging volumeUsed in managed database CDC architectureNot required
ExecutionContinuous capture plus ingestion pipelineScheduled query
Source loadReads change stream after setupQueries source tables directly
Intermediate updatesCan preserve captured change events, subject to connector semanticsCaptures latest changed row state between runs
LatencyDepends on capture and apply scheduleDepends on query schedule
Best fitHigh-fidelity incremental database replicationSimpler incremental extraction

Choose CDC when​

  • change history matters;
  • deletes and intermediate changes matter;
  • low-latency replication is required;
  • source log capture is supported.

Choose query-based when​

  • the source has a reliable cursor column;
  • scheduled freshness is acceptable;
  • source queries are permitted;
  • CDC setup is unavailable or unnecessary.

21. Managed streaming connectors vs standard streaming sources​

Lakeflow Connect includes managed streaming connector support for certain sources.

The current AWS managed streaming connector documentation lists RabbitMQ.

Do not assume that every message broker has the same managed connector experience.

For sources such as Kafka, Kinesis, Google Pub/Sub and Pulsar, Databricks also documents standard ingestion options using Structured Streaming, Lakeflow pipelines and SQL, depending on the source.

The distinction is:

  • Managed streaming connector: source-specific managed lifecycle and authentication.
  • Standard streaming source: you configure the source and pipeline with the supported Spark, SQL or Lakeflow APIs.

This is where the previous Structured Streaming chapter becomes important.

22. Community connectors​

Community connectors extend the platform to sources that do not have a first-party managed connector.

They are open-source and community-maintained.

They can use the Lakeflow Connect interface and run as ingestion pipelines backed by Lakeflow pipelines.

Governance warning​

Community support is not equivalent to a Databricks-managed connector SLA.

Assess:

  • repository activity;
  • issue response;
  • security posture;
  • dependency maintenance;
  • source API changes;
  • testing;
  • compatibility;
  • license;
  • ownership.

For a Tier-1 data product, do not treat a community connector as production-supported without an explicit internal ownership model.

23. Custom connectors​

If a required source has no managed or community connector, a custom connector may be appropriate.

A custom connector typically has to implement source-specific behavior for:

  • listing source tables or objects;
  • discovering schemas;
  • reading source metadata;
  • reading source data;
  • optionally reading delete information.

Custom connector development is a software engineering responsibility.

It requires:

  • interface compatibility;
  • authentication;
  • incremental state design;
  • schema handling;
  • retries;
  • testing;
  • packaging;
  • deployment;
  • ongoing maintenance.

Architect decision​

Build a custom connector only when the business value justifies owning its lifecycle. If a standard connector, partner connector or managed integration already meets the requirement, prefer it.

24. Direct application ingestion with Zerobus​

Lakeflow Connect also includes the Zerobus Ingest API for writing records directly from applications into Databricks streaming tables.

This is a different pattern from pulling data from an existing source system.

APPLICATION
    |
    v
ZEROBUS INGEST API
    |
    v
STREAMING TABLE
    |
    v
DOWNSTREAM DATA PRODUCTS

Use this when an application needs to push records directly into the lakehouse through the supported API.

Do not confuse direct application writes with CDC replication from an operational database.

Verify the API's current availability, supported table storage and operational limits for the target cloud and workspace.

25. Destination tables and the lakehouse model​

Managed ingestion pipelines commonly write to streaming tables.

The data can then enter a medallion architecture:

SOURCE
  |
  v
CONNECTOR
  |
  v
BRONZE
  |
  v
SILVER
  |
  v
GOLD

Bronze​

Preserve the ingested source representation with source metadata and ingestion context.

Silver​

Apply data-quality controls, standardization, deduplication, business keys and conformance.

Gold​

Publish governed data products for reporting, analytics, ML and applications.

The connector's destination should not become the only place where data-quality and business-transformation logic exists.

26. SCD semantics​

A destination table may need different row-history behavior.

Common modes include:

ModeMeaning
SCD Type 1Maintain the latest row state
SCD Type 2Preserve historical versions
Append-onlyAppend records without merging into current state

These choices are business semantics, not merely technical options.

Example​

A customer changes location:

BEFORE: customer 100 → Mumbai
AFTER:  customer 100 → Bengaluru

SCD1 retains Bengaluru as the current value.

SCD2 preserves both the Mumbai and Bengaluru versions with appropriate validity metadata.

Append-only is appropriate when the desired output is an event or observation history rather than one current row per key.

Verify the available history modes for the chosen connector and API.

27. Schema evolution​

Source systems change.

Examples:

  • new source column;
  • removed column;
  • changed type;
  • renamed field;
  • nested JSON change;
  • new SaaS object attribute.

A production ingestion architecture needs an explicit schema strategy.

Ask:

  1. Is the change backward-compatible?
  2. Should it be accepted automatically?
  3. Should the pipeline stop?
  4. Should unexpected data be rescued?
  5. Who approves a breaking change?
  6. How are downstream contracts protected?

Architect rule​

Connector-managed schema evolution does not remove the need for data contracts.

A source schema change may be technically ingestible and still break a business consumer.

28. Incremental ingestion is not automatically CDC​

This distinction is crucial.

An incremental read may capture only the latest state of changed records.

A CDC stream may capture a sequence of source operations.

Example:

09:00 UPDATE customer to A
09:02 UPDATE customer to B
09:04 UPDATE customer to C

A query-based extraction running at 09:05 may retrieve only C, depending on the source query and cursor semantics.

A transaction-log-based CDC feed can capture the sequence of changes, subject to the connector's documented behavior.

Do not use the words incremental, CDC and real-time as if they mean the same thing.

29. Networking architecture​

The connector must reach the source.

The network architecture depends on the source and connector family.

SaaS​

SERVERLESS PIPELINE
       |
       v
CONTROLLED EGRESS
       |
       v
SAAS API

Restricted serverless egress policies may require the source API domains to be explicitly allowed.

Cloud database​

Possible patterns include:

  • Private Link;
  • VPC or VNet peering;
  • supported private connectivity;
  • approved public endpoint.

On-premises database​

Possible network patterns include:

  • VPN;
  • AWS Direct Connect;
  • Azure ExpressRoute;
  • routed private connectivity.

The database CDC gateway must be able to reach the source database from the network where it runs.

Architect rule​

Validate DNS, ports, TLS, routing, firewall rules, egress policy and source allowlists before pipeline deployment.

30. Security architecture​

A production connector should follow least privilege.

SERVICE IDENTITY
      |
      v
UNITY CATALOG CONNECTION
      |
      v
SOURCE-SPECIFIC ACCESS
      |
      v
INGESTION PIPELINE
      |
      v
GOVERNED DESTINATION

Design for:

  • dedicated service identity;
  • minimum source permissions;
  • read-only access wherever possible;
  • scoped API permissions;
  • secret rotation;
  • controlled destination privileges;
  • staging storage permissions;
  • auditability;
  • sensitive-data classification.

Do not use a powerful human administrator account for unattended ingestion.

31. Source-system impact​

Ingestion can affect the source system.

CDC​

Can consume transaction-log resources, replication slots, log retention and source-side I/O.

Query-based​

Runs queries against source tables and can compete with business workloads.

SaaS​

Consumes API quota and can trigger provider rate limits.

File services​

Can generate API requests, document reads and metadata scans.

The source system is part of the architecture.

A production design must include source-owner approval and source impact assessment.

32. Cost architecture​

Lakeflow Connect cost is not just the cost of the destination pipeline.

Consider:

Cost componentWhat drives it
Ingestion gatewayContinuous classic compute for database CDC
Ingestion pipelineServerless or configured pipeline compute
Staging storageTemporary CDC data and recovery buffer
Source databaseQuery load, logs and replication resources
SaaS providerAPI quota or subscription limits
NetworkCross-cloud transfer and connectivity
DestinationDelta storage and maintenance
DownstreamTransformations, BI, ML and serving

A particularly important distinction:

A managed database CDC gateway may consume compute continuously even when the destination ingestion pipeline is idle.

Model this before choosing between CDC and query-based ingestion.

33. Reliability and recovery​

Every connector design needs a recovery plan.

Transient API failure​

Use connector-supported retry behavior and alert on sustained failures.

Source database unavailable​

The gateway may stop capturing changes. Validate source log retention and connector recovery requirements.

Gateway failure​

Recover the gateway and confirm that the source still retains the changes required for catch-up.

Destination pipeline failure​

Use the supported pipeline recovery and staging behavior. Do not delete staging data or checkpoints casually.

Schema change​

Apply the approved schema-evolution policy and restart or redeploy as required.

Credential expiry​

Rotate or repair the governed connection and verify the next successful extraction.

Backfill​

Plan how to reload a source table or date range without duplicating or corrupting downstream data.

34. Observability​

A production ingestion dashboard should cover five dimensions.

Connector health​

  • connection status;
  • authentication failures;
  • pipeline status;
  • gateway status.

Source progress​

  • source lag;
  • CDC capture lag;
  • cursor high-water mark;
  • API pagination progress;
  • source log retention risk.

Data movement​

  • rows and bytes ingested;
  • insert/update/delete counts where available;
  • table-level progress;
  • failed or skipped objects.

Data quality​

  • schema changes;
  • invalid rows;
  • duplicate keys;
  • null rates;
  • reconciliation differences.

Business freshness​

  • age of newest ingested record;
  • expected vs actual update time;
  • downstream data product readiness;
  • SLA breaches.

A pipeline can be green and still be behind the source. Monitor freshness, not only job status.

35. Lakeflow Connect vs Auto Loader​

These capabilities complement each other.

DimensionLakeflow ConnectAuto Loader
Primary sourceSupported enterprise apps, databases and servicesCloud object storage
AuthenticationConnector-specific governed connectionStorage identity and governed access
Database CDCManaged database connectors where supportedNot its purpose
File discoveryService-specific source behaviorDirectory listing or file events
SchemaConnector-specific evolutionFile schema inference and evolution
Typical destinationGoverned streaming tablesDelta tables / Lakeflow pipeline targets

Example combined design​

SAAS API ───────► LAKEFLOW CONNECT ──┐
                                    │
DATABASE CDC ───► LAKEFLOW CONNECT ─┤
                                    ├──► BRONZE / SILVER / GOLD
CLOUD FILES ────► AUTO LOADER ─────┤
                                    │
KAFKA ──────────► STREAMING SOURCE ─┘

Choose the ingestion technology by source behavior, not by trying to force every source through the same connector.

36. Lakeflow Connect vs Lakehouse Federation​

These also solve different problems.

Lakehouse Federation​

Allows supported external systems to be queried through Databricks without first copying all data into Delta tables.

Lakeflow Connect​

Moves source data into governed destination tables for managed ingestion and downstream lakehouse processing.

Use Federation when the goal is governed access and querying without full ingestion.

Use Connect when the goal is to land and incrementally maintain source data in the lakehouse.

A combined architecture may use Federation for selected live access and Connect for data that needs durable lakehouse materialization.

37. Lakeflow Connect vs third-party ingestion platforms​

Third-party tools can be valuable, especially when they support sources, transformation options, deployment models or operational guarantees not covered by the managed Databricks connectors.

Compare based on:

  • source coverage;
  • CDC semantics;
  • deletion handling;
  • schema evolution;
  • source impact;
  • latency;
  • deployment and networking;
  • operational support;
  • observability;
  • licensing;
  • portability;
  • total cost.

Do not select a connector platform based only on the number of connectors in a marketing list.

The correct connector is the one that satisfies the source and business contract.

38. SAP and ERP integration architecture​

For SAP landscapes, do not assume a native managed Lakeflow Connect connector is available for every SAP source or extraction method.

First verify the current supported connector catalogue and release status for the workspace.

Possible enterprise patterns include:

SAP S/4HANA / ECC
        |
        +--> SAP SLT / supported CDC
        |
        +--> ODP / extractors
        |
        +--> approved partner connector
        |
        +--> governed file / object-storage landing
                       |
                       v
              Databricks ingestion
                       |
                       v
                BRONZE / SILVER
                       |
                       v
                  GOLD / BI / ML

The right design depends on:

  • SAP source release;
  • extraction interface;
  • delta semantics;
  • table volume;
  • business downtime constraints;
  • SAP licensing and connector support;
  • network topology;
  • required latency;
  • source-side impact.

Example​

If an organization already uses SAP SLT to replicate selected tables into a supported relational landing database, a supported database CDC or query-based connector can be evaluated for the next stage.

If SLT or an SAP extraction framework lands files into cloud object storage, Auto Loader may be more appropriate.

Do not promise direct SAP table CDC through Lakeflow Connect without confirming the supported connector and source prerequisites.

39. Enterprise reference architecture​

A global retailer might have:

Sources

  • ERP database;
  • CRM;
  • e-commerce SaaS;
  • supplier document repository;
  • event broker.

Ingestion decisions

SourceCandidate patternKey question
ERP databaseManaged CDCCan log-based capture meet the latency and source constraints?
CRMManaged SaaS connectorWhich objects support incremental API extraction?
Supplier filesFile connector or Auto Loader after landingWhere are files stored and how are they discovered?
Event brokerManaged connector or standard streaming sourceIs continuous event processing required?
Unsupported applicationCommunity/custom connector or third-party toolWho owns connector maintenance?

Target architecture

ERP CDC ─────────┐
CRM SaaS ────────┤
Supplier files ──┼──► GOVERNED INGESTION
Event stream ────┤              |
Custom source ───┘              v
                         BRONZE TABLES
                               |
                               v
                         QUALITY CONTRACTS
                               |
                               v
                         SILVER ENTITIES
                               |
                  ┌────────────┼────────────┐
                  v            v            v
              GOLD SALES   GOLD FINANCE  GOLD CUSTOMER
                  |            |            |
                  └────────────┼────────────┘
                               v
                       BI / ML / PRODUCTS

The architecture is source-aware at ingestion and domain-oriented at the destination.

40. Hands-on lab — choose the right connector​

For each requirement, choose the most suitable ingestion pattern and justify it.

ScenarioCandidate
Salesforce CRM records with supported incremental API fieldsManaged SaaS connector
PostgreSQL changes with low-latency captureManaged database CDC connector
Database with a reliable update timestamp but no usable CDC setupQuery-based connector
Files arriving in an S3 or ADLS landing zoneAuto Loader
Kafka events requiring custom processingStructured Streaming or Lakeflow pipeline standard source
Unsupported SaaS API with no managed connectorEvaluate community/custom/third-party connector
Application pushes event records directly to a streaming tableEvaluate Zerobus Ingest

For each decision, document the trade-off in source impact, latency, correctness, cost and support ownership.

41. Hands-on lab — design a database CDC rollout​

Build a rollout plan for a PostgreSQL source.

Before the first run​

  • Confirm database version and connector support.
  • Validate logical replication and source-side prerequisites.
  • Create a dedicated least-privilege source identity.
  • Validate network connectivity from the gateway.
  • Confirm primary keys and supported data types.
  • Agree on source log retention and replication monitoring.
  • Select destination catalog and schema.
  • Size the gateway for initial snapshot and steady-state change volume.

Initial load​

  • Select a representative table set.
  • Run the snapshot.
  • Compare source and destination counts.
  • Validate key uniqueness and important totals.
  • Confirm CDC capture starts at the expected boundary.

Steady state​

  • Monitor gateway health.
  • Monitor capture lag.
  • Monitor ingestion pipeline failures.
  • Validate inserts, updates and deletes.
  • Test controlled restart and recovery.
  • Document operational ownership.

Use the actual PostgreSQL connector instructions for the exact configuration and supported version.

42. Hands-on lab — query-based ingestion​

Choose a table with a reliable modification timestamp.

Example source data:

order_idamountmodified_at
100125009:00
100250009:04
100312509:07

Design:

  1. Choose the cursor column.
  2. Confirm its behavior for every update.
  3. Determine whether it is monotonic enough for the connector.
  4. Configure a successful-run high-water mark.
  5. Run the next scheduled extraction.
  6. Update a source row and observe the destination.
  7. Test a late or backdated source update.
  8. Test deletion behavior.
  9. Measure source query load.
  10. Reconcile source and destination state.

The critical exercise is proving that the chosen cursor does not silently miss updates.

43. Hands-on lab — SaaS ingestion and governance​

Choose a supported SaaS connector in your workspace.

Build:

SOURCE APPLICATION
        |
        v
GOVERNED CONNECTION
        |
        v
MANAGED INGESTION PIPELINE
        |
        v
DESTINATION STREAMING TABLE
        |
        v
SILVER DATA PRODUCT

Test:

  1. Authentication.
  2. Object selection.
  3. Incremental-read capability.
  4. Pagination.
  5. API rate-limit behavior.
  6. Credential rotation.
  7. Schema change.
  8. Delete handling.
  9. Pipeline restart.
  10. Source-to-target reconciliation.

Then document which behaviors are managed by Databricks and which remain your team's responsibility.

44. Troubleshooting guide​

SymptomLikely area to inspect
Connection test failsDNS, TLS, credentials, source firewall
Database gateway cannot startCompute sizing, network, source permissions
CDC lag increasesSource log throughput, gateway health, network, destination backlog
Initial snapshot failsGateway sizing, source load, unsupported types, permissions
SaaS extraction stopsToken expiry, API limit, endpoint permission, connector error
Query-based connector misses updatesCursor correctness, timestamp precision, update semantics
New column causes failureConnector schema evolution policy
Destination freshness degradesPipeline schedule, source lag, processing duration
Duplicate or incorrect target rowsPrimary key, SCD mode, ordering, replay behavior
Cost grows unexpectedlyAlways-on gateway, ingestion frequency, source/API volume, destination compute

Avoid restarting or deleting state as the first troubleshooting action. Establish the failure boundary first.

45. Production design checklist​

Source readiness​

  • Exact connector and release state verified.
  • Source version supported.
  • Source owner approval obtained.
  • Authentication model approved.
  • Table/object scope documented.
  • Primary keys and cursor/CDC semantics verified.
  • Source impact assessed.

Connectivity and security​

  • Network path validated.
  • Firewall and egress rules reviewed.
  • Unity Catalog connection created.
  • Least-privilege identity configured.
  • Credential rotation tested.
  • Sensitive-data access reviewed.

Pipeline architecture​

  • Managed vs query-based vs custom selected intentionally.
  • Initial snapshot planned.
  • Incremental method documented.
  • Gateway sizing completed where required.
  • Staging volume and retention understood.
  • Destination catalog/schema defined.
  • SCD and delete semantics agreed.

Reliability and quality​

  • Retry/restart strategy tested.
  • Source log retention monitored where CDC is used.
  • Schema evolution policy approved.
  • Backfill and replay plan tested.
  • Source-to-target reconciliation defined.
  • Freshness SLA defined.

Operations and cost​

  • Pipeline event monitoring enabled.
  • Gateway monitoring enabled where applicable.
  • API/source limits monitored.
  • Cost model includes continuously running components.
  • On-call ownership assigned.
  • Recovery runbook approved.

Delivery​

  • Configuration source-controlled.
  • Deployment automated.
  • DEV/TEST/PROD parameters separated.
  • Changes reviewed.
  • Rollback and recovery process documented.

46. Enterprise architect exercise​

Design ingestion for a global organization with:

  • Oracle ERP;
  • PostgreSQL order management;
  • Salesforce CRM;
  • Workday HR;
  • SharePoint documents;
  • Kafka application events;
  • a custom procurement application.

Requirements:

  • CRM freshness under 30 minutes;
  • finance reconciliation every night;
  • near-real-time order updates;
  • five-year customer history;
  • strict data access controls;
  • source systems must remain stable;
  • all ingested data must be auditable.

Your architecture must answer:

  1. Which sources use managed connectors?
  2. Which sources use CDC?
  3. Which sources use query-based ingestion?
  4. Which need a community or custom connector?
  5. Which use Auto Loader or standard streaming?
  6. Where are source connections stored?
  7. Where is the CDC gateway deployed?
  8. What is staged and for how long?
  9. How are deletes represented?
  10. Which tables need SCD Type 2?
  11. How is schema evolution controlled?
  12. What is the source impact?
  13. What network paths are required?
  14. What are the source and destination SLAs?
  15. How is replay performed?
  16. How is data quality measured?
  17. How are costs allocated?
  18. Who owns each connector and data product?

A real architect should be able to defend each choice and explain why alternatives were not selected.

47. Interview questions​

Fundamentals​

  1. What is Lakeflow Connect?
  2. What is the difference between a connection and an ingestion pipeline?
  3. What connector families are available?
  4. How does Lakeflow Connect integrate with Unity Catalog?
  5. How is a managed connector different from a standard connector?

Database and CDC​

  1. What is the role of the ingestion gateway?
  2. Why is staging storage used?
  3. Why does database CDC need a continuous capture process?
  4. What is the difference between an initial snapshot and ongoing CDC?
  5. What happens if source transaction logs are no longer available?
  6. What are the cost implications of an always-running gateway?
  7. When would you choose query-based ingestion instead of CDC?

SaaS and files​

  1. How do SaaS connectors perform incremental extraction?
  2. What is a high-water mark?
  3. How do API rate limits affect ingestion architecture?
  4. How do you handle source schema evolution?
  5. How do enterprise file connectors differ from Auto Loader?

Architecture​

  1. How would you integrate multiple sources into a medallion architecture?
  2. How do Lakeflow Connect and Lakeflow Jobs work together?
  3. How do you secure source credentials?
  4. How do you validate source-to-target completeness?
  5. How would you design an unsupported-source connector?
  6. How would you design replay without duplicating data?
  7. How do you calculate total ingestion cost?

Enterprise architect​

  1. How would you design Oracle CDC for a high-volume ERP?
  2. How would you ingest SaaS data without exhausting API quotas?
  3. How would you choose between CDC, query-based ingestion and federation?
  4. How would you monitor source capture lag?
  5. How would you deploy connector pipelines across DEV, TEST and PROD?
  6. How would you build a connector operating model for 100+ source systems?

48. The architect mental model​

Remember five decisions:

1. SOURCE CAPABILITY
        |
        v
2. CAPTURE MECHANISM
        |
        v
3. CONNECTIVITY + IDENTITY
        |
        v
4. DESTINATION + CHANGE SEMANTICS
        |
        v
5. OPERATIONS + COST + SLA

Source capability​

What can the source reliably expose: transaction logs, APIs, files, cursor columns or events?

Capture mechanism​

Should the design use managed CDC, query-based extraction, SaaS API reads, a file connector, Auto Loader or Structured Streaming?

Connectivity and identity​

How does the platform reach the source securely and with least privilege?

Destination and change semantics​

Are records append-only, current-state, historical SCD2 or event history?

Operations and cost​

What happens when the source, gateway, API, network or destination fails?

49. Final enterprise takeaway​

Lakeflow Connect is not simply a catalogue of source connectors.

It is a managed ingestion capability that brings source-specific extraction, incremental loading, connectivity, pipeline execution and governance into the Databricks platform.

But the architect must still understand what happens behind the connector.

A successful ingestion design must prove:

  • the correct source data is captured;
  • source changes are not silently missed;
  • the source is not overloaded;
  • credentials and network paths are governed;
  • destination semantics are correct;
  • schema changes are controlled;
  • recovery is possible;
  • freshness and cost meet the business contract.

Architect principle: select the connector by source semantics, not by connector availability alone. A connector that can reach a source is not necessarily a connector that can meet the enterprise correctness, latency, recovery and governance requirements.

Official references​