08.2 Batch Pipelines in Databricks

Architect goal: move from writing one transformation notebook to designing a repeatable, restartable, observable and governed batch data platform.

A batch pipeline processes data in bounded units rather than continuously.

Examples:

• daily finance load
• hourly procurement refresh
• nightly customer master synchronization
• weekly reconciliation
• monthly financial close
• periodic ML feature preparation

A batch pipeline is not just a script that runs on a schedule. It is a system with state, dependencies, contracts, recovery behavior and operational ownership.

1. Batch Processing Mental Model

A production batch pipeline can be viewed as:

SOURCE ↓ INGEST ↓ LAND ↓ BRONZE ↓ VALIDATE ↓ SILVER ↓ QUALITY GATE ↓ GOLD ↓ CONSUME

Around this flow sit:

• orchestration
• identity
• governance
• observability
• cost controls
• recovery
• metadata
• lineage

The data path and the control path should be designed separately.

2. Data Plane vs Control Plane

Data plane

Moves and transforms business data.

Examples:

• Spark
• SQL
• Delta tables
• source connectors
• cloud storage

Control plane

Controls how the workload runs.

Examples:

• Lakeflow Jobs
• schedules
• dependencies
• parameters
• retries
• alerts
• run history
• permissions

A useful architecture is:

CONTROL PLANE │ schedule / dependencies parameters / retries notifications / policy │ ▼ DATA PIPELINE │ ┌──┼──┐ ▼ ▼ ▼ Bronze Silver Gold │ ▼ DATA CONSUMERS

Lakeflow Jobs provides workflow automation for coordinating multiple tasks, with tasks and dependencies forming a DAG. Jobs also support triggers, parameters, notifications and Git settings.

3. What Makes a Batch Pipeline Production-Ready?

Question	Required design
What data should be processed?	Increment boundary
When should it run?	Trigger
In what order?	Dependencies
What if it fails?	Retry / recovery
What if it runs twice?	Idempotency
How do we know it worked?	Quality checks
Who can access it?	Identity + governance
How do we investigate?	Logs + lineage
How much does it cost?	Compute + storage metrics
How do we change it safely?	Version control + deployment
How do we replay history?	Backfill strategy
What is the SLA?	Operational contract

If these questions cannot be answered, the pipeline is not yet an enterprise pipeline.

4. Batch Boundaries

A batch needs a clearly defined boundary.

Common boundaries:

Time-based

2026-11-07 00:00 → 2026-11-08 00:00

Sequence-based

source_sequence > 983420

File-based

files arriving in batch 20261108

Business-event based

financial close period = 2026-10

Do not confuse a schedule with a processing boundary.

A job can run hourly while processing a source watermark.

5. Full Load vs Incremental Load

Full load

Source ↓ Read everything ↓ Replace target

Useful for:

• small reference tables
• initial migration
• reconciliation
• controlled rebuilds

Problems at scale:

• expensive
• slow
• source-heavy
• increases recovery time

Incremental load

Source ↓ Identify changed records ↓ Process only changes ↓ Merge into target

Typical mechanisms:

• source CDC
• last-modified timestamp
• source sequence
• watermark
• change files
• event batches

For large enterprise datasets, incremental processing should normally be the default.

6. Watermarks

A watermark records how far a pipeline has successfully processed.

Example:

last_successful_timestamp

2026-11-07 23:00:00

Next run can process records beyond that boundary.

A stronger design stores:

• source system
• object
• watermark type
• previous value
• new value
• run ID
• status
• timestamp

Do not update the watermark before the target transaction is safely committed.

Otherwise:

watermark advances ↓ target write fails ↓ records are skipped

That is a classic data-loss failure.

7. Batch IDs

A batch identifier gives every execution an operational identity.

Example:

batch_id = FINANCE_20261108_0100

Useful metadata includes:

batch_id source_system source_object run_id ingest_timestamp processing_timestamp record_count status

Benefits:

• traceability
• reconciliation
• support investigations
• replay
• audit
• duplicate detection

8. Idempotency

A pipeline is idempotent when repeating the same logical processing does not create an incorrect additional result.

Suppose Batch 1042 writes 1,000 orders and the job fails after the data is committed. A naive append on retry can create 2,000 rows.

An idempotent design produces the same 1,000 correct business records.

Common patterns:

MERGE

Match using a stable business key.

Replace a bounded partition

Useful when the processing boundary is a complete partition such as a business date.

Transactional overwrite

Replace a known target scope atomically.

Deterministic staging

Write a run-specific staging result and publish it only after validation.

9. Retry Is Not Recovery

A retry answers:

Should I execute the task again?

Recovery answers:

Can I safely execute the task again?

Databricks supports configurable task retries and timeout policies in Lakeflow Jobs.

But retry does not automatically make data processing idempotent.

For example:

Task ↓ write 50% ↓ network failure ↓ retry ↓ append same 50%

You can create duplicates.

Therefore:

Retry policy + Idempotent data operation + Correct checkpoint/state handling

Safe recovery

Repair runs likewise require task-level idempotency. A repair can rerun unsuccessful tasks, but it does not make their writes safe automatically.

10. Orchestration

A multi-step pipeline might look like:

START │ ▼ INGEST SAP │ ▼ VALIDATE INGEST │ │ PASS FAIL │ │ ▼ ▼ SILVER QUARANTINE │ ▼ QUALITY CHECK │ ▼ GOLD │ ▼ PUBLISH │ ▼ NOTIFY

This is a DAG.

Tasks represent work. Dependencies represent ordering. Conditions represent control flow.

Lakeflow Jobs supports sequential and parallel dependencies, conditional flows, retries, task values and For each processing.

11. Parallelism

Not every task should run sequentially.

INGEST │ ┌──┼──────────┐ ▼ ▼ ▼ Finance Procurement CRM │ │ │ └──┼──────────┘ ▼ PUBLISH

Independent branches can execute concurrently.

Benefits:

• lower wall-clock time
• better compute utilization
• independent failure domains

But excessive parallelism can create:

• source-system overload
• warehouse contention
• concurrency limits
• network saturation
• cost spikes

Architect for controlled parallelism, not maximum parallelism.

12. Parameters

Production pipelines should not hard-code every environment or processing boundary.

Typical parameters:

environment source_system business_date start_timestamp end_timestamp catalog schema target_table run_mode

Example:

environment = prod business_date = 2026-11-08 run_mode = incremental

Lakeflow Jobs supports job parameters, task parameters, dynamic value references and task values.

This allows one pipeline definition to support:

• DEV
• TEST
• PROD
• backfills
• reprocessing
• controlled manual runs

13. Environment Separation

Do not rely on developers changing notebook variables manually.

A mature pattern is:

SAME CODE │ ┌────────────┼────────────┐ ▼ ▼ ▼ DEV TEST PROD │ │ │ catalog catalog catalog compute compute compute secrets secrets secrets

Environment-specific configuration should be externalized. Code should remain portable.

14. Data Quality Gates

A quality gate decides whether downstream processing is allowed.

Bronze ↓ Schema check ↓ Record count ↓ Null check ↓ Duplicate check ↓ Business rule check ↓ PASS ─────→ Silver / Gold │ FAIL ↓ Quarantine + alert

A good quality framework records:

• rule
• threshold
• actual value
• status
• batch ID
• run ID
• affected record count

Create machine-readable operational evidence rather than only printing a failure message.

15. Quarantine

Bad records should not automatically disappear.

A quarantine design might contain:

quarantine_table ├── source_record ├── rule_id ├── error_reason ├── batch_id ├── detected_at └── replay_status

This supports investigation, correction, replay and audit.

For regulated environments, quarantine retention and access should also be governed.

16. Dependency Design

A dependency should represent a real business or technical requirement.

Good:

SAP_INGEST ↓ FINANCE_SILVER ↓ FINANCE_GOLD

Bad:

TASK_A ↓ TASK_B ↓ TASK_C ↓ TASK_D

when B, C and D are actually independent.

Unnecessary dependencies serialize work and increase SLA.

The architect question is:

What must be true before this task can safely execute?

17. Failure Isolation

Imagine:

INGEST │ ┌──────────┼──────────┐ ▼ ▼ ▼ Finance Procurement CRM │ │ │ ▼ ▼ ▼ Gold Gold Gold

A CRM failure should not necessarily block Finance.

This requires:

• independent branches
• clear dependencies
• appropriate conditional execution
• separate checkpoints
• domain-level ownership

18. Backfills

A production data platform must support historical correction.

Normal processing 2026-11-08 ↓ Issue discovered ↓ Backfill 2026-10-01 → 2026-10-07

Backfill design should include:

• parameterized date range
• isolated run identity
• idempotent writes
• controlled concurrency
• downstream impact assessment
• reconciliation
• audit trail

Never modify production logic manually just to process historical data.

19. Late-Arriving Data

Suppose the business date is November 7 but source data arrives November 9.

A pipeline that only processes yesterday can permanently miss the record.

Better designs use:

• source sequence
• ingestion timestamp
• sliding processing window
• CDC
• reconciliation
• periodic correction jobs

Example:

Daily run ↓ Process last 3 days ↓ MERGE ↓ Reconcile older periods periodically

The exact window depends on source behavior and SLA.

20. SAP → Databricks Example

For an SAP procurement pipeline:

S/4HANA │ ▼ CDC / extraction │ ▼ Bronze Purchase Orders │ ├── batch_id ├── source_sequence └── ingestion timestamp │ ▼ Silver Purchase Orders │ ├── deduplicate ├── normalize ├── apply CDC └── validate │ ▼ Gold Procurement Mart │ ├── supplier spend ├── open PO ├── delivery performance └── purchasing KPI

This should be orchestrated as separate tasks rather than one giant notebook.

21. One Giant Notebook Is an Anti-Pattern

A common beginner design puts ingestion, transformation, validation, publishing, notification and cleanup into one notebook.

Problems:

• difficult testing
• poor failure isolation
• hard retries
• long recovery time
• weak ownership boundaries
• difficult deployment

Prefer modular tasks:

01_ingest 02_validate 03_silver 04_quality 05_gold 06_publish 07_notify

The exact number of tasks depends on operational boundaries. Do not split tasks simply to create more nodes.

22. When to Use Multiple Jobs

Create separate jobs when workflows have different:

• SLAs
• owners
• schedules
• failure domains
• security boundaries
• deployment lifecycles

For example:

Finance Ingestion Job ↓ Finance Transformation Job ↓ Finance Publishing Job

But do not create a separate job for every tiny notebook. Too many jobs create operational fragmentation.

23. Job-to-Job Composition

Lakeflow Jobs can use a Run Job task to trigger another job. Current Databricks documentation limits nested Run Job depth to three levels and does not support circular dependencies.

Use this carefully.

Good:

Enterprise_Orchestrator │ ├── Finance_Job ├── Procurement_Job └── Customer_Job

Avoid deeply nested job chains because debugging and ownership become increasingly difficult.

24. Observability

A production pipeline should expose:

Operational metrics

• start time
• end time
• duration
• status
• retry count
• task status

Data metrics

• input records
• output records
• rejected records
• duplicate records
• changed records

Platform metrics

• compute time
• DBUs
• storage growth
• shuffle
• spill
• cluster utilization

Business metrics

• revenue totals
• procurement totals
• transaction counts
• reconciliation differences

The most useful monitoring combines all four.

25. SLA Monitoring

Suppose:

SLA = Gold data available by 06:00

Do not monitor only whether the job eventually succeeds.

Monitor:

Expected completion ↓ Actual completion ↓ SLA status

A job can be technically successful but operationally failed.

Example:

Job succeeded at 09:30 SLA = 06:00

The pipeline status is green. The business service is red.

26. Cost Controls

Batch architecture should consider:

• job compute vs all-purpose compute
• serverless where appropriate
• autoscaling
• Photon
• task concurrency
• data volume
• shuffle
• small files
• unnecessary full scans
• redundant transformations

A useful optimization sequence is:

Reduce unnecessary data ↓ Reduce unnecessary computation ↓ Improve execution plan ↓ Tune compute ↓ Tune schedule/concurrency

Do not start cost optimization by simply selecting a smaller cluster.

27. Security

Every production batch pipeline needs:

• service identity
• least-privilege permissions
• Unity Catalog access
• secret management
• source credentials
• target permissions
• auditability
• environment separation

Never embed credentials or tokens inside notebooks.

Security belongs in the platform architecture.

28. Transactional Publishing

A powerful pattern is:

Transform ↓ Validate ↓ Stage ↓ Publish atomically

Do not expose partially processed Gold data to consumers.

For critical datasets:

Run 1042 ↓ stage_1042 ↓ quality PASS ↓ publish

Consumers should see a coherent dataset rather than a half-written batch.

29. Reconciliation

A pipeline should prove that it processed what it was supposed to process.

Example:

SAP records = 1,000,000 Bronze records = 1,000,000

Silver: Valid = 998,500 Rejected = 1,500

Gold: Business rows = expected result

Reconciliation should explain differences rather than merely reporting them.

30. Production Runbook

For every production pipeline document:

Before run

• source availability
• expected volume
• dependency status
• parameter values

During run

• task status
• record counts
• quality gates
• SLA progress
• resource usage

On failure

• identify failed task
• inspect logs
• determine partial writes
• verify checkpoint/watermark
• determine whether retry is safe
• repair or backfill

After run

• reconcile
• validate Gold
• confirm SLA
• record anomalies

This is what turns a pipeline into an operational service.

31. Hands-On Lab 1 — Build a Batch DAG

Create:

01_ingest 02_validate 03_silver 04_quality 05_gold 06_publish

Configure dependencies:

ingest ↓ validate ↓ silver ↓ quality ↓ gold ↓ publish

Then introduce a deliberate failure in the quality task.

Observe:

1. which tasks execute
2. which tasks are skipped
3. how the DAG reports status
4. what happens on retry

32. Hands-On Lab 2 — Parameterize the Pipeline

Create:

environment business_date run_mode

Run the same workflow for:

DEV / 2026-11-07 / incremental DEV / 2026-11-08 / incremental DEV / 2026-11-01 / backfill

Verify that the code does not need modification.

33. Hands-On Lab 3 — Build an Idempotent Load

Load a dataset twice.

Then verify:

1. row count
2. business-key uniqueness
3. update behavior
4. delete behavior
5. rerun behavior

The second execution must produce the same logical result as the first execution.

34. Hands-On Lab 4 — Add a Quality Gate

Implement checks for:

• record count
• null business keys
• duplicate keys
• invalid dates
• invalid status

If the quality score fails:

Gold must not publish

Instead:

quarantine + alert + operational evidence

35. Hands-On Lab 5 — Backfill

Create a parameterized historical run.

Process:

2026-10-01 → 2026-10-07

Verify:

• no duplicate data
• correct batch identity
• correct downstream results
• reconciliation
• auditability

36. Architect Exercise

Design a production pipeline for:

50 million SAP records per day, 24-hour source availability, Gold SLA of 30 minutes, occasional late-arriving records, strict audit requirements and seven-year financial retention.

Your design must answer:

1. Full or incremental?
2. What is the processing boundary?
3. How is the watermark stored?
4. What makes the pipeline idempotent?
5. How are retries handled?
6. How are late records handled?
7. How is quality enforced?
8. How is backfill implemented?
9. How are failures isolated?
10. How is SLA measured?
11. How is cost controlled?
12. How is the pipeline governed?

37. Interview Questions

Beginner

What is batch processing?

Processing data in bounded executions rather than continuously.

Why use incremental loads?

To reduce source load, processing time and compute cost.

Intermediate

What is a watermark?

A state value representing the boundary through which data has been successfully processed.

Why is idempotency important?

Because retries, reruns and repairs should not corrupt the target with duplicate or inconsistent data.

Advanced

Is retry enough to make a pipeline reliable?

No. Retry only re-executes the task. The data operation must itself be safe to repeat.

Architect

How would you design a 30-minute SLA batch platform?

Separate ingestion and transformation, use incremental processing, parallelize independent domains, enforce quality gates, design idempotent writes, parameterize the processing boundary, instrument data and operational metrics, and design explicit recovery and backfill paths.

38. Architect Mental Model

Remember:

BATCH PIPELINE │ ┌────────────┼────────────┐ ▼ ▼ ▼ DATA FLOW CONTROL FLOW STATE │ │ │ Bronze Jobs/DAG watermark Silver retries batch ID Gold schedule checkpoints │ │ │ └────────────┼────────────┘ ▼ OPERABILITY quality / SLA / cost lineage / audit / DR

The strongest batch architecture is not the one with the most notebooks.

It is the one that can answer:

What should run, what state has been processed, what happens if it fails halfway, how do we safely rerun it, and how do we prove the result is correct?

That is the foundation for every larger Databricks data-engineering architecture.