11. Lakeflow Jobs & Orchestration — Production Workflow Engineering

Goal: learn how to turn individual Databricks workloads into reliable, observable, recoverable enterprise workflows.

Lakeflow Jobs is the Databricks workflow-orchestration layer. A job coordinates tasks, dependencies, parameters, triggers, retries, notifications, monitoring and operational recovery.

Official documentation: https://docs.databricks.com/aws/en/jobs

---

1. The architect mental model

A beginner thinks:

Notebook A
   ↓
Notebook B
   ↓
Notebook C

An architect thinks:

TRIGGER
   ↓
LAKEFLOW JOB
   ↓
DEPENDENCY DAG
   ├── ingest
   ├── validate
   ├── transform
   └── publish
          ↓
       OUTPUT

The job is not merely a scheduler. It is the operational contract of the workflow.

It defines:

• when work starts
• what depends on what
• what can run in parallel
• what happens after failure
• how retries work
• how parameters flow
• how runs are monitored
• how code is versioned
• how production is recovered

---

2. Animated architecture

Animation: the trigger starts the job, work moves through the task DAG, successful branches converge on the output, and the failure path demonstrates retry and repair.

Remember:

Tasks perform work. Jobs control work.

---

3. Job, task and run

Concept	Meaning
Job	Saved workflow definition
Task	One unit of work
Job run	One execution of the job
Task run	Execution of one task
Dependency	Execution relationship
Trigger	Condition that starts a run

Example:

daily_sales_job
       │
       ├── ingest_sales
       ├── validate_sales
       ├── transform_sales
       └── publish_sales

A single execution creates one job run and multiple task runs.

---

4. DAG design

Tasks and dependencies form a directed acyclic graph.

                 INGEST
                /      \
               ↓        ↓
          CUSTOMERS    ORDERS
               \        /
                ↓      ↓
                   JOIN
                    ↓
                 PUBLISH

A dependency should represent a real data or operational dependency.

Do not create artificial chains such as:

A → B → C → D

when B, C and D could safely run in parallel.

Better:

       ┌── B ──┐
A ─────┼── C ──┼──► D
       └── E ──┘

This reduces critical-path latency.

---

5. Triggers

Lakeflow Jobs supports trigger patterns including:

• scheduled execution
• file arrival
• table update
• model update
• upstream job completion
• continuous execution
• manual or external invocation

Official documentation: https://docs.databricks.com/aws/en/jobs/triggers

Schedule-driven

02:00 every day
       ↓
     JOB

Data-driven

new data arrives
       ↓
     JOB

Architecturally, prefer a data-driven trigger when data availability is the true dependency. Do not poll repeatedly when the platform can react to the event.

---

6. Parameters

Avoid hard-coding:

environment = prod
region = IN
business_date = today

Instead define workflow parameters.

JOB PARAMETERS

environment
region
business_date
load_type

The same workflow can then execute against different environments or business dates without copying the implementation.

Official documentation: https://docs.databricks.com/aws/en/jobs/parameters

---

7. Dynamic values and task values

Runtime metadata can be used for:

• job identity
• run identity
• trigger type
• repair count
• parameters
• task results

Example:

TASK A
discover latest partition
        │
        │ task value
        ↓
TASK B
process latest partition

Use task values for small pieces of workflow-control state.

Do not use them as a substitute for proper data storage.

---

8. Control flow

Production workflows often branch.

                 QUALITY CHECK
                       │
               ┌───────┴───────┐
               ↓               ↓
             PASS             FAIL
               ↓               ↓
          TRANSFORM        QUARANTINE
               │
               ↓
            PUBLISH

Lakeflow Jobs provides control-flow patterns such as If/else, Run if and For each.

This makes business and operational decisions visible in the DAG rather than hiding them inside notebooks.

---

9. Retries

Distributed systems have transient failures.

Examples:

• temporary network failure
• throttling
• compute startup failure
• temporary external dependency outage

Retries can automatically recover transient problems.

But:

A retry is safe only when the task is retry-safe.

Consider an append operation:

first attempt → writes rows → fails
second attempt → writes rows again

This can create duplicates.

Prefer idempotent patterns such as deterministic MERGE or safe replacement where appropriate.

---

10. Retry vs repair

Retry

A task is attempted again.

FAIL
 ↓
RETRY
 ↓
SUCCESS

Repair

A failed multi-task run can be repaired by rerunning unsuccessful tasks and required downstream tasks rather than restarting successful work.

A ✓
B ✓
C ✗
D skipped

REPAIR

A ✓
B ✓
C rerun
D rerun

Repair can reduce recovery time and compute cost.

But repair does not make a task idempotent. If the failed task partially wrote data, rerunning it can duplicate output.

Official documentation: https://docs.databricks.com/gcp/en/jobs/repair-job-failures

---

11. Idempotency

Ask this for every production task:

What happens if this task runs twice?

Unsafe:

append today's data

Safer:

MERGE by business key
or
deterministic replacement

Idempotency is one of the most important foundations of reliable orchestration.

---

12. Concurrency

Suppose a job runs every 10 minutes but one execution takes 25 minutes.

02:00 ───── RUN 1 ─────────────
02:10 ───── RUN 2 ─────────────
02:20 ───── RUN 3 ─────────────

Overlapping runs can cause:

• duplicate processing
• conflicting writes
• excess compute
• unexpected downstream state

Ask:

1. Can two runs write the same target?
2. Should a newer run wait?
3. Is the workflow idempotent?
4. What concurrency is acceptable?
5. What does the business SLA require?

---

13. Timeouts

A task normally takes 10 minutes but suddenly runs for eight hours.

hung task
   ↓
compute consumed
   ↓
downstream blocked
   ↓
SLA missed

Use timeouts based on observed runtime distributions rather than arbitrary values.

---

14. Observability

A green job is not enough.

Monitor:

Reliability

• success rate
• failure rate
• retry count
• repair count
• skipped tasks

Performance

• task duration
• queue time
• critical-path duration
• compute utilization

Data

• rows processed
• rejected rows
• freshness
• completeness

Cost

• compute consumption
• repeated retries
• overlapping runs

Business

• dataset ready time
• dashboard freshness
• downstream availability

The best model is:

TECHNICAL SIGNAL
       ↓
PIPELINE STATE
       ↓
BUSINESS IMPACT

---

15. Critical path

Consider:

A ──► B ──► D
 \         ↑
  └──► C ──┘

If:

A = 5 min
B = 20 min
C = 2 min
D = 5 min

the critical path is:

A → B → D

Total latency is approximately 30 minutes.

Making C faster does not materially improve the end-to-end SLA.

Architect rule: optimize the critical path, not simply the slowest-looking individual task.

---

16. Lakeflow Jobs vs Lakeflow pipelines

This distinction is fundamental.

Lakeflow Jobs

Procedural orchestration:

RUN A
 ↓
RUN B
 ↓
IF CONDITION
 ↓
RUN C

Lakeflow pipelines

Declarative data processing:

BRONZE
  ↓
SILVER
  ↓
GOLD

Think:

Jobs coordinate work. Pipelines define data processing.

Databricks currently recommends Lakeflow pipelines for most new ETL workloads, while Lakeflow Jobs remains the orchestration layer.

Official documentation: https://docs.databricks.com/aws/en/data-engineering/choose-etl-approach

---

17. Enterprise pattern

A production data product can look like:

                    TRIGGER
                       │
                       ↓
                CUSTOMER JOB
                       │
          ┌────────────┼────────────┐
          ↓            ↓            ↓
       CRM LOAD     ORDERS LOAD   SUPPORT LOAD
          │            │            │
          └────────────┼────────────┘
                       ↓
                QUALITY GATE
                  /        \
                 /          \
              PASS          FAIL
               ↓              ↓
        LAKEFLOW PIPELINE  QUARANTINE
               ↓
         BRONZE → SILVER
               ↓
              GOLD
          /      |      \
         BI      ML    DATA PRODUCTS

This separates:

• orchestration
• transformation
• governance
• quality
• consumption

---

18. Git and reproducibility

A production run should be reproducible from a known version of source code.

Prefer:

Git
 ↓
validation
 ↓
deployment
 ↓
production job
 ↓
run tied to source version

Lakeflow Jobs supports Git-backed source for supported task types.

Official documentation: https://docs.databricks.com/aws/en/jobs/git

---

19. Declarative Automation Bundles

For enterprise CI/CD, use source-controlled deployment definitions rather than relying on manual production configuration.

Current Databricks terminology:

Databricks Asset Bundles → Declarative Automation Bundles

The architecture becomes:

Developer
   ↓
Git
   ↓
Pull Request
   ↓
Tests / validation
   ↓
Bundle deployment
   ↓
DEV
   ↓
TEST
   ↓
PROD

Databricks currently recommends Declarative Automation Bundles for CI/CD.

Official documentation: https://docs.databricks.com/aws/en/dev-tools/bundles

---

20. Anti-patterns

Giant notebook

One notebook performs everything.

Result:

• difficult testing
• poor failure isolation
• large blast radius
• difficult recovery

Artificial dependencies

Independent tasks are unnecessarily serialized.

Result:

• higher latency
• wasted compute

Hidden dependencies

A task depends on another task but the DAG does not express it.

Result:

• race conditions

Non-idempotent retry

Retry duplicates data.

Manual production editing

The UI configuration diverges from source control.

No owner

Nobody is accountable when the workflow fails.

---

21. Hands-on lab

Build:

INGEST
  ↓
VALIDATE
  ↓
TRANSFORM
  ↓
PUBLISH
  ↓
NOTIFY

Add:

• schedule
• parameters
• dependencies
• retries
• timeout
• notification
• quality branch
• idempotent publish

Then deliberately fail TRANSFORM.

Answer:

1. Which task becomes failed?
2. Which tasks are skipped?
3. What does retry do?
4. What does repair do?
5. Which tasks rerun?
6. Can PUBLISH safely run twice?

---

22. Architect checklist

Workflow

• Dependencies are explicit
• Independent tasks can run in parallel
• Critical path is known
• Control flow is intentional

Reliability

• Retry policy exists
• Tasks are idempotent
• Repair has been tested
• Timeouts are defined
• Concurrency is controlled

Parameters

• Environment is configurable
• Business date is configurable
• Runtime metadata is available
• Task values are used appropriately

Observability

• Job status monitored
• Task duration monitored
• Freshness monitored
• Retry and repair monitored
• Business SLA monitored

DevOps

• Source control enabled
• Production code versioned
• CI/CD defined
• Environment promotion controlled
• Manual production changes minimized

---

23. Interview questions

Beginner

1. What is a Databricks job?
2. What is a task?
3. What is a job run?
4. What is a trigger?
5. Why do tasks form a DAG?

Intermediate

6. How do you parameterize a job?
7. What is retry?
8. What is repair?
9. Why is idempotency important?
10. How do you prevent overlapping runs?

Advanced

11. How would you design parallel branches?
12. How would you design failure notifications?
13. How would you identify the critical path?
14. When would you use If/else?
15. When would you use For each?

Architect

16. When should logic live in Lakeflow Jobs versus Lakeflow pipelines?
17. How would you design DEV, TEST and PROD promotion?
18. How would you make a 200-task workflow maintainable?
19. How would you design recovery for a Tier-1 data product?
20. How would you prevent configuration drift?
21. How would you balance reliability, latency and cost?

---

24. Final architect mental model

TRIGGER
   ↓
JOB
   ↓
DAG
   ↓
TASKS
   ↓
DATA / CONTROL FLOW
   ↓
SUCCESS ─────────► OUTPUT
   │
   └─ FAILURE ───► RETRY
                      ↓
                    REPAIR
                      ↓
                 SAFE RE-RUN

And around the entire workflow:

SECURITY
   │
   ├── CI/CD
   ├── OBSERVABILITY
   ├── COST
   └── SLA
        │
        ▼
       JOB

The architect does not ask only:

“How do I schedule this notebook?”

The architect asks:

“What is the operational contract of this data product, and can the platform execute, recover, observe, secure and promote it reliably?”

That is enterprise Databricks orchestration.