03.2 Jobs, Stages & Tasks

Phase 3 — Apache Spark Fundamentals

Spark does not execute a DataFrame as one giant operation.

It turns your code into a hierarchy:

Spark Application
       |
      Job
       |
     Stages
       |
     Tasks
       |
   Partitions

This hierarchy explains:

• Why an action creates a job.
• Why one job can contain multiple stages.
• Why a stage contains many tasks.
• Why a task normally processes one partition.
• Why joins and aggregations often create shuffle boundaries.
• Why one oversized partition creates a straggler.
• Why Spark UI shows Jobs → Stages → Tasks.
• Why adding workers does not necessarily make a workload faster.

---

1. The complete execution hierarchy

Level	What it represents
Application	Driver + executors for one Spark application
Job	Computation triggered by an action
Stage	Set of tasks separated by dependency/shuffle boundaries
Task	Scheduled unit of work for one partition
Partition	Logical slice of distributed data

The relationship is:

Application
│
├── Job 0
│   ├── Stage 0
│   │   ├── Task 0 → Partition 0
│   │   ├── Task 1 → Partition 1
│   │   └── Task 2 → Partition 2
│   │
│   └── Stage 1
│       ├── Task 3 → Partition 0
│       ├── Task 4 → Partition 1
│       └── Task 5 → Partition 2
│
└── Job 1
    └── ...

A task is the runtime unit; a partition is the data unit that the task processes.

---

2. What is a Spark job?

A job represents the computation required to execute an action.

Typical actions include:

df.count()
df.collect()
df.show()
df.write.format("delta").save(path)

For example:

df.filter("amount > 100").count()

The filter is a transformation. The count is an action.

The action causes Spark to execute the required lineage:

filter()
   |
   | transformation
   v
count()
   |
   | action
   v
JOB

Apache Spark defines a job in terms of an action and the tasks required to evaluate that action.
Apache Spark — Job Scheduling

---

3. One notebook can create many jobs

Consider:

df.count()
df.show()
df.write.saveAsTable("catalog.schema.sales")

Conceptually:

Application
│
├── Job 0 → count()
├── Job 1 → show()
└── Job 2 → write()

Do not assume:

One notebook = one job.

A notebook can submit many Spark jobs.

This matters when diagnosing:

• Many small jobs.
• Driver overhead.
• Job scheduling delays.
• Repeated scans.
• Repeated transformations.
• Expensive actions accidentally executed multiple times.

---

4. A transformation does not normally create a job

This code:

filtered = df.filter("amount > 100")

builds lineage without immediately executing the full dataset.

Then:

filtered.count()

causes execution.

Mental model:

Transformation
     |
     v
Lineage / plan
     |
     | lazy evaluation
     v
Action
     |
     v
Job

This is the connection between lazy evaluation and the Job concept.

---

5. What is a stage?

A stage is a set of tasks that can execute within a dependency boundary.

The most useful practical rule is:

A shuffle boundary normally separates stages.

Example:

Read
  |
Filter
  |
Select
  |
GroupBy
  |
SHUFFLE
  |
Aggregation
  |
Write

Conceptually:

Stage 0
Read → Filter → Select → Shuffle Write
                         |
                         v
                    SHUFFLE
                         |
                         v
Stage 1
Shuffle Read → Aggregate → Write

This is why one action can produce multiple stages.

---

6. Why does Spark need stages?

Spark wants to execute as much work as possible without moving data unnecessarily.

Suppose:

Partition 0 → filter → projection
Partition 1 → filter → projection
Partition 2 → filter → projection

Each partition can perform these operations independently.

Now consider:

grouped = df.groupBy("customer_id").count()

Records for one customer may exist in many partitions.

Spark needs to bring related records together:

Partition 0 ──┐
Partition 1 ──┼──> SHUFFLE ──> grouped partitions
Partition 2 ──┘

That redistribution becomes a natural stage boundary.

---

7. What is a task?

A task is the smallest unit of execution scheduled by Spark for a stage.

A useful approximation is:

1 task ≈ 1 partition for 1 stage

Suppose Stage 0 has 500 input partitions.

500 partitions
      ↓
~500 tasks

Those tasks execute concurrently according to available task slots.

---

8. Task concurrency

Suppose compute has:

8 workers
8 cores per worker

A simplified capacity model is:

8 × 8 = 64 task slots

If a stage has 640 tasks:

Round 1 → ~64 tasks
Round 2 → ~64 tasks
Round 3 → ~64 tasks
...

The exact behavior depends on scheduling, task resource requirements and runtime conditions.

The key idea:

Tasks are scheduled onto available execution capacity; they are not all required to run simultaneously.

---

9. More compute does not create missing parallelism

Imagine:

Executors: 8
Cores: 8 each
Partitions: 2

The stage has only two units of partition-level work.

Even if the cluster has many more task slots, most capacity can remain idle.

Therefore:

More compute cannot create parallel work that the stage does not contain.

---

10. Narrow dependency

A narrow dependency means each child partition depends on a limited set of upstream partitions.

Common examples include:

• filter
• select
• withColumn
• Many projection/map-style transformations

Conceptually:

Parent Partition 0 ──> Child Partition 0
Parent Partition 1 ──> Child Partition 1
Parent Partition 2 ──> Child Partition 2

No broad redistribution is required.

This allows Spark to pipeline multiple operations inside the same stage.

---

11. Wide dependency

A wide dependency requires data from multiple upstream partitions to be redistributed.

Typical examples include:

• groupBy
• Many joins
• distinct
• orderBy
• repartition

Conceptually:

P0 ──┐
P1 ──┼──> Shuffle ──> Q0
P2 ──┤              Q1
P3 ──┘              Q2

Redistribution can involve:

• Network transfer
• Serialization
• Disk I/O
• Memory pressure
• Shuffle files
• Additional scheduling

---

12. Narrow vs wide

Characteristic	Narrow	Wide
Data movement	Limited	Redistribution
Shuffle	Usually no	Usually yes
Stage boundary	Usually no	Usually yes
Examples	filter, select	groupBy, join, orderBy
Network pressure	Lower	Potentially high
Performance sensitivity	Moderate	High

Useful shorthand:

Narrow = data can stay local.

Wide = data must move.

---

13. Example: filter only

result = (
    df
    .filter("country = 'IN'")
    .select("customer_id", "amount")
)

Conceptually:

Stage 0
│
├── Read
├── Filter
└── Select

The operations can be pipelined without a global redistribution.

---

14. Example: filter + groupBy

result = (
    df
    .filter("country = 'IN'")
    .groupBy("customer_id")
    .sum("amount")
)

Conceptually:

Stage 0
│
├── Read
├── Filter
└── Shuffle Write
       |
       v
    SHUFFLE
       |
       v
Stage 1
│
├── Shuffle Read
└── Aggregate

The filter can reduce the amount of data entering the shuffle.

Filter and project early when they reduce the data entering an expensive shuffle.

---

15. Example: join

orders.join(customers, "customer_id")

A shuffle-based join can look like:

Orders partitions ───┐
                     ├──> Shuffle → Join
Customers partitions ┘

But Spark can choose a different physical strategy when a suitable broadcast is possible.

Therefore:

The logical operation is not enough to predict the exact physical execution.

Inspect the physical plan and Spark UI.

---

16. Broadcast join changes the picture

Suppose the customer table is small enough to broadcast.

Small Customers
       |
    Broadcast
       |
 +-----+-----+-----+
 |     |     |     |
 v     v     v     v
E1    E2    E3    E4
 |     |     |     |
Orders partitions
       |
      Join

Instead of shuffling both sides, Spark can distribute the small side to executors.

This can remove a large shuffle.

Later we will cover broadcast thresholds, hints, failures and memory implications in performance engineering.

---

17. Stage pipelining

Spark can combine multiple narrow operations into one stage.

Read
 ↓
Filter
 ↓
Select
 ↓
withColumn
 ↓
Map
 ↓
Shuffle

Conceptually:

ONE STAGE
----------------------------
Read → Filter → Select
             → withColumn
             → Map
----------------------------
              |
            Shuffle
              |
              v
          NEXT STAGE

Therefore:

The number of transformations in your code is not the number of stages.

---

18. Why physical plan matters

For DataFrame/Spark SQL workloads, inspect:

result.explain("formatted")

Look for operators such as:

• Scan
• Filter
• Project
• Exchange
• Sort
• BroadcastHashJoin
• SortMergeJoin
• Aggregate
• Window
• WholeStageCodegen

An Exchange is particularly important because it represents data redistribution in the physical plan.

Example:

Filter
  |
Project
  |
Exchange
  |
HashAggregate

This tells you much more than counting lines of PySpark code.

---

19. Jobs, stages and tasks in Spark UI

The hierarchy appears directly in Spark UI:

JOB
 |
 +-- Stage 0
 |    +-- Tasks
 |
 +-- Stage 1
 |    +-- Tasks
 |
 +-- Stage 2
      +-- Tasks

The Spark Web UI Jobs tab shows job status, duration, event timeline, DAG visualization and associated stages. Stage details expose task-level metrics including duration, shuffle read/write, scheduler delay, GC time and spill.
Apache Spark — Web UI

Databricks recommends starting performance diagnosis from the Jobs Timeline, then opening the longest stage and checking skew, spill and I/O.
Databricks — Diagnose cost and performance issues using Spark UI

---

20. Stage IDs are runtime identifiers

Do not interpret:

Stage 0 = first business step
Stage 1 = second business step

as a permanent semantic contract.

When debugging, focus on:

• Stage description
• Associated SQL
• Physical plan
• Input/output
• Shuffle
• Task distribution
• Duration

---

21. Why stages can be skipped

Spark can avoid recomputing work when required data is already available through mechanisms such as caching or checkpointing.

The Spark UI can show skipped stages.

The important lesson is:

The DAG describes required computation, not necessarily work that must execute again every time.

---

22. Task retry

Spark can retry failed tasks.

Task 17
   |
 Failure
   |
 Retry
   |
 +--+--+
 |     |
Success Failure
         |
       Retry / stage failure

This resilience helps Spark tolerate transient failures.

However:

Retries are not free.

Repeated failures can dramatically increase runtime and cost.

---

23. Speculative execution

One task can become a straggler:

Task 0   12 sec
Task 1   11 sec
Task 2   13 sec
Task 3   10 sec
Task 4   7 min   ← straggler

Spark supports speculative execution, where a slow task can be relaunched when configured and eligible.

Current Spark configuration documents spark.speculation and related thresholds for speculative task execution.
Apache Spark — Configuration

Speculation is not a substitute for fixing:

• Data skew
• Slow storage
• Bad partitioning
• Infrastructure problems

---

24. One huge partition can dominate a stage

Suppose:

Task 0 → 500 MB → 20 sec
Task 1 → 480 MB → 19 sec
Task 2 → 510 MB → 21 sec
Task 3 → 50 GB  → 18 min

Most tasks finish quickly.

The stage still waits for the long task.

Therefore:

Stage completion time is strongly influenced by its slowest outstanding tasks.

This is why task duration distribution is more useful than average task duration alone.

---

25. The straggler pattern

A Spark UI task distribution may look like:

20s  █████
21s  █████
19s  ████
22s  █████
18s  ████
9m   █████████████████████████████

Investigate:

1. Partition size.
2. Key distribution.
3. Shuffle read.
4. Spill.
5. GC.
6. Input source.
7. Executor health.

Do not start by increasing the entire cluster.

---

26. Shuffle is the bridge between stages

        STAGE 0
P0 ─┐
P1 ─┼── transformations ──> shuffle write
P2 ─┤
P3 ─┘
              |
              | network / disk
              v
        STAGE 1
       shuffle read
              |
        aggregation/join

The shuffle redistributes data according to downstream partitioning requirements.

The next tutorial will go much deeper into this mechanism.

---

27. Shuffle read and shuffle write

Shuffle write

Data produced by upstream tasks for downstream consumption.

Shuffle read

Data fetched by downstream tasks from shuffle output.

Large shuffle does not automatically mean the query is wrong.

But large shuffle can indicate:

• Expensive join
• Large aggregation
• Poor filtering
• Excessive repartitioning
• Data layout issues
• Poor partition strategy

Databricks identifies high shuffle read/write as a signal that a stage spends significant time moving data between executors.
Databricks — Spark stage high I/O

---

28. Complete example

orders = spark.table("sales.orders")
customers = spark.table("sales.customers")

result = (
    orders
    .filter("order_date >= '2026-01-01'")
    .select("customer_id", "amount")
    .groupBy("customer_id")
    .sum("amount")
    .join(customers, "customer_id")
)

result.write.mode("overwrite").saveAsTable("sales.customer_summary")

Do not mentally execute this line by line.

Think in execution boundaries:

JOB
 |
 +-- Stage 0
 |    Read orders
 |      ↓
 |    Filter
 |      ↓
 |    Project
 |      ↓
 |    Shuffle
 |
 +-- Stage 1
 |    Shuffle read
 |      ↓
 |    Aggregate
 |      ↓
 |    Join strategy
 |
 +-- downstream work
      ↓
     Write

The exact stage count depends on the physical plan and chosen join/write strategy.

Never memorize a fixed stage count for a SQL statement. Read the plan.

---

29. Performance interpretation

When a job is slow, ask in this order:

Question 1 — Which job?

Maybe the wrong action is expensive.

Question 2 — Which stage?

Find the longest stage.

Question 3 — Why is that stage slow?

Check:

• High input
• High output
• High shuffle
• Spill
• Skew
• Low I/O but high CPU
• Small files
• UDFs
• Cartesian/exploding operations

Question 4 — Which tasks are slow?

Compare task durations.

Question 5 — Which partitions are problematic?

Look for data imbalance.

This creates a disciplined debugging process.

---

30. Databricks performance workflow

Current Databricks guidance can be summarized as:

Slow / expensive workload
          |
          v
     Jobs Timeline
          |
          v
     Longest Stage
          |
      +---+---+
      |       |
    Skew    Spill
      |       |
      +---+---+
          |
      High I/O?
       /     \
     YES      NO
      |        |
  Read/write   SQL / plan /
  optimization  UDF / join etc.

Databricks — Diagnose cost and performance issues using Spark UI

---

31. Hands-on lab — one action, one job

Create:

df = spark.range(0, 1_000_000)
df.count()

Open Spark UI and record:

Item	Observation
Job ID
Job duration
Number of stages
Number of tasks
Input
Output

Then run:

df.show()

Observe that another action can create another job.

---

32. Hands-on lab — introduce a shuffle

Run:

df = spark.range(0, 10_000_000)

result = (
    df
    .withColumn("bucket", df.id % 100)
    .groupBy("bucket")
    .count()
)

result.show()

Open Spark UI.

Look for:

• Multiple stages
• Shuffle write
• Shuffle read
• Task distribution

Then:

result.explain("formatted")

Look for an Exchange.

---

33. Hands-on lab — observe task parallelism

Run:

df = spark.range(0, 10_000_000)

print("Partitions:", df.rdd.getNumPartitions())
df.count()

Record:

• Partition count
• Number of tasks
• Executor count
• Task duration distribution

Then experimentally compare:

df2 = df.repartition(200)

print("Partitions:", df2.rdd.getNumPartitions())
df2.count()

Do not conclude that 200 partitions is automatically better.

The goal is to observe:

Partitions
    ↓
Tasks
    ↓
Scheduling
    ↓
Parallelism
    ↓
Runtime

---

34. Hands-on lab — find a straggler

Use a dataset with deliberately imbalanced key distribution.

Conceptually:

customer_id = 1       → millions of rows
customer_id = 2..N    → relatively few rows

Run a grouping operation and inspect task duration.

Identify:

• Longest task
• Shuffle read
• Spill
• Partition imbalance

Do not fix it yet.

We will solve this systematically in the partitions and shuffle topics.

---

35. Hands-on lab — physical plan

Run:

query = (
    spark.table("sales.orders")
    .filter("amount > 100")
    .groupBy("customer_id")
    .sum("amount")
)

query.explain("formatted")

Create this analysis table:

Operator	What to ask
Scan	How much data is read?
Filter	Is pruning possible?
Exchange	Why is data moving?
Aggregate	How much state is maintained?
Sort	Why is ordering required?

This builds the habit of connecting:

code → plan → stages → tasks

---

36. Common mistakes

Mistake 1 — “One transformation equals one stage”

False.

Multiple narrow operations can be pipelined into one stage.

Mistake 2 — “One action equals one stage”

False.

One action creates a job, and the job can contain multiple stages.

Mistake 3 — “One task equals one executor”

False.

An executor can run many tasks over time and multiple tasks concurrently depending on resources.

Mistake 4 — “One task equals one row”

False.

A task normally processes a partition containing many rows.

Mistake 5 — “More workers always means faster”

False.

If the stage has too little parallelism, extra workers can sit idle.

Mistake 6 — “Every shuffle means the query is bad”

False.

Some operations inherently require redistribution.

The goal is to minimize unnecessary shuffle, not eliminate all shuffle.

Mistake 7 — “Average task time is enough”

False.

A single straggler can dominate stage completion time.

---

37. Architect-level scenario

Situation

A job has:

Stage 0 → 2 minutes
Stage 1 → 25 minutes
Stage 2 → 1 minute

Stage 1 has:

Task 0   → 35 sec
Task 1   → 37 sec
Task 2   → 40 sec
...
Task 199 → 22 min

Architect diagnosis

Do not immediately increase cluster size.

Ask:

1. Is Stage 1 a shuffle stage?
2. What is the partition size of Task 199?
3. Is there key skew?
4. Is there spill?
5. How much shuffle read did that task perform?
6. Is the task on a healthy executor?
7. Is the data distribution itself unbalanced?

Likely root causes:

• Data skew
• Oversized partition
• Spill
• Slow input
• Infrastructure anomaly

The architectural skill is:

Translate UI symptoms into a data-distribution hypothesis.

---

38. Architect mental model

When looking at Spark code, mentally translate:

PYSPARK
   ↓
LOGICAL PLAN
   ↓
PHYSICAL PLAN
   ↓
EXCHANGES / DEPENDENCIES
   ↓
STAGES
   ↓
TASKS
   ↓
PARTITIONS
   ↓
EXECUTORS
   ↓
CPU / MEMORY / NETWORK / STORAGE

This is the bridge from developer thinking to Spark architect thinking.

---

39. Interview questions

Q1. What is a Spark job?

The computation triggered by an action and the tasks required to evaluate it.

Q2. What is a stage?

A set of tasks separated from other execution regions by shuffle/dependency boundaries.

Q3. What is a task?

A scheduled unit of work that normally processes one partition for a stage.

Q4. Why does groupBy often create another stage?

Because records may need to be redistributed across partitions, requiring a shuffle.

Q5. What is a narrow dependency?

A dependency where downstream partitions depend on limited upstream partitions and broad redistribution is not required.

Q6. What is a wide dependency?

A dependency requiring data redistribution across partitions, commonly involving shuffle.

Q7. Can one job have multiple stages?

Yes.

Q8. Can one notebook create multiple jobs?

Yes. Each action can trigger a separate job.

Q9. Why can one task be much slower than others?

Data skew, oversized partitions, spill, slow I/O, GC or infrastructure problems.

Q10. How do you find the bottleneck?

Identify the slow job, longest stage, slow tasks, then inspect input/output, shuffle, spill, skew and the physical plan.

Q11. What is an Exchange in a physical plan?

A physical operator representing redistribution of data, commonly associated with a shuffle boundary.

Q12. Why does adding workers not always help?

Because performance depends on available parallel work, partition balance, dependencies and the actual bottleneck.

---

40. Final cheat sheet

ACTION
  |
  v
JOB
  |
  +------------------+
  |                  |
  v                  v
STAGE 0            STAGE 1
  |                  |
TASKS              TASKS
  |                  |
PARTITIONS         PARTITIONS
  |                  |
EXECUTORS          EXECUTORS

Wide dependency / shuffle
        = stage boundary

Narrow dependency
        = can often pipeline in same stage

Remember

Action → Job

Shuffle boundary → Stage boundary

Partition → Task

Task → Executor

Slowest task → often controls stage completion

Physical plan → source of truth

---

Official references

• Apache Spark — Job Scheduling
• Apache Spark — Web UI
• Apache Spark — Configuration
• Databricks — Diagnose cost and performance issues using Spark UI
• Databricks — Spark stage high I/O

Next

03.3 Transformations — understand how Spark transformations build lineage, how narrow and wide transformations behave, which transformations introduce shuffles, and how to reason about their physical execution.