Skip to main content

03.5 DAG

Phase 3 — Apache Spark Fundamentals

A DAG (Directed Acyclic Graph) is the dependency model Spark uses to reason about distributed computation. It connects transformations and lineage to stages, tasks and executors.

Spark DAG architecture

1. The mental model​

Think about Spark execution as:

User code
   ↓
Transformations
   ↓
Lineage / dependency graph
   ↓
DAG
   ↓
Stages
   ↓
Tasks
   ↓
Executors

The DAG is directed because dependencies flow from upstream to downstream. It is acyclic because execution does not contain circular dependency paths.

2. Why Spark needs a DAG​

A distributed computation cannot simply be executed as a sequence of Python statements. Spark needs to determine:

  • which operations depend on other operations;
  • which operations can be pipelined;
  • where data must move between partitions;
  • where stage boundaries occur;
  • which tasks can execute concurrently;
  • what must be recomputed after a failure.

That dependency structure is the foundation for scheduling.

3. Lineage vs DAG​

Lineage describes how a result depends on source data and previous transformations. The DAG expresses those dependencies as a graph that can be scheduled.

Do not think of the DAG as simply a picture of Python code. It is an execution-oriented representation of dependencies.

4. Vertices and edges​

A simplified graph is:

[Read Orders]
      ↓
   [Filter]
      ↓
   [Project]
      ↓
 [Aggregate]
      ↓
   [Result]

Nodes represent computation/data representations and edges represent dependencies. Spark's Web UI exposes DAG visualizations for jobs and stages. Spark Web UI

5. Narrow dependencies​

A narrow dependency allows a child partition to be computed from a limited parent-partition set without global redistribution.

Parent P0 ───→ Child P0
Parent P1 ───→ Child P1
Parent P2 ───→ Child P2
Parent P3 ───→ Child P3

Typical examples include filtering and projection/map-style operations.

Because no global redistribution is required, compatible narrow operations can often be pipelined inside the same stage.

6. Wide dependencies​

A wide dependency requires a child partition to obtain data from multiple parent partitions.

P0 ──┐
P1 ──┼──→ C0
P2 ──┼──→ C1
P3 ──┘

Typical examples include groupBy, many joins, repartition and global sort operations.

Wide dependencies normally require shuffle, which redistributes data across executors.

7. Shuffle and stage boundaries​

A simplified execution graph:

Stage 0
Read → Filter → Project → Shuffle Write
                              │
                              ▼
Stage 1
Shuffle Read → Aggregate → Result

The critical principle is:

A shuffle is a major dependency boundary and commonly separates stages.

Stages are not Python statements. Several transformations can execute inside one stage when their dependencies allow pipelining.

8. DAG → stages → tasks​

A useful model is:

ACTION
  ↓
JOB
  ↓
DAG
  ↓
STAGES
  ↓
TASKS
  ↓
EXECUTORS

Spark's job scheduling documentation describes jobs as being divided into stages and stages into tasks. Spark Job Scheduling

9. Exchange in the physical plan​

When analyzing DataFrame or SQL execution, use:

df.explain("formatted")

Pay particular attention to Exchange operators. They are strong indicators of data redistribution.

Examples conceptually include:

Exchange hashpartitioning(...)
Exchange rangepartitioning(...)

For every Exchange ask:

  1. Why is redistribution required?
  2. How much data is moving?
  3. Can filtering happen earlier?
  4. Is the partitioning appropriate?
  5. Is the join or aggregation strategy appropriate?
  6. Is key skew creating oversized partitions?

10. DAG and partition parallelism​

If a stage has 1,000 partitions, Spark can create roughly one task per partition for that stage.

Stage
 ├─ Task 0  → Partition 0
 ├─ Task 1  → Partition 1
 ├─ Task 2  → Partition 2
 └─ ...

Actual concurrency depends on available task slots and resources.

DAG structure determines dependency order; partitioning determines much of the available parallelism.

11. DAG and fault tolerance​

Suppose one task fails:

Stage 0
 ├─ Task 0 ✓
 ├─ Task 1 ✓
 ├─ Task 2 ✗
 └─ Task 3 ✓

Spark can retry failed tasks and, when required, recompute upstream data using lineage. This is a fundamental part of Spark's fault-tolerant execution model.

12. DAG and caching​

Without persistence, a later action may require upstream computation again:

Source → Filter → Join → Aggregate → Action A
                     
Source → Filter → Join → Aggregate → Action B

With persistence, a materialized reuse point can change the execution path:

Source → Filter → Join → Aggregate → CACHE
                                      ├→ Action A
                                      └→ Action B

Cache is therefore a performance/reuse mechanism; it does not remove the need to understand the DAG.

13. Architect reading order​

When a production job is slow, read the execution graph in this order:

  1. Input — how much data is scanned?
  2. Filters — can selective predicates reduce work early?
  3. Projection — are unnecessary columns removed?
  4. Joins — broadcast, shuffle, cardinality and skew?
  5. Exchange — where is data redistributed?
  6. Aggregation/sort/window — what expensive global operations exist?
  7. Tasks — are task durations balanced?
  8. Output — is there a driver-side boundary or inefficient write?

14. Example: expensive DAG​

S3 / Delta
    ↓
Scan 5 TB
    ↓
Join customer
    ↓
Exchange
    ↓
GroupBy
    ↓
Exchange
    ↓
Sort
    ↓
Write Gold

Do not immediately increase cluster size. An architect first asks why 5 TB is being scanned, whether filters can move earlier, whether a broadcast join is appropriate, whether the join is skewed, why the Exchanges exist, and whether the Gold model can eliminate repeated work.

15. DAG anti-patterns​

  • Treating every transformation as a stage.
  • Assuming every shuffle is bad.
  • Increasing compute before understanding data movement.
  • Ignoring skew because the DAG shape looks normal.
  • Ending a distributed pipeline with an unnecessary collect/toPandas.
  • Optimizing for fewer stages instead of efficient execution.

The objective is not the smallest-looking DAG. The objective is less unnecessary work and data movement.

16. Hands-on lab — create a DAG​

from pyspark.sql import functions as F

orders = spark.range(0, 10_000_000).withColumn(
    "customer_id", F.col("id") % 100_000
)

result = (
    orders
    .filter(F.col("id") > 1_000_000)
    .groupBy("customer_id")
    .agg(F.count("*").alias("order_count"))
)

result.count()

Open the Spark UI and inspect Jobs, the DAG visualization, stages, task counts, shuffle read and shuffle write.

17. Hands-on lab — find Exchange​

result.explain("formatted")

For every Exchange, document the operation causing redistribution and estimate whether the movement is necessary.

18. Hands-on lab — skew​

skewed = (
    spark.range(0, 10_000_000)
    .withColumn(
        "key",
        F.when(F.col("id") < 9_000_000, F.lit("HOT"))
         .otherwise(F.col("id"))
    )
)

skewed.groupBy("key").count().show()

Inspect task duration and partition distribution. The purpose is to understand how a structurally correct DAG can still contain a severe straggler.

19. Production troubleshooting​

Slow job
   ↓
Find slow stage
   ↓
Inspect DAG
   ↓
Find Exchange / shuffle
   ↓
Inspect task distribution
   ├── one task much slower → skew / straggler
   └── all tasks slow       → scan / compute / resource issue

The Spark Web UI exposes stage-level shuffle, task duration, spill, executor and scheduling metrics that connect graph structure to runtime behavior. Spark Web UI

20. Architect scenario​

A production job has 20 stages, 3 TB input and 500 GB shuffle. In one stage, 199 tasks finish in 20 seconds while one task takes 18 minutes.

The immediate conclusion should not be “the DAG has too many stages.” Investigate:

  • key distribution;
  • partition sizes;
  • join/aggregation keys;
  • skew;
  • shuffle distribution;
  • executor logs;
  • spill and task metrics.

The likely optimization target is the straggling partition, not blindly reducing stage count.

21. Enterprise decision framework​

ObservationInvestigation direction
Large scanData/file pruning
Large ExchangeReduce unnecessary redistribution
Huge shuffleFilter/project earlier; revisit join/aggregation
One very slow taskSkew/straggler analysis
All tasks CPU-heavyOptimize expressions/compute
Many tiny tasksReview partition/file strategy
Driver-heavy final stepRemove collect/toPandas
Repeated DAG executionPersistence/materialization
Excessive sortCheck whether ordering is required
Slow joinJoin strategy, cardinality and skew

22. Architect mental model​

Memorize:

CODE
 ↓
TRANSFORMATIONS
 ↓
LINEAGE
 ↓
DAG
 ↓
DEPENDENCIES
 ↓
STAGE BOUNDARIES
 ↓
TASKS
 ↓
EXECUTORS
 ↓
DATA MOVEMENT
 ↓
RESULT

The most expensive part of a distributed system is often moving data, not executing the transformation itself.

This principle will repeatedly appear in Spark, Delta Lake and Databricks performance engineering.

23. Interview questions​

  1. What is a DAG? A directed acyclic graph representing dependencies between computation steps.
  2. Why does Spark use a DAG? To understand dependencies, pipeline compatible operations and determine executable stages.
  3. What creates a stage boundary? A dependency requiring an intermediate redistribution/materialization, commonly a shuffle.
  4. What is a narrow dependency? A dependency where a child partition depends on a limited parent partition set without global redistribution.
  5. What is a wide dependency? A dependency where child partitions depend on multiple parent partitions, typically requiring shuffle.
  6. Is every transformation a new stage? No.
  7. Is every shuffle bad? No. Many correct algorithms require redistribution.
  8. What is Exchange? A physical-plan operator indicating data redistribution.
  9. How do you troubleshoot a slow DAG? Identify the slow stage, inspect shuffle and task distribution, then investigate skew, scan, compute and resources.
  10. Why can one task make a stage slow? The stage cannot complete until its required tasks finish; a straggler can dominate elapsed time.
  11. How does caching affect a DAG? It provides a materialized reuse point that can avoid repeating upstream computation.
  12. DAG vs physical plan? The physical plan describes execution operators; the DAG/UI exposes runtime dependency, stage and task structure.

24. Final cheat sheet​

NARROW DEPENDENCY
      ↓
PIPELINED OPERATIONS
      ↓
SAME STAGE

WIDE DEPENDENCY
      ↓
SHUFFLE / EXCHANGE
      ↓
STAGE BOUNDARY
      ↓
NEW TASK SET
  • DAG = dependency graph.
  • Narrow dependencies can usually be pipelined.
  • Wide dependencies usually require redistribution.
  • Shuffle is a major stage boundary.
  • Exchange is a key physical-plan clue.
  • DAG structure and runtime metrics must be analyzed together.
  • Optimize unnecessary work and data movement, not the visual number of boxes.

Official references​

Next​

03.6 Partitions — understand the fundamental unit of Spark parallelism: partition sizing, task parallelism, file partitions, repartition vs coalesce, skew, and production partition strategy.