03.9 Spark Mental Model
Architect goal: Look at Spark code and predict its execution shape before opening the Spark UI.
1. The execution model
BUSINESS LOGIC → SPARK API → LOGICAL PLAN → PHYSICAL PLAN → DAG
↓
JOB → STAGES → TASKS
↓
PARTITIONS → EXECUTORS
↓
CPU / MEMORY / NETWORK / DISK
Spark has a driver that coordinates the application and executors that perform distributed computation. Transformations are lazy; actions trigger the required computation. Apache Spark RDD Programming Guide
2. Your code describes work; Spark decides execution
Read → Filter → Select → GroupBy → Aggregate → Write
Do not think of these Python calls as immediate sequential processing. Spark builds an execution representation, optimizes it, and executes a physical plan across partitions.
Mental shift: a DataFrame represents a distributed computation over data; it is not the entire table sitting in Python memory.
3. Driver, executors, partitions, tasks
DRIVER
planning + coordination
│
┌────────────┼────────────┐
↓ ↓ ↓
EXECUTOR 1 EXECUTOR 2 EXECUTOR 3
tasks tasks tasks
│ │ │
partitions partitions partitions
The driver plans and coordinates. Executors process partitions and run tasks. Avoid moving large results to the driver with operations such as collect() or toPandas().
4. Transformations build; actions execute
Transformations such as filter, select, join, groupBy, and aggregation describe computation. Actions such as count, show, collect, and write require execution.
Transformations → logical work → ACTION → JOB
Spark's lazy evaluation allows the engine to optimize the computation before execution. RDD Programming Guide
5. One action can produce multiple stages
Stage 0
Scan → Filter → Partial Aggregate
↓
SHUFFLE
↓
Stage 1
Shuffle Read → Final Aggregate → Output
A job is an execution unit associated with an action. Stages are execution regions separated by dependencies that require redistribution.
6. Narrow vs wide dependency
Narrow
A child partition can be computed from a limited set of parent partitions. Filtering and many projection/map operations are typical examples.
P0 → P0'
P1 → P1'
P2 → P2'
Wide
The child partition needs data from multiple parent partitions. Grouping, many joins, repartitioning, and global ordering can require redistribution.
P0 ─┐
P1 ─┼→ SHUFFLE → Q0
P2 ─┤ → Q1
P3 ─┘ → Q2
Wide dependencies are why shuffle boundaries are central to Spark performance. RDD Programming Guide
7. DAG, logical plan, and physical plan
These are different representations:
Source code → Logical intent → Optimized physical operators → Distributed execution DAG → Stages / tasks
The physical plan can contain operators introduced or rearranged by the optimizer. Therefore performance tuning requires both plan inspection and runtime evidence.
8. Partitions are the unit of distributed work
Dataset → Partition 0 / Partition 1 / ... / Partition N
↓
TASK
↓
EXECUTOR
Too few partitions can limit parallelism. Too many tiny partitions can create scheduling and file overhead. The correct shape depends on data volume, distribution, operators, cluster resources, and runtime behavior.
9. Shuffle changes the execution shape
Upstream partitions → SHUFFLE → New partition layout → Downstream tasks
Shuffle can consume network, CPU, memory, and disk resources. It is therefore one of the most important performance boundaries.
10. AQE means runtime behavior can adapt
Adaptive Query Execution can use runtime statistics to improve execution. Depending on workload and runtime, adaptive behavior can include coalescing shuffle partitions, changing join strategies, and handling skew.
Initial physical plan → Runtime statistics → Adaptive decision → Continue execution
The plan inspected before execution is not necessarily the complete story of runtime behavior.
11. Why identical code can behave differently
Runtime behavior depends on data volume, data distribution, file layout, partition size, join cardinality, skew, statistics, runtime version, cluster resources, adaptive execution, and concurrent workloads.
Same code + different data = different execution behavior
12. Four resource dimensions
| Bottleneck | Evidence | Investigate |
|---|---|---|
| CPU | High CPU, balanced tasks | Expensive expressions, UDFs, compute capacity |
| Memory | Spill, GC, pressure | Partition size, aggregation, joins, caching |
| Network | Large shuffle, remote reads | Join strategy, repartitioning, data movement |
| Storage/I/O | High scan/read time | Files, pruning, layout, storage access |
A workload can move between bottleneck types at different stages.
13. Complete execution example
Read
↓
Filter
↓
Projection
↓
Partial aggregation
↓
SHUFFLE
↓
Final aggregation
↓
Write
Distributed execution becomes:
Driver
↓
Stage 0 → tasks → input partitions
↓
Shuffle
↓
Stage 1 → tasks → shuffle partitions
↓
Write
This is the execution picture to carry into PySpark.
14. Reduce data before expensive operations
If a valid filter reduces 100 TB to 5 TB before a shuffle, the downstream operation can potentially process dramatically less data.
100 TB → Filter → 5 TB → GroupBy → Shuffle
The same principle applies to column pruning, valid pre-aggregation, and avoiding unnecessary repartitioning.
15. Joins deserve special attention
Depending on data size, statistics, skew, and runtime behavior, a join may use a broadcast-style strategy or redistribute both sides.
Small side → Broadcast → Large side
versus:
Large side → Shuffle → Join
Small side → Shuffle → Join
Do not memorize that broadcast is always faster. Ask whether the smaller side can safely be broadcast and whether runtime evidence supports it.
16. Fault tolerance and caching
If a task fails, Spark can retry work for the affected partition. The dependency/lineage model allows lost intermediate results to be recomputed when required. RDD Programming Guide
Appropriate persistence allows repeated actions to reuse computed partitions:
Compute → Persist → Action A
└→ Action B
Caching is an optimization, not a default requirement. Use it when reuse justifies the memory/storage cost.
17. The Spark UI now becomes intuitive
When the UI shows one task taking far longer than its peers, reason backward:
Slow task → abnormal partition? → skewed key/input? → shuffle/operator? → data-model or algorithm decision
The UI provides evidence; the architect determines the cause.
18. Five-question diagnostic framework
- What data is being processed? Volume, columns, files, distribution.
- How is it partitioned? Count, size, balance, partitioning key.
- Where does data move? Shuffle, joins, repartition, sort, Exchange.
- Where is time spent? Jobs → stages → tasks → operators.
- Which resource limits progress? CPU, memory, network, or storage.
These five questions turn Spark performance investigation into a repeatable method.
19. Architect execution canvas
BUSINESS REQUIREMENT
↓
SOURCES / VOLUME / DISTRIBUTION
↓
LOGICAL TRANSFORMATIONS
↓
PHYSICAL OPERATORS
↓
SHUFFLE BOUNDARIES
↓
STAGES → TASKS → PARTITIONS
↓
EXECUTORS / CPU / MEMORY / NETWORK / DISK
↓
OBSERVABILITY / SPARK UI
↓
OPTIMIZATION / ARCHITECTURAL DECISION
20. Hands-on lab — predict before running
df = spark.read.table("sales")
result = (df.filter("amount > 100")
.select("customer_id", "amount")
.groupBy("customer_id")
.sum("amount"))
result.count()
Before running, predict:
| Question | Your prediction |
|---|---|
| Actions | |
| Jobs | |
| Shuffle location | |
| Wide dependency | |
| Expected stages | |
| What determines task count? | |
| Metric that reveals skew | |
| Physical operators |
Then compare your prediction with the Spark UI and physical plan. The goal is to learn to predict execution shape.
21. Hands-on lab — controlled optimization
Create a baseline, change exactly one thing, and rerun.
Run A: runtime / shuffle / spill / P50 / P95 / MAX
Run B: runtime / shuffle / spill / P50 / P95 / MAX
Possible experiments: reduce columns before shuffle, filter earlier where valid, change join strategy, adjust partitioning, or address skew.
Document not only whether it became faster, but why.
22. Common misconceptions
- Every transformation immediately creates a physical dataset — false; transformations are lazy.
- Every transformation creates a stage — false; execution dependencies determine stage boundaries.
- More executors always improve performance — false; skew and I/O bottlenecks can remain.
- More partitions are always better — false; excessive partitions create overhead.
- The SQL plan tells the whole story — false; runtime evidence matters.
- Caching everything improves performance — false; caching has a cost.
23. Interview questions
- Explain Spark in one minute.
- Explain DataFrame → logical plan → physical plan → DAG → stage → task → partition.
- Why are transformations lazy?
- What triggers a job?
- What creates a stage boundary?
- Explain narrow vs wide dependency.
- Why does shuffle cost so much?
- Why can adding executors fail to fix skew?
- How do partitions affect parallelism?
- How does AQE affect execution?
- Walk through a 10 TB aggregation.
- How would you design a pipeline to minimize data movement?
- When would you broadcast a dimension?
- How would you distinguish a code problem from a data-distribution problem?
- How would you prove an optimization worked?
24. Final mental model
DRIVER
↓
ACTION → JOB
↓
STAGES
↓
TASKS
↓
PARTITIONS
↓
EXECUTORS
↓
CPU / MEMORY / NETWORK / DISK
↓
RUNTIME METRICS
↓
ROOT CAUSE
↓
ARCHITECTURAL CHANGE
Performance engineering is a loop:
DESIGN → EXECUTE → OBSERVE → DIAGNOSE → OPTIMIZE → MEASURE
25. Spark fundamentals complete
- 03.0 Spark Architecture
- 03.1 Driver & Executors
- 03.2 Jobs, Stages & Tasks
- 03.3 Transformations
- 03.4 Actions & Lazy Evaluation
- 03.5 DAG
- 03.6 Partitions
- 03.7 Shuffle
- 03.8 Spark UI
- 03.9 Spark Mental Model
Next: 04.1 PySpark DataFrames
Official references: Apache Spark RDD Programming Guide · Apache Spark Web UI · Spark SQL Performance Tuning