03.8 Spark UI

Architect rule: Do not tune Spark from source code alone. Use the Spark UI to connect what the code requested to what the cluster actually executed.

1. Why the Spark UI matters

The Spark UI is the execution microscope for a Spark application. It exposes jobs, stages, tasks, executors, SQL execution details, storage, environment/configuration, and streaming information when applicable.

Code / SQL
   ↓
Logical plan
   ↓
Physical plan
   ↓
Jobs
   ↓
Stages
   ↓
Tasks on executors
   ↓
Runtime metrics

For troubleshooting, read the path in reverse:

Symptom → slow job → slow stage → task distribution
        → shuffle / spill / scan / join evidence → physical operator → root cause

Apache Spark documents the Web UI as a monitoring and diagnostic surface for application status, resource consumption, jobs, stages, storage, executors, SQL plans, and configuration. Spark Web UI

2. Opening Spark UI in Databricks

For a running Databricks workload, open the relevant compute resource and select Spark UI. Databricks recommends starting with the Jobs timeline and then drilling into the longest stage, skew/spill, I/O behavior, and other causes of slow execution. Databricks Spark UI guide

UI labels can evolve across Databricks Runtime releases. The execution concepts are more important than memorizing the exact buttons.

For Apache Spark applications, event logging plus the Spark History Server is the standard mechanism for reconstructing historical application UIs. Spark monitoring

3. Jobs tab — find where the time went

The Jobs view answers:

• What jobs executed?
• Which job is slow?
• How long did it run?
• Which stages belong to it?
• Were there scheduling gaps?
• What does the event timeline look like?

A DataFrame transformation does not execute immediately. An action such as count, show, or write triggers the execution.

result = orders.groupBy("customer_id").sum("amount")
result.count()

The event timeline is useful for identifying:

• Long idle gaps.
• Many tiny jobs.
• Unexpected serial execution.
• Repeated stages.
• Executor add/remove events.
• One stage dominating total runtime.

Databricks specifically recommends the Jobs timeline as the starting point for diagnosing slow workloads. Jobs timeline

4. Stages tab — locate the bottleneck

A job is normally too coarse for root-cause analysis. Find the stage that contributes most of the elapsed time.

Metric	Diagnostic meaning
Input	Data read by the stage
Output	Data written by the stage
Shuffle read	Data fetched for downstream processing
Shuffle write	Data produced for downstream processing
Task count	Degree of task-level parallelism
Duration	Stage elapsed time
Task distribution	Whether work is balanced

Apache Spark stage details also expose scheduler delay, GC time, peak execution memory, shuffle fetch wait, and shuffle spill metrics. Spark Web UI

First-pass rule

Total job runtime
      ↓
Longest stage
      ↓
Inspect task distribution
      ↓
Inspect shuffle / scan / spill / operators

Do not assume the longest stage is the root cause. It may be waiting on one skewed task, a large shuffle, slow storage, or memory pressure.

5. Tasks — where skew becomes visible

A stage consists of tasks. If 999 tasks finish quickly while one task runs for minutes, the stage can remain blocked by that straggler.

P50    ████                  8 sec
P75    █████                 11 sec
P95    ██████                14 sec
MAX    ███████████████████   190 sec  ← investigate

Compare:

• Median.
• 75th/95th percentile.
• Maximum.
• Input size per task.
• Shuffle read per task.
• Spill per task.

A large max-to-median ratio is a strong signal for skew or an abnormal partition.

Typical causes

• Hot join key.
• Highly concentrated customer/account/product.
• NULL-heavy key.
• Low-cardinality partitioning.
• Uneven source files.
• Very large window partition.

Adding workers does not automatically solve a hot partition; the oversized partition still has to be processed.

6. Shuffle metrics — follow the bytes

When a stage contains significant shuffle, inspect:

• Shuffle read.
• Shuffle write.
• Remote bytes read.
• Local bytes read.
• Fetch wait time.
• Shuffle spill.
• Task distribution.

Ask:

How many bytes are moving compared with how many bytes are ultimately useful?

Example:

Input:          800 GB
Pre-shuffle:    750 GB
Shuffle write:  700 GB
Shuffle read:   700 GB
Final output:    20 GB

Potential improvements include earlier filtering, column pruning, pre-aggregation, better join strategy, skew handling, avoiding unnecessary repartitioning, and better table/file layout.

7. Spill — when memory pressure becomes disk work

Shuffle, aggregation, and sort operators can require substantial execution memory. When intermediate state cannot remain in memory, Spark can spill data to disk.

Input
  ↓
Hash / sort / aggregation state
  ↓
Memory pressure
  ↓
Spill
  ↓
Disk I/O
  ↓
Merge / continue

High spill does not automatically mean “increase executor memory.”

First ask:

1. Is the partition too large?
2. Is there skew?
3. Is the aggregation unnecessarily wide?
4. Is a join producing excessive intermediate data?
5. Are too many objects being created?
6. Is an inefficient UDF or representation involved?

Only then decide whether compute sizing is part of the solution.

8. SQL tab — connect code to physical operators

The SQL view is critical for Spark SQL and DataFrame workloads. A DataFrame action can appear in SQL execution information even when the original code was not a SQL string. Spark Web UI

Look for operators such as:

• Scan
• Filter
• Project
• Exchange
• BroadcastExchange
• Sort
• HashAggregate
• SortMergeJoin
• BroadcastHashJoin
• Window

Exchange is a major clue

Scan
 ↓
Filter
 ↓
Exchange  ← redistribution
 ↓
Sort / Aggregate / Join

If the stage is slow and the plan contains a large Exchange, return to shuffle diagnosis.

9. Physical plan vs runtime evidence

Suppose the plan is:

SortMergeJoin
  ├── Exchange hashpartitioning(customer_id)
  │     └── Scan orders
  └── Exchange hashpartitioning(customer_id)
        └── Scan customers

The physical plan tells you what Spark intends to execute.

The UI tells you what happened while executing it:

• How much data was scanned?
• How much was shuffled?
• How much was remote?
• How long did sorting take?
• Was one task much slower?
• Did tasks spill?
• Was the distribution balanced?

Architecture-level performance engineering requires both views.

10. Executors tab — resource evidence

The Executors view helps answer:

Is the workload actually using the resources I provisioned?

Inspect:

• Active executors.
• Cores and memory.
• Disk usage.
• Task counts.
• Shuffle activity.
• GC time.
• Executor failures.
• Logs where available.

High GC can indicate excessive object creation, large in-memory structures, poorly behaved UDFs, over-caching, memory pressure, or object-heavy Python/JVM interactions.

Do not respond automatically with “add more executors.” First classify the workload as CPU-bound, memory-bound, I/O-bound, or skew-bound.

11. Storage tab — understand persistence

The Storage view is useful when RDDs or DataFrames are persisted. It can show cached objects, storage level, partitions, memory usage, and distribution across executors.

Caching should be justified by reuse.

Read huge dataset
      ↓
cache()
      ↓
used once
      ↓
memory consumed without enough reuse

The correct question is:

Will this dataset be reused enough to justify its storage footprint?

12. Environment tab — verify configuration

The Environment view is a configuration sanity check. Apache Spark documents it as a place to inspect effective Spark properties and runtime information. Spark configuration

Verify rather than assume:

spark.sql.shuffle.partitions
spark.sql.adaptive.enabled
spark.sql.autoBroadcastJoinThreshold
spark.executor.memory
spark.driver.memory

The effective configuration depends on the Databricks Runtime and workload configuration.

13. SQL metrics — operator-level evidence

Execution views expose useful metrics including:

• Number of output rows.
• Data size.
• Scan time.
• Metadata time.
• Shuffle bytes written.
• Remote/local bytes read.
• Fetch wait time.
• Sort time.
• Peak memory.
• Spill size.

Example:

Scan orders
  output rows = 9.8 billion

Filter
  output rows = 120 million

Exchange
  shuffle bytes = 2.4 TB

Aggregate
  output rows = 80 million

This is evidence that filtering earlier may reduce data crossing the Exchange, assuming the predicate can legally be pushed earlier.

14. Complete troubleshooting example

Symptom

A daily customer-revenue pipeline normally takes 12 minutes. Today it takes 48 minutes.

Step 1 — Jobs

Job 17
Duration: 48 min
Status: succeeded

Step 2 — Stages

Stage 31   4 min
Stage 32   5 min
Stage 33  38 min  ← investigate
Stage 34   1 min

Step 3 — Tasks

P50:  35 sec
P95:  42 sec
MAX:  36 min

That is a strong skew signal.

Step 4 — Shuffle

The maximum task has far more shuffle read than the median task.

Step 5 — SQL plan

Exchange hashpartitioning(customer_id)

Step 6 — Data inspection

One customer represents an unusually large fraction of the records.

Root cause

A hot customer key creates a skewed shuffle partition.

Correct response

Investigate AQE skew handling, join strategy, key salting where semantically appropriate, pre-aggregation, data-model changes, and partitioning strategy. Then benchmark again.

15. Second example — I/O-bound workload

Suppose:

Stage duration: 18 min
CPU utilization: modest
Scan time: dominant
Shuffle: low
Task distribution: balanced

This does not look like a shuffle problem.

Investigate:

• Number of files.
• File sizes.
• Partition pruning.
• Data skipping.
• Columns read.
• Storage latency.
• Table layout.
• Whether the query scans much more data than necessary.

16. Third example — too many tiny tasks

Suppose:

Input: 4 GB
Tasks: 18,000
Most tasks: milliseconds
Scheduler overhead: significant

Possible causes:

• Too many tiny input files.
• Excessive partitioning.
• Overly aggressive repartitioning.
• Poor upstream output layout.

A larger cluster is not automatically the answer. Improve file/partition layout when that is the bottleneck.

17. Performance investigation decision tree

JOB IS SLOW
   │
   ├── Long stage?
   │      └── inspect task distribution
   │             ├── outliers → skew / straggler
   │             └── all slow → shuffle / scan / CPU / spill
   │
   ├── High shuffle?
   │      └── inspect Exchange + join/aggregate strategy
   │
   ├── High spill?
   │      └── inspect partition size, skew, memory pressure
   │
   ├── High scan time?
   │      └── inspect pruning, files, layout, storage
   │
   └── High GC?
          └── inspect object/memory behavior

18. Hands-on lab — generate an observable workload

from pyspark.sql import functions as F

orders = (
    spark.range(0, 10_000_000)
    .withColumn("customer_id", (F.col("id") % 200_000).cast("long"))
    .withColumn("amount", (F.col("id") % 1000).cast("double"))
)

result = (
    orders
    .filter(F.col("amount") > 100)
    .groupBy("customer_id")
    .agg(
        F.count("*").alias("order_count"),
        F.sum("amount").alias("revenue")
    )
)

result.explain("formatted")
result.count()

Record:

Observation	Your result
Job ID
Longest stage
Number of tasks
Input bytes
Shuffle read
Shuffle write
Median task duration
Maximum task duration
Spill
Physical Exchange operators

The goal is to connect each metric to the execution model.

19. Hands-on lab — create obvious skew

skewed = (
    spark.range(0, 10_000_000)
    .withColumn(
        "customer_id",
        F.when(F.col("id") < 9_000_000, F.lit("HOT"))
         .otherwise(F.col("id").cast("string"))
    )
)

skewed.groupBy("customer_id").count().count()

Inspect Jobs, the longest stage, task duration distribution, shuffle read, the maximum task, and the physical plan. Explain why one key can dominate the stage.

20. Hands-on lab — controlled optimization

Create a baseline and change one thing.

Run A
Runtime:
Shuffle:
Spill:
P50:
P95:
MAX:

Run B
Runtime:
Shuffle:
Spill:
P50:
P95:
MAX:

Possible single changes:

• Reduce columns before a shuffle.
• Filter earlier.
• Change join strategy.
• Adjust partitioning.
• Address skew.

Calculate runtime improvement and shuffle reduction. This turns performance tuning into an engineering experiment rather than guesswork.

21. What the UI cannot tell you by itself

The UI can tell you that a partition is huge; it cannot automatically tell you the business reason.

UI says: Partition 317 is huge
        ↓
Which key caused it?
        ↓
Why is that key concentrated?
        ↓
Is the concentration expected?
        ↓
Should the algorithm or data model change?

Architect-level tuning connects runtime symptoms to data-model and workload decisions.

22. Common mistakes

• Looking only at total runtime.
• Looking only at average task time.
• Increasing cluster size before diagnosing the bottleneck.
• Changing five settings at once.
• Ignoring the physical plan.
• Ignoring data distribution.
• Treating every slow stage as a compute-sizing problem.

23. Production performance checklist

• Identify the slow job.
• Identify the longest stage.
• Inspect task distribution.
• Compare max vs median.
• Check shuffle read/write.
• Check spill.
• Check scan time.
• Check GC.
• Inspect physical plan.
• Identify expensive operators.
• Verify effective Spark configuration.
• Change one variable.
• Re-run.
• Compare runtime and resource metrics.
• Record the reason for the improvement.

24. Architect-level mental model

BUSINESS WORKLOAD
      ↓
SPARK CODE / SQL
      ↓
PHYSICAL PLAN
      ↓
JOBS
      ↓
STAGES
      ↓
TASKS
      ↓
EXECUTOR RESOURCES
      ↓
RUNTIME METRICS
      ↓
ROOT CAUSE
      ↓
ARCHITECTURAL CHANGE

The important skill is not knowing where the buttons are. It is being able to say:

“The workload is slow because this operator creates this data movement, which produces this task distribution, which causes this resource pressure.”

That is the difference between using Spark UI and engineering Spark performance.

25. Interview questions

Foundation

1. What is Spark UI?
2. What is the difference between a job, stage, and task?
3. Which UI tab do you start with for a slow job?
4. What does the Jobs event timeline tell you?
5. What is shuffle read vs shuffle write?

Performance

6. How do you identify data skew?
7. Why is max task duration more useful than average in skew diagnosis?
8. What does shuffle spill mean?
9. How do you identify an I/O-bound stage?
10. How do you identify an executor-memory problem?
11. How do you identify excessive task fragmentation?
12. What does an Exchange operator indicate?

Architect

13. A pipeline doubled in runtime but processed the same row count. How do you investigate?
14. One task runs 50× longer than the others. What is your diagnostic sequence?
15. A team wants to double cluster size. What evidence would you request first?
16. How would you distinguish a data-model problem from a compute-sizing problem?
17. How do you prove that an optimization actually improved performance?

26. Cheat sheet

UI area	Primary question
Jobs	What ran and where is the time?
Event Timeline	What is the execution shape?
Stages	Which stage is expensive?
Tasks	Is work balanced?
SQL	Which operators caused the work?
Executors	Are resources healthy?
Storage	Is caching helping?
Environment	Are expected settings active?
Streaming	Is the streaming query keeping up?

Golden rule

Do not ask:
"Which Spark setting should I change?"

Ask:
"What evidence does the UI give me about the bottleneck?"

Next

Continue to 03.9 Spark Mental Model — the final Spark fundamentals topic that connects transformations, actions, DAGs, partitions, shuffle, stages, tasks, executors, and the UI into one execution model.