Skip to main content

03.1 Driver & Executors

Phase 3 — Apache Spark Fundamentals

The most important Spark runtime relationship is:

The driver coordinates; executors execute.

This distinction is essential for Databricks performance engineering. When a workload is slow or failing, you need to determine whether the bottleneck is the driver, executors, partitions, shuffle, storage, network, or execution plan.

Driver and executor architecture

1. The one-minute mental model​

Spark Application | Driver plan / schedule / coordinate | +----+----+ | | Executor Executor ... Executor | | Tasks Tasks | | Partitions Partitions | | Data Data

In current Databricks architecture, a multi-node compute resource has one driver and one or more worker nodes. Databricks runs one Spark executor per worker node, so worker and executor are commonly used interchangeably in Databricks architecture.
Databricks — Compute configuration

2. What is the driver?​

The driver is the coordinating process for a Spark application.

It is responsible for:

  • Maintaining the Spark application context.
  • Receiving notebook/job commands.
  • Building and coordinating execution.
  • Creating and scheduling jobs.
  • Tracking stages and tasks.
  • Communicating with executors.
  • Maintaining notebook/application state.
  • Returning results requested by the application.

Think of it as the orchestrator of one Spark application.

Notebook / Job | SparkSession | DRIVER | +-----+-----+ | | | Plan Schedule Monitor | Executors

The driver is not intended to process the entire dataset locally.

3. Driver versus distributed computation​

Consider a transformation such as:

df.filter("amount > 100").groupBy("customer_id").sum("amount")

The driver does not normally pull the entire DataFrame into memory and calculate the result itself.

Instead, Spark builds an execution plan and distributes the work across executors.

Driver | | execution plan v Executors | +--> partition 1 +--> partition 2 +--> partition 3 +--> ...

This is the foundation of distributed processing.

4. What makes the driver a bottleneck?​

A driver bottleneck occurs when too much work becomes driver-side work.

Common examples:

  • collect()
  • toPandas()
  • Large Python objects
  • Large result sets
  • Excessive notebook state
  • Very complex query plans
  • Too many tiny jobs
  • Excessive job submission
  • Driver-side loops
  • Heavy metadata processing

Example:

df.collect()

This asks Spark to return the complete result to the driver.

Safer for inspection:

df.limit(100).collect()

For production processing, keep the result distributed:

df.write.mode("overwrite").saveAsTable("catalog.schema.target")

Architect rule​

Do not add workers to solve a driver bottleneck.

Workers increase executor capacity. They do not automatically increase driver memory or driver CPU.

Databricks notes that a larger driver can be appropriate when substantial data is collected back to the notebook/driver.
Databricks — Compute configuration

5. Driver memory​

Driver memory can be consumed by:

  • Spark application state
  • Query planning
  • Notebook state
  • Metadata
  • Result objects
  • Python objects
  • Collected data
  • Driver-side libraries
SymptomInvestigate
Huge collect()Driver
Huge toPandas()Driver
Huge query planDriver
Many notebooks attachedDriver state
Driver CPU very highDriver workload
Executors mostly idlePossible driver bottleneck

A common mistake is increasing workers when the driver itself is under memory pressure.

6. Driver CPU​

The driver also performs CPU work:

  • Planning
  • Scheduling
  • Serialization
  • Notebook/application logic
  • Metadata processing
  • Python code executed outside Spark
  • Managing many jobs/tasks

A classic pattern is:

Driver CPU 90% Executor CPU 15% Workers 12

Adding workers here is usually the wrong first move.

Investigate:

  1. Driver-side Python.
  2. Large result collection.
  3. Complex execution plans.
  4. Excessive job submission.
  5. Metadata pressure.
  6. Many tiny tasks.

Databricks identifies long gaps in the Spark Jobs Timeline as potentially caused by an overloaded driver, complex plans, or non-Spark code that leaves workers idle.
Databricks — Gaps between Spark jobs

7. What is an executor?​

An executor is a Spark process responsible for executing tasks for the application.

Executors perform distributed work such as:

  • Reading data

  • Filtering

  • Projection

  • Joins

  • Aggregations

  • Sorts

  • Shuffle

  • Caching/persistence

  • Writing output

    Executor | +-- Task 1 -> Partition 1 +-- Task 2 -> Partition 2 +-- Task 3 -> Partition 3 +-- Task N -> Partition N

Executors are where most distributed data processing happens.

8. Worker node versus executor​

This is a common interview question.

Worker node​

The cloud compute machine/VM.

Executor​

The Spark process running on that worker.

Worker Node | +-- Spark Executor +-- CPU cores +-- Memory +-- Local disk +-- Spark execution

Databricks runs one executor per worker node in its standard multi-node architecture.
Databricks — Compute configuration

Therefore:

Worker = infrastructure resource. Executor = Spark process using that resource.

9. Executor cores and task concurrency​

Suppose each executor has eight usable execution cores.

Executor | +-- Core 1 -> Task +-- Core 2 -> Task +-- Core 3 -> Task +-- Core 4 -> Task +-- Core 5 -> Task +-- Core 6 -> Task +-- Core 7 -> Task +-- Core 8 -> Task

If you have four such executors:

4 executors x 8 cores approximately 32 concurrent task slots

This is a capacity model, not a guarantee of useful 32-way parallelism.

Useful parallelism also depends on:

  • Number of partitions
  • Stage dependencies
  • Task duration
  • CPU
  • I/O
  • Shuffle
  • Skew
  • Scheduling overhead

10. Why more cores do not always mean faster​

Imagine:

Executors: 8 Cores: 8 each Partitions: 2

You cannot efficiently use all available task slots because the stage contains only two partitions.

Or:

Partitions: 800 One partition: extremely large 799 partitions: normal

Near the end of the stage, most executors can become idle while one task processes the oversized partition.

Therefore:

Compute capacity is useful only when the workload has enough balanced parallel work.

11. Executor memory​

Executor memory supports distributed task execution.

Major consumers include:

  • Aggregations
  • Joins
  • Sorts
  • Shuffle
  • Cached/persisted data
  • Serialization
  • JVM objects
  • Python processing
  • Execution state

Think:

Executor | +-- CPU +-- Execution memory +-- Storage/cache memory +-- Process overhead +-- Local disk

Do not assume all configured memory is available for arbitrary application objects.

12. Driver memory versus executor memory​

Memorize this table:

Workload patternLikely pressure
Huge collect()Driver
Huge toPandas()Driver
Complex planDriver
Notebook/application stateDriver
Large aggregationExecutor
Large joinExecutor
ShuffleExecutor + network + local disk
Large cacheExecutor
Skewed partitionOne executor/task
Many parallel tasksExecutors

The first diagnostic question should be:

Which process is under pressure?

13. Executor local disk and spill​

Executor local disk is important for intermediate processing.

When memory is insufficient for an operation, Spark may spill intermediate data to disk.

Large sort / aggregation | v Memory fills | v Spill | v Local disk I/O | v Slower stage

High spill is not automatically a failure, but excessive spill is often a performance signal.

Databricks' current Spark UI guidance recommends checking skew and spill when diagnosing long stages.
Databricks — Spark UI guide

14. Memory pressure is not always an OOM​

A workload can become unhealthy before it crashes:

More data | v Memory pressure | v GC increases | v Spill increases | v Disk I/O increases | v Stage gets slower | v Possible executor failure

Do not wait for OOM before investigating memory pressure.

15. Executor OOM​

A common failure pattern is:

Task | v Memory pressure | v Executor lost | v Task retry | +---- success -> continue | +---- repeated failure -> stage/job failure

Databricks documents that when executors run out of memory, Spark removes them and retries their tasks; repeated failures can eventually fail the stage or job.
Databricks — Spark memory issues

Common causes include:

  • Too few shuffle partitions
  • Large broadcast
  • UDF memory usage
  • Window without suitable partitioning
  • Data skew
  • Large aggregation
  • Oversized partitions

16. Do not immediately increase executor memory​

Suppose an executor fails.

First investigate:

  1. Skew
  2. Partition size
  3. Broadcast joins
  4. Shuffle partitions
  5. UDFs
  6. Window functions
  7. Aggregation state
  8. Unnecessary cache
  9. Memory per core

Only then decide whether compute sizing is actually the correct fix.

Databricks recommends looking at memory-per-core when validating memory pressure rather than considering only total node memory.
Databricks — Spark memory issues

17. Driver OOM​

Common causes:

collect()​

result = df.collect()

toPandas()​

pdf = df.toPandas()

Large Python objects​

records = [row.asDict() for row in df.collect()]

The third example is particularly dangerous because it first collects data and then creates another large Python representation.

Better approach:

summary = ( df .groupBy("country") .count() )

Then collect only the small summary if necessary.

18. Why toPandas deserves special attention​

The data flow is:

S3 / Delta | v Executors | v Small result | v Driver | v Pandas DataFrame

The pandas DataFrame lives on the driver-side Python process.

Therefore:

Use pandas for genuinely small results, not as a way to bypass distributed Spark processing.

19. Driver-side Python can leave workers idle​

Consider:

for customer in customers: run_some_python_logic(customer)

If this is ordinary Python rather than Spark execution, workers may sit idle.

Databricks currently identifies non-Spark code as a cause of gaps in the Spark Jobs Timeline and notes that workers can remain idle while compute continues to incur cost.
Databricks — Gaps between Spark jobs

Architectural preference:

Python loop | v Ask: can Spark express this? | v Spark DataFrame / SQL | v Distributed execution

20. Driver ↔ executor communication​

The driver schedules work and receives execution status.

DRIVER | | task scheduling / status | +--+-----------+-----------+ | | | v v v Executor 1 Executor 2 Executor N | | | +--------------+-----------+ | task results status/metrics

Communication overhead matters when:

  • There are huge numbers of tiny tasks.
  • Tasks are extremely short.
  • Too many jobs are submitted.
  • The driver is overloaded.
  • The application creates excessive scheduling overhead.

This is one reason small files and tiny partitions can hurt performance even when the cluster has plenty of CPU.

21. Executor lifecycle​

A simplified lifecycle:

Compute starts | Worker provisioned | Executor starts | Executor registers | Tasks execute | Tasks complete | Executor remains available | Compute terminates

With autoscaling, the number of workers/executors can change while a workload is running.

Databricks supports minimum and maximum worker settings for autoscaling.
Databricks — Compute configuration

22. Executor loss​

Executors can disappear for different reasons:

  • Memory failure
  • Cloud instance interruption
  • Autoscaling
  • Network problems
  • Runtime failure
  • Infrastructure problems

Do not classify every executor loss as OOM.

Correct sequence:

Executor lost | Check Compute Event Log | +----+----------------+ | | v v Infrastructure event? No clear event | | v v Investigate cloud Spark UI / logs | v Memory / task / data distribution | v Root cause

Databricks recommends checking the compute Event Log first for executor failures, including autoscaling or spot/preemptible instance events.
Databricks — Failing jobs or executors removed

23. One slow executor: investigate skew​

Suppose task durations look like:

Executor 1 42 sec Executor 2 44 sec Executor 3 41 sec Executor 4 47 sec Executor 5 43 sec Executor 6 8 min 31 sec

Do not immediately replace the machine.

Possible explanations:

  • One oversized partition
  • Data skew
  • Spill
  • GC
  • Slow I/O
  • Network issue
  • Input imbalance

This is a classic straggler pattern.

24. Real scenario: driver bottleneck​

Observed:

Driver CPU 90% Executor CPU 15% Shuffle Low Workers 12

Diagnosis:

The distributed compute is underutilized.

Investigate:

  • Driver-side Python
  • collect()
  • toPandas()
  • Complex query plan
  • Metadata
  • Excessive job submission
  • Tiny jobs
  • Notebook/application logic

Wrong fix:

Add 20 workers.

Better fix:

Remove or reduce driver-side work.

25. Real scenario: executor bottleneck​

Observed:

Driver CPU 25% Executor CPU 90% Shuffle High Spill High Failed tasks Repeated

Investigate:

  • Join strategy
  • Aggregation
  • Partition count
  • Data skew
  • Memory per core
  • Broadcast size
  • Input/output volume

Possible fixes depend on root cause:

  • Repartition
  • Fix skew
  • Change join strategy
  • Adjust adaptive behavior
  • Avoid oversized broadcasts
  • Increase memory per core
  • Rewrite transformations

26. Spark UI: where to look​

Databricks currently recommends:

  1. Start with the Jobs Timeline.
  2. Identify the longest stage.
  3. Look for skew or spill.
  4. Determine whether the stage is I/O bound.
  5. Investigate other causes of long runtime.

Useful signals include:

  • Executor count
  • Active tasks
  • Failed tasks
  • Task duration
  • Shuffle read
  • Shuffle write
  • Spill
  • GC time
  • Input/output
  • Executor duration

Databricks — Diagnose cost and performance issues using Spark UI

Apache Spark's Web UI also exposes executor resource usage, task counts, shuffle totals, storage memory and GC-related signals.
Apache Spark — Web UI

27. Driver logs versus executor logs​

Driver logs​

Useful for:

  • Application exceptions
  • Planning/scheduling issues
  • Driver-side failures
  • Notebook/application output
  • Streaming startup problems
  • Driver hangs

Executor logs​

Useful for:

  • Task-specific failures
  • Executor exceptions
  • UDF failures
  • Worker-side processing problems
  • Executor-specific errors

Databricks documents driver and executor logs as separate troubleshooting sources. Log access depends on compute access mode and permissions.
Databricks — Debugging with Spark UI

Also:

Never print credentials, tokens, or secrets into driver logs.

Databricks warns that sensitive values placed in Spark configuration properties or environment variables can be exposed through driver logs.
Databricks — Secret configuration guidance

28. Thread dumps​

A thread dump is a snapshot of thread states.

It can help investigate:

  • Hanging driver
  • Hanging executor
  • Long-running task
  • Blocking operations
  • Thread contention

The Spark UI exposes thread-dump information from the Executors view.
Databricks — Debugging with Spark UI

Think of it as:

“What are the JVM threads doing right now?”

29. Hands-on lab — inspect the driver​

Run:

print("Spark version:", spark.version) print("Application ID:", spark.sparkContext.applicationId) print("Master:", spark.sparkContext.master) print("Application name:", spark.sparkContext.appName)

Then open the Spark UI.

Record:

MetricYour observation
Application ID
Driver
Workers
Executors
Jobs
Stages

The goal is to connect:

Notebook → SparkSession → Driver → Executors → Tasks

30. Hands-on lab — distributed execution​

Create a dataset:

df = spark.range(0, 10_000_000)

print("Partitions:", df.rdd.getNumPartitions())

Run:

result = ( df .filter("id % 2 = 0") .groupBy() .count() )

result.show()

Open the Spark UI and find:

  • Job
  • Stage
  • Tasks
  • Executors
  • Duration
  • Input/output

Do not optimize it yet.

First learn to observe execution.

31. Hands-on lab — driver collection​

Compare:

small = df.limit(100).collect() print(len(small))

with the conceptual danger of:

Do NOT run blindly on a large dataset.

df.collect()

Question:

Where does the returned result go?

Answer:

The driver/application process.

32. Hands-on lab — classify the bottleneck​

For a workload you know is slow, capture:

MetricObservation
Driver CPU
Driver memory
Executor CPU
Executor memory
Executor count
Failed tasks
Shuffle read
Shuffle write
Spill
Longest stage
Longest task

Then classify:

Driver Executor Shuffle Skew I/O Scheduling Compute sizing

This is the beginning of professional Spark performance engineering.

33. Architect decision tree​

Slow workload | Is driver busy? /
YES NO | | Driver Executors | | plan / CPU / memory collect / shuffle / spill Python | | +----+-----+ | Root cause | Targeted change

The key principle:

Diagnose first. Resize second.

34. Common anti-patterns​

Anti-pattern 1 — collect everything​

df.collect()

Risk: driver memory pressure.

Anti-pattern 2 — convert a huge DataFrame to pandas​

df.toPandas()

Risk: driver memory pressure and loss of distributed processing.

Anti-pattern 3 — add workers without checking the bottleneck​

Risk: higher cost with little or no performance improvement.

Anti-pattern 4 — increase executor memory without investigating skew​

Risk: paying for larger machines while the real issue is one oversized partition.

Anti-pattern 5 — use Python loops for distributed transformations​

Risk: workers remain underutilized while the driver performs local work.

35. Architect sizing model​

Do not think:

Bigger cluster = faster

Think:

Performance

parallelism × efficient execution × compute capacity × good data layout

Subject to:

  • Driver capacity
  • Executor memory
  • CPU
  • Network
  • Storage I/O
  • Shuffle
  • Partitioning
  • Skew
  • Query plan

This model will be reused throughout the Performance Engineering phase.

36. Production checklist​

Driver​

  • Avoid unbounded collect()
  • Avoid unbounded toPandas()
  • Monitor CPU and memory
  • Control notebook/application state
  • Avoid excessive job submission
  • Investigate complex plans

Executors​

  • Appropriate memory per core
  • Appropriate worker type
  • Enough partitions
  • No severe skew
  • No excessive spill
  • Reasonable shuffle
  • Appropriate cache usage

Compute​

  • Correct compute type
  • Sensible autoscaling
  • Appropriate min/max workers
  • Appropriate driver size
  • Cost guardrails

37. Architect-level interview questions​

Q1. What is the driver?​

The process that coordinates a Spark application, builds/coordinates execution and schedules work.

Q2. What is an executor?​

A Spark process that executes tasks against data partitions.

Q3. What is the difference between worker and executor?​

A worker is the underlying compute node; an executor is the Spark process running on that node.

Q4. Why can collect() cause driver OOM?​

Because the result is returned to the driver.

Q5. Why does adding workers not fix driver OOM?​

Workers provide executor capacity. Driver memory is a separate resource.

Q6. Why can one executor be much slower than the others?​

Skew, an oversized partition, spill, GC, I/O or infrastructure issues can create a straggler.

Q7. How do you investigate executor failure?​

Check the compute Event Log, Spark UI, failed tasks and executor logs, then classify the cause as infrastructure, memory, data distribution or execution-related.

Q8. What is the first place to look for a slow Spark job?​

The Jobs Timeline and longest stage, followed by skew, spill and I/O characteristics.

Q9. When should you increase driver size?​

After proving the driver is the bottleneck and the workload genuinely needs more driver CPU or memory.

Q10. When should you increase executor memory?​

After proving distributed execution is memory-bound and the issue is not better solved by fixing skew, partitioning, joins, aggregations or another design change.

38. Final mental model​

USER / JOB | SPARKSESSION | DRIVER plan / schedule / monitor | distributed tasks | +----+----+----+ | | | | v v v v E1 E2 E3 EN | | | | tasks tasks tasks tasks | | | | partitions / data | STORAGE S3 / Delta

Four rules to remember​

Driver problem → investigate driver.

Executor problem → investigate distributed execution.

One slow task → investigate partitioning and skew.

Repeated executor loss → investigate Event Log, memory and infrastructure.

If these four rules become instinctive, the next Spark topics become much easier.

Official references​

Next​

03.2 Jobs, Stages & Tasks — how Spark turns an action into a job, divides it into stages, creates tasks from partitions, and uses dependency/shuffle boundaries to build the execution graph.