Skip to main content

03. Spark Architecture

Phase 3 — Apache Spark Fundamentals

Apache Spark is the distributed execution engine behind a large part of Databricks data engineering. To become effective with Databricks, you need to understand not only how to write PySpark or SQL, but where the code runs, how Spark builds an execution plan, how work is divided, and how that work reaches the machines that process the data.

Spark architecture

1. The Spark mental model​

Start with one sentence:

The driver plans and coordinates; executors perform distributed work against partitions of data.

A simplified execution path is:

User / Notebook / Job
        |
        v
   SparkSession
        |
        v
      DRIVER
  plan + coordinate
        |
        v
  Cluster Resources
        |
   +----+----+
   |         |
   v         v
Worker 1   Worker 2 ... Worker N
   |         |
Executor   Executor
   |         |
Tasks      Tasks
   |         |
Partitions of data

This mental model explains a huge percentage of Spark behavior.

2. What is a Spark application?​

A Spark application is the execution context for a workload.

Conceptually:

One Spark Application
│
├── Driver process
│
├── SparkContext / SparkSession
│
├── Executors
│   ├── Executor 1
│   ├── Executor 2
│   └── Executor N
│
└── One or more Jobs

A single application can execute multiple actions and therefore multiple jobs.

Do not confuse:

  • Application — lifetime of the Spark execution context.
  • Job — work triggered by an action.
  • Stage — a phase of a job separated by shuffle boundaries.
  • Task — work for one partition within a stage.
  • Partition — a unit of distributed data processing.

The hierarchy is:

Application → Job → Stage → Task → Partition

This hierarchy becomes extremely important later when diagnosing Spark UI performance.

3. Where Spark fits inside Databricks​

Databricks is a platform; Spark is an execution engine.

A useful enterprise mental model is:

Databricks Platform
│
├── Workspace
│
├── Unity Catalog
│
├── Jobs / Lakeflow
│
├── Compute
│     |
│     └── Databricks Runtime
│             |
│             └── Apache Spark
│
└── Cloud Storage
      └── S3 / Delta

Databricks manages the Spark environment for you. You normally do not manually create a SparkContext or initialize a cluster manager in a Databricks notebook. citeturn828704search9

For current Databricks compute, distributed processing runs on worker nodes and Databricks runs one Spark executor per worker node; therefore Databricks documentation often uses worker and executor almost interchangeably. citeturn828704search3

4. Driver​

The driver is the control process for a Spark application.

Its major responsibilities include:

  • Running your application logic.
  • Maintaining the Spark execution context.
  • Building logical and physical execution plans.
  • Coordinating executors.
  • Scheduling tasks.
  • Tracking application state.
  • Returning results to the application.

Conceptually:

                 DRIVER
                    |
       +------------+------------+
       |            |            |
   Plan work    Schedule      Track
       |         tasks        state
       |            |            |
       +------------+------------+
                    |
             Executor processes

Important​

The driver does not normally perform all the distributed data processing itself.

If your code causes a huge amount of data to be collected back to the driver, you can create a driver bottleneck.

For example:

df.collect()

may move a large result to the driver.

That is very different from:

df.write.format("delta").saveAsTable("catalog.schema.table")

where the distributed work can remain distributed.

5. Executors​

Executors are processes responsible for executing tasks sent by the driver.

An executor can:

  • Run tasks.
  • Process partitions.
  • Keep intermediate data.
  • Cache data when requested.
  • Perform shuffle operations.
  • Report execution status back to the driver.

Conceptually:

Driver
  |
  +---- Task 0 ----> Executor 1
  |
  +---- Task 1 ----> Executor 2
  |
  +---- Task 2 ----> Executor 3
  |
  +---- Task 3 ----> Executor 1

Multiple tasks can run concurrently depending on available executor cores and workload characteristics.

6. Worker vs executor​

This distinction matters on interviews.

A worker node is a compute machine.

An executor is a Spark process running on that worker.

In the standard Databricks architecture, Databricks runs one executor per worker node. citeturn828704search3

So:

Worker Node
└── Spark Executor
    ├── Task
    ├── Task
    └── Task

Therefore, when you see:

"Executor 5 is slow"

you should think about the worker hosting that executor and then inspect its tasks, memory, CPU, I/O and shuffle behavior.

7. SparkSession and SparkContext​

In modern Spark applications, SparkSession is the main entry point for working with Spark.

Typical Databricks Python code:

spark

The spark variable represents the SparkSession that Databricks provides to the notebook.

You normally do not need to create it manually.

The conceptual relationship is:

SparkSession
      |
      v
SparkContext
      |
      v
Spark execution infrastructure

SparkSession provides APIs for:

  • DataFrames
  • SQL
  • Catalog operations
  • Configuration
  • Access to the underlying Spark context

8. Cluster manager — where does it fit?​

In generic Apache Spark architecture, a cluster manager allocates resources for Spark applications.

The simplified open-source model is:

Spark Application
       |
     Driver
       |
Cluster Manager
       |
+------+------+
|             |
Worker       Worker
|             |
Executor     Executor

Depending on deployment mode, Spark can integrate with different cluster managers.

In Databricks, however, the platform abstracts much of this infrastructure management from the user. You normally select and configure Databricks compute rather than manually managing the Spark cluster manager.

This is an important distinction:

Learn the Spark architecture even when Databricks hides the infrastructure implementation.

You need the architecture to understand performance, failures and resource behavior.

9. From code to execution​

Suppose you write:

result = (
    orders
    .filter("status = 'COMPLETED'")
    .groupBy("customer_id")
    .sum("amount")
)

result.write.mode("overwrite").saveAsTable("gold.customer_sales")

What happens?

Step 1 — Build the plan​

The DataFrame operations describe what you want Spark to do.

Step 2 — Lazy evaluation​

The transformations do not immediately execute the complete computation.

Step 3 — Action​

The write triggers execution.

Step 4 — Optimize​

Spark/Databricks constructs and optimizes the execution plan.

Step 5 — Schedule​

The driver breaks execution into stages and tasks.

Step 6 — Distribute​

Tasks are sent to executors.

Step 7 — Process partitions​

Each task processes its assigned partition.

Step 8 — Write result​

The distributed output is written to the target.

DataFrame transformations
          |
          v
     Logical plan
          |
          v
   Optimized / physical plan
          |
          v
     Jobs + Stages
          |
          v
        Tasks
          |
          v
     Executors
          |
          v
      Partitions
          |
          v
       Storage

Spark uses lazy evaluation: transformations build processing logic and an action triggers execution. citeturn828704search9

10. Why partitions are fundamental​

Spark does not normally send "the whole table" to one executor.

It processes partitions.

Imagine a 1 TB dataset:

1 TB Dataset
│
├── Partition 0
├── Partition 1
├── Partition 2
├── ...
└── Partition N

Tasks operate on these partitions.

For a stage with 2,000 partitions, Spark can conceptually create approximately 2,000 tasks for that stage.

The actual concurrency depends on available compute resources.

This leads to one of the most important Spark relationships:

Parallelism is constrained by the number and size of partitions and the available execution resources.

We will study partitioning in depth later.

11. Jobs​

A job is triggered by an action.

Common examples include:

df.count()
df.collect()
df.show()
df.write.format("delta").save(...)

The important point is:

A transformation describes work; an action asks Spark to execute the required work.

A single Spark application can have multiple jobs.

Application
│
├── Job 0
├── Job 1
├── Job 2
└── Job 3

Spark's scheduler can manage multiple jobs within an application, depending on how they are submitted and the configured scheduling mode. citeturn828704search2

12. Stages​

A job is divided into stages.

A simplified example:

Job
│
├── Stage 0
│     └── map/filter work
│
├── Stage 1
│     └── shuffle + aggregation
│
└── Stage 2
      └── final write

A major concept is the shuffle boundary.

When data must move between executors to satisfy an operation such as a large aggregation or join, Spark may create a shuffle.

That movement can introduce:

  • Network I/O
  • Disk I/O
  • Serialization
  • Additional scheduling overhead

This is why shuffle becomes one of the most important performance topics in Spark.

13. Tasks​

A task is the unit of work Spark sends to an executor.

A simplified stage:

Stage 1
│
├── Task 0 → Partition 0
├── Task 1 → Partition 1
├── Task 2 → Partition 2
├── Task 3 → Partition 3
└── ...

If a stage has 500 partitions, it can have approximately 500 tasks.

Tasks are where your transformations ultimately execute against specific partitions.

14. The complete hierarchy​

Memorize this:

Spark Application
       |
       +-- Job
       |    |
       |    +-- Stage
       |         |
       |         +-- Task
       |              |
       |              +-- Partition
       |
       +-- Job
            |
            +-- Stage
                 |
                 +-- Task

Interview answer​

If asked:

"Explain Spark execution hierarchy."

Answer:

"A Spark application contains one or more jobs. Jobs are triggered by actions and are divided into stages based on execution dependencies and shuffle boundaries. Each stage contains tasks, and each task processes a partition of data on an executor."

That is the foundation for the rest of this course.

15. Local vs distributed execution​

Not every Spark workload requires multiple workers.

A single-node environment can run Spark in local mode.

For distributed processing:

Driver
  |
  +--- Worker 1 / Executor
  +--- Worker 2 / Executor
  +--- Worker 3 / Executor

For small workloads, single-node compute can be appropriate. For large distributed workloads, multiple worker nodes provide parallel execution. citeturn828704search1

This is why compute selection matters even before tuning Spark code.

16. What happens when an executor fails?​

Spark is designed for distributed execution and task retry.

Conceptually:

Task 27
   |
Executor 2 fails
   |
   v
Task retry
   |
Executor 4
   |
   v
Task succeeds

A failure does not necessarily mean the entire application immediately fails.

Spark can retry failed tasks, although repeated executor failures, memory pressure, corrupted data, unavailable resources or other conditions can ultimately fail a stage/job.

This becomes important when reading the Spark UI and diagnosing executor loss. Databricks specifically recommends using the Spark UI to inspect jobs, stages, task behavior, shuffle, spill and executor issues. citeturn828704search0

17. Driver failure vs executor failure​

These have different blast radii.

Executor failure​

Potential impact:

  • Tasks on that executor fail.
  • Spark may retry them elsewhere.
  • The stage can potentially recover.

Driver failure​

Potential impact:

  • The application loses its coordinator.
  • The Spark application may fail.
  • Notebook/job execution context can be lost.

Therefore:

Driver stability is especially important for production workloads.

Common driver-pressure patterns include:

  • Huge collect()
  • Huge toPandas()
  • Excessive metadata operations
  • Very large query plans
  • Too much application-side processing

18. Driver memory vs executor memory​

This distinction is fundamental.

Driver memory​

Used for things such as:

  • Application state
  • Query planning
  • Scheduling
  • Notebook/application logic
  • Results returned to the application

Executor memory​

Used for:

  • Task execution
  • Shuffle
  • Cached data
  • Intermediate data
  • Execution structures

A common mistake is to increase executor memory when the actual problem is driver memory.

Always identify which process is under pressure.

19. Databricks Runtime​

Databricks Runtime provides the environment in which Spark workloads run.

It includes Spark plus Databricks optimizations and additional components.

For example:

Databricks Runtime
│
├── Apache Spark
├── Databricks optimizations
├── Runtime libraries
└── Cloud/platform integrations

Databricks also provides Photon for supported workloads. Photon is an important performance topic, but do not confuse Photon with Spark itself.

We will cover Photon deeply in the Performance Engineering phase.

20. Spark vs Photon​

A useful mental model:

Databricks workload
       |
       +------------------+
       |                  |
   Spark execution    Photon acceleration
       |                  |
       +---------+--------+
                 |
             Compute

Spark remains the programming and execution abstraction you need to understand.

Photon is a Databricks-native vectorized execution engine that accelerates supported workloads.

Therefore:

Learn Spark first. Then learn how Databricks accelerates Spark/SQL workloads.

21. Spark architecture and storage​

Spark is compute.

S3, ADLS, GCS and other systems are storage.

A typical AWS flow:

                  Databricks
                     |
                  Compute
                     |
             +-------+-------+
             |               |
          Driver          Executors
                             |
                             |
                           S3
                             |
                       Delta / Parquet

This separation is fundamental to the lakehouse architecture.

Compute can be created and terminated independently from durable cloud storage.

That is one of the reasons lakehouse platforms can scale compute independently of storage.

22. Spark architecture and Unity Catalog​

Unity Catalog governs data assets; Spark executes the processing.

Conceptually:

User / Job
    |
    v
Spark workload
    |
    v
Compute
    |
    +------> Unity Catalog authorization
    |
    +------> Cloud storage

Unity Catalog does not replace Spark.

Spark does not replace Unity Catalog.

They solve different architectural problems:

LayerPrimary responsibility
SparkDistributed computation
Databricks ComputeCompute infrastructure
Unity CatalogGovernance and access control
S3Durable object storage
Delta LakeTransactional data layer

23. How to read Spark architecture as an architect​

When someone says:

"The pipeline is slow."

Do not immediately change the cluster size.

Ask:

  1. Is the driver healthy?
  2. How many jobs are being triggered?
  3. Which stage is slow?
  4. Is there a shuffle?
  5. How many partitions exist?
  6. Are tasks balanced?
  7. Is there data skew?
  8. Is there spill?
  9. Is the workload I/O-bound or CPU-bound?
  10. Are executors failing?
  11. Is the data layout efficient?
  12. Is the compute type appropriate?

The Spark UI exposes jobs, stages, tasks, executor behavior and shuffle metrics specifically to help answer these questions. citeturn828704search0turn828704search4

24. Hands-on: inspect your Spark environment​

Run this in a Databricks notebook:

print("Spark version:", spark.version)

print("Application ID:")
print(spark.sparkContext.applicationId)

print("Master:")
print(spark.sparkContext.master)

print("App name:")
print(spark.sparkContext.appName)

Then inspect:

spark.conf.get("spark.sql.shuffle.partitions")

You are learning to connect:

Notebook → SparkSession → SparkContext → Application → Compute

Do not focus on memorizing configuration values yet.

Focus on understanding the architecture.

25. Hands-on: create a small execution​

df = spark.range(0, 1_000_000)

result = (
    df
    .filter("id % 2 = 0")
    .groupBy()
    .count()
)

result.show()

Ask yourself:

  • What is the application?
  • Which operation is the action?
  • When does execution start?
  • Where are tasks executed?
  • What partitions are processed?
  • What would appear in the Spark UI?

The answer to these questions matters more than the code itself.

26. Spark UI: your diagnostic microscope​

The Spark UI gives you the execution evidence.

Important areas include:

Jobs​

Shows jobs and their execution timeline.

Stages​

Shows stage-level duration, tasks, input/output and shuffle.

Executors​

Shows executor resource and task information.

SQL​

Shows SQL/DataFrame execution details.

The Spark Web UI exposes job DAGs, stage metrics, task progress, input/output and shuffle metrics. citeturn828704search4

Architect workflow​

Slow pipeline
     |
     v
Jobs
     |
     v
Longest Stage
     |
     +---- Skew?
     +---- Shuffle?
     +---- Spill?
     +---- I/O?
     +---- Executor failure?
     |
     v
Root cause
     |
     v
Targeted optimization

Databricks recommends starting with the Jobs Timeline and then examining the longest stage, skew/spill and I/O characteristics. citeturn828704search0

27. Common beginner misconceptions​

Misconception 1​

"Spark is the cluster."

No.

Spark is the distributed processing engine. Databricks compute provides the infrastructure and runtime environment.

Misconception 2​

"Driver processes all the data."

No.

The driver coordinates distributed execution. Executors process distributed partitions.

Misconception 3​

"One table equals one task."

No.

Tasks operate on partitions.

Misconception 4​

"One job equals one stage."

No.

A job can contain multiple stages.

Misconception 5​

"More workers always make Spark faster."

No.

The bottleneck may be shuffle, skew, I/O, driver pressure, small files, poor partitioning or another issue.

Misconception 6​

"Databricks hides Spark, so Spark architecture is unnecessary."

The opposite.

Databricks abstracts infrastructure management, but Spark concepts are essential for debugging and performance engineering.

28. Architect reference architecture​

For an enterprise AWS lakehouse:

                    USERS / JOBS / BI
                           |
                           v
                    DATABRICKS
                           |
                    +------+------+
                    |             |
                 Compute      Unity Catalog
                    |             |
              +-----+-----+       |
              |           |       |
           Driver      Workers    |
                        |          |
                    Executors      |
                        |          |
                        +-----+----+
                              |
                              v
                         Amazon S3
                              |
                       Delta / Parquet

Add:

  • SSO / identity
  • Service principals
  • Compute policies
  • Network controls
  • Audit logging
  • CI/CD
  • Lakeflow Jobs

and this becomes an enterprise execution platform rather than simply a Spark cluster.

29. Architect-level interview questions​

Q1. What is the difference between a driver and an executor?​

The driver coordinates the Spark application and schedules work. Executors run tasks and process partitions.

Q2. What is the relationship between worker and executor in Databricks?​

Databricks commonly runs one Spark executor per worker node, so the terms are often used interchangeably in Databricks architecture documentation. citeturn828704search3

Q3. What triggers a Spark job?​

An action triggers execution.

Q4. What determines task count?​

A task generally corresponds to processing one partition for a stage.

Q5. Why does Spark create multiple stages?​

Stages separate portions of execution around dependency/shuffle boundaries.

Q6. Why can a driver run out of memory?​

Large results collected to the driver, excessive application state, large plans or application-side processing can create driver pressure.

Q7. How do you troubleshoot a slow Spark job?​

Start with Jobs → identify the longest stage → inspect tasks, shuffle, skew, spill, I/O and executor behavior → identify root cause → apply targeted optimization.

Q8. Why is Spark knowledge essential for Databricks?​

Because Databricks provides a managed platform around distributed processing; Spark remains fundamental to understanding execution, resource usage and performance.

30. Final mental model​

Memorize this architecture:

                  APPLICATION
                      |
                  SPARKSESSION
                      |
                    DRIVER
               plan + coordinate
                      |
              +-------+-------+
              |               |
           WORKER           WORKER
              |               |
          EXECUTOR         EXECUTOR
              |               |
          TASKS             TASKS
              |               |
        PARTITIONS        PARTITIONS
              |               |
              +-------+-------+
                      |
                 STORAGE
              S3 / Delta

And memorize this execution hierarchy:

Application
    ↓
  Jobs
    ↓
 Stages
    ↓
 Tasks
    ↓
Partitions
    ↓
   Data

If you understand this hierarchy, you have the foundation for understanding Spark performance, shuffle, partitioning, joins, caching, skew, the Spark UI and Databricks optimization.

Official references​

Next​

03.1 Driver and Executors — deep dive into driver responsibilities, executor lifecycle, cores, memory, task concurrency, executor loss, driver bottlenecks and Databricks troubleshooting.