Skip to main content

03.4 Actions & Lazy Evaluation

Phase 3 — Apache Spark Fundamentals

The most important Spark execution rule is simple:

Transformations describe what should happen. Actions tell Spark to actually do it.

This distinction is the foundation for understanding why a Spark application can build a large chain of transformations almost instantly, but suddenly spend minutes or hours when count(), collect(), write, or another action is executed.

Spark actions and lazy evaluation architecture

1. Execution mental model​

Consider:

orders = spark.table("sales.orders")

result = (
    orders
    .filter("status = 'COMPLETE'")
    .select("customer_id", "amount")
    .groupBy("customer_id")
    .sum("amount")
)

At this point, Spark has not necessarily processed all rows. It has built a computation plan.

When you execute:

result.count()

Spark must execute the required lineage.

Driver program
     |
     | transformations
     v
Logical plan / lineage
     |
     | action
     v
Query optimization
     |
     v
Physical execution plan
     |
     v
Job
     |
     +---- Stage 0 ----> Tasks
     |
     +---- Stage 1 ----> Tasks
     |
     v
Result / write side effect

This is why source-code execution time and Spark workload execution time are not the same thing.

2. What is an action?​

An action is an operation that requests a result or causes a side effect.

ActionTypical purposeMain consideration
count()Count rowsUpstream computation may be required
collect()Bring all rows to driverDriver OOM risk
show()Display rowsStill triggers execution
take(n)Return limited rowsDriver-side result
first()Return one rowExecutes a query
head()Return first rowsDriver-side result
toPandas()Convert to pandasDriver memory pressure
writePersist dataExecutes distributed work
foreachApply side effectsExecutes distributed work

RDD actions include reduce, count, collect, take and save operations.

Apache Spark documents that an action triggers computation of the preceding transformation chain. Apache Spark — RDD Programming Guide

3. Lazy evaluation​

Lazy evaluation means Spark does not immediately execute every transformation.

df1 = spark.read.parquet("/data/orders")
df2 = df1.filter("amount > 100")
df3 = df2.select("order_id", "customer_id", "amount")

These statements primarily construct a plan.

Now:

df3.count()

is an execution trigger.

df1 = read
df2 = filter
df3 = project

No complete scan yet
        |
        v
     count()
        |
        v
NOW EXECUTE

Lazy evaluation gives Spark the opportunity to optimize the complete computation rather than blindly executing every intermediate step.

4. Why laziness matters​

Suppose you write:

df = (
    spark.read.table("orders")
    .filter("country = 'IN'")
    .filter("status = 'COMPLETE'")
    .select("customer_id", "amount")
)

Spark can reason about the complete query.

Read
  |
Filter country
  |
Filter status
  |
Project columns
  |
Action

The optimizer can often combine and push operations closer to the data source.

The architectural principle is:

Do not think of each DataFrame statement as an independent execution. Think of the chain as a plan that Spark will optimize and execute when required.

5. Action → Job​

A useful mental model is:

One execution-triggering action
          |
          v
        Job
          |
          v
   One or more stages
          |
          v
       Tasks

For example:

df.groupBy("customer_id").count().show()

The show() action causes Spark to execute the required upstream plan.

A job may contain multiple stages because shuffle boundaries can divide execution into separate stages.

This connects directly to previous topics:

  • Transformation builds the plan.
  • Action triggers execution.
  • Job represents execution of that request.
  • Stage represents a portion of execution separated by dependencies such as shuffle.
  • Task executes work for a partition.

6. Not every action has the same execution behavior​

Consider:

df.count()
df.show()
df.write.mode("overwrite").saveAsTable("gold.orders")

These are separate execution requests.

If the DataFrame is not persisted and Spark cannot reuse an equivalent subplan, Spark may need to recompute upstream work for each request.

Source
  |
Filter
  |
Join
  |
Aggregation
  |
  +--> count()
  |
  +--> show()
  |
  +--> write()

Without reuse, the expensive lineage can be executed repeatedly.

This is one of the most important performance implications of lazy evaluation.

7. Repeated actions and recomputation​

Example:

result = (
    spark.read.table("large_orders")
    .filter("status = 'COMPLETE'")
    .groupBy("customer_id")
    .sum("amount")
)

result.count()
result.show(20)
result.write.mode("overwrite").saveAsTable("gold.customer_sales")

If result is not persisted and Spark cannot reuse an equivalent subplan, each action can cause upstream work to be performed again.

For an expensive pipeline:

10 TB read
   |
large filter
   |
shuffle
   |
aggregation
   |
   +---- count()
   |
   +---- show()
   |
   +---- write()

That can become an architectural problem.

The solution is not “always cache”.

Instead ask:

  1. Is the result reused?
  2. Is recomputation expensive?
  3. Can the cached result fit available resources?
  4. Is the reuse window long enough?
  5. Would writing an intermediate Delta table be more appropriate?

8. Cache and persist​

Spark provides persistence mechanisms so computed partitions can be reused.

result = expensive_pipeline

result.cache()

result.count()       # materializes cache
result.show(20)      # can reuse cached data
result.write.saveAsTable("gold.customer_sales")

Spark documentation explains that persisted data can be reused by later actions instead of recomputing the full lineage. Apache Spark — RDD Programming Guide

For DataFrames, cache() uses Spark SQL's caching mechanism.

You can also use:

result.persist()

and later:

result.unpersist()

Important​

Calling cache() does not mean the entire dataset is instantly stored.

The cache is populated as partitions are computed.

Therefore the first action still performs the expensive computation.

9. Cache materialization​

This is a common interview question.

What happens when you call:

df.cache()

Nothing expensive necessarily happens immediately.

Then:

df.count()

Spark executes the plan and materializes cached partitions as they are computed.

Then:

df.show(20)

can reuse the cached representation.

cache()
  |
  | mark for persistence
  v
count()
  |
  | compute partitions
  v
CACHE
  |
  +---- show()
  +---- write()
  +---- other reuse

10. When NOT to cache​

Caching everything is an anti-pattern.

Avoid caching when:

  • the DataFrame is used only once;
  • the dataset is enormous and cannot be usefully retained;
  • memory pressure causes eviction or spilling;
  • the pipeline is cheap to recompute;
  • downstream logic does not reuse the same result;
  • the cache competes with more important workloads.

The right question is:

Will avoiding recomputation save more resources than persistence consumes?

Spark SQL provides cache inspection and unpersist operations through its catalog/DataFrame APIs. Spark SQL Performance Tuning

11. Cache vs checkpoint​

These concepts are related but serve different architectural purposes.

Cache / persist​

Primarily about performance and reuse.

Long lineage
    |
cache
    |
faster reuse

Checkpoint​

Primarily about cutting lineage and establishing a checkpoint boundary.

Source
  |
many transformations
  |
very long lineage
  |
checkpoint
  |
new starting point
  |
more transformations

Checkpointing is useful when lineage becomes very long or when you need a controlled recovery boundary.

Do not use checkpointing merely because a pipeline is slow; first identify the actual bottleneck.

Apache Spark documents that checkpointing can remove references to parent RDDs and persist checkpointed data to a checkpoint directory. Apache Spark — RDD API

12. Cache vs checkpoint vs Delta materialization​

For enterprise Databricks architecture, distinguish three patterns:

PatternPrimary purposeTypical lifetime
Cache / persistAccelerate repeated computationWorkload/session
CheckpointBreak lineage / recovery boundaryControlled checkpoint lifecycle
Delta tableDurable business/data boundaryPersistent

Decision model:

Need reuse only inside current workload?
          |
         YES
          |
     cache/persist

Need to break very long lineage?
          |
         YES
          |
      checkpoint

Need durable reusable data product?
          |
         YES
          |
      Delta table

13. Driver memory: collect()​

One of the most dangerous action patterns is:

df.collect()

This asks Spark to return records to the driver.

Executor 1 ─┐
Executor 2 ─┤
Executor 3 ─┼──> Driver memory
Executor 4 ─┤
Executor 5 ─┘

If the result is huge, the driver can run out of memory.

Safer patterns​

Instead of:

rows = df.collect()

consider:

df.show(20)

or:

df.limit(1000).collect()

or write the result to a distributed data store.

Do not use collect() as a casual inspection technique on production-scale data.

14. toPandas() is also a driver-side boundary​

This is equally important:

pdf = df.toPandas()

The resulting pandas DataFrame lives on the driver.

Distributed Spark DataFrame
          |
          | toPandas()
          v
      Driver memory
          |
          v
     pandas DataFrame

Use it only when the resulting dataset is intentionally small enough for driver memory.

A common enterprise mistake is to build a distributed pipeline and then accidentally destroy the distributed architecture at the final step with collect() or toPandas().

15. show() is an action too​

This is frequently misunderstood.

df.show()

does not merely print metadata.

It requires Spark to execute enough of the plan to produce the requested rows.

Therefore:

df = expensive_pipeline()

does not necessarily perform the expensive computation, while:

df.show()

can trigger execution.

This explains why a notebook cell containing only show() can suddenly create a Spark job.

16. count() is useful but not free​

df.count()

is commonly used for validation.

However, on a large dataset it can require substantial computation.

Bad pattern:

df.count()
df.show()
df.count()
df.write.save(...)

In a production pipeline, every unnecessary action should be questioned.

Better:

Build plan
   |
validate only where required
   |
persist/materialize if multiple expensive consumers exist
   |
perform business output

17. Write operations are execution triggers​

Consider:

(
    df.filter("status = 'COMPLETE'")
      .write
      .mode("append")
      .format("delta")
      .saveAsTable("gold.orders")
)

The write is not just a storage API call.

It is an execution request.

Spark must:

  1. read source data;
  2. apply transformations;
  3. perform required shuffles/sorts;
  4. execute tasks;
  5. produce output files;
  6. complete the write.

So the performance of a write pipeline depends on the entire upstream plan.

18. Actions and the physical plan​

Use:

df.explain("formatted")

Look for:

  • Scan;
  • Filter;
  • Project;
  • Exchange;
  • Sort;
  • Aggregate;
  • Join;
  • Broadcast;
  • Adaptive execution operators.

The action itself is not normally the expensive business operation; it is the trigger that causes Spark to execute the plan.

The physical plan is the bridge between Python/SQL code and actual distributed execution.

19. Complete example​

from pyspark.sql import functions as F

orders = spark.table("silver.orders")

customer_sales = (
    orders
    .filter(F.col("status") == "COMPLETE")
    .select("customer_id", "amount")
    .groupBy("customer_id")
    .agg(F.sum("amount").alias("total_sales"))
)

customer_sales.explain("formatted")

customer_sales.count()

customer_sales.write.mode("overwrite").saveAsTable("gold.customer_sales")

Mental model​

spark.table()
     |
filter
     |
select
     |
groupBy / aggregate
     |
logical plan
     |
optimized plan
     |
physical plan
     |
count()
     |
JOB
     |
STAGES
     |
TASKS

Then the write is another execution request unless the workload can reuse an already materialized representation.

20. Hands-on lab — prove laziness​

Run:

df = spark.range(0, 100_000_000)

filtered = df.filter("id % 10 = 0")

print("Transformations defined")

print(filtered.count())

Observe​

  1. How long does the transformation definition take?
  2. When does Spark UI show execution?
  3. How many jobs are created?
  4. How many stages?
  5. How many tasks?

Learning objective​

Prove experimentally that constructing the transformation chain is different from executing it.

21. Hands-on lab — repeated actions​

df = (
    spark.range(0, 50_000_000)
    .filter("id % 2 = 0")
    .withColumn("bucket", F.col("id") % 100)
)

df.count()
df.show(10)

Inspect Spark UI.

Then compare with:

df_cached = df.cache()

df_cached.count()
df_cached.show(10)

Questions​

  • How many jobs did you observe?
  • Did the second action reuse cached data?
  • What changed in the Storage tab?
  • What happened to execution time?
  • What happened to executor memory?

22. Hands-on lab — driver safety​

Start with:

small = spark.range(0, 1000)
small.collect()

Then reason about:

large.collect()

Do not blindly execute the second example on production-scale data.

Instead ask:

Where will the result physically live?

If the answer is “driver memory”, verify that the result is intentionally small.

23. Hands-on lab — cache lifecycle​

df = (
    spark.range(0, 20_000_000)
    .filter("id % 3 = 0")
)

df.cache()

df.count()

print(df.storageLevel)

df.unpersist()

Observe:

cache()
   ↓
first action
   ↓
materialization
   ↓
reuse
   ↓
unpersist()

24. Production debugging workflow​

When a notebook cell suddenly takes 20 minutes, do not immediately increase cluster size.

Step 1 — Identify the action​

Ask:

What triggered execution?

Was it count, show, collect, write, toPandas, or another action?

Step 2 — Inspect the plan​

df.explain("formatted")

Step 3 — Inspect Spark UI​

Look for:

  • number of jobs;
  • stage duration;
  • shuffle;
  • task skew;
  • input/output;
  • spill;
  • executor utilization.

Step 4 — Identify recomputation​

Ask whether the same expensive lineage is being executed multiple times.

Step 5 — Decide whether persistence is justified​

Only then consider cache, persist or materialization.

25. Common anti-patterns​

Anti-pattern 1 — collect everything​

rows = production_df.collect()

Problem: driver memory risk.

Anti-pattern 2 — repeated count()​

df.count()
df.count()
df.count()

Problem: unnecessary execution unless reuse is intentionally optimized.

Anti-pattern 3 — blind cache​

Caching every DataFrame creates memory pressure and can cause eviction.

Anti-pattern 4 — assuming cache() executes​

Developers sometimes assume cache() has already loaded the dataset. It has not necessarily done so.

Anti-pattern 5 — toPandas on large data​

df.toPandas()

Problem: distributed data is brought to the driver.

Anti-pattern 6 — debugging with expensive actions repeatedly​

A notebook containing dozens of count(), show() and collect() calls can become an accidental performance test.

26. Architect scenario​

You have a 5 TB silver pipeline:

S3 / Delta
   |
filter
   |
join
   |
aggregation
   |
5 downstream consumers

The team runs the expensive DataFrame five times.

An engineer proposes:

“Let's increase the cluster size.”

An architect asks:

  1. Are the five consumers executing the same lineage?
  2. Can the intermediate result be reused?
  3. Is cache appropriate?
  4. Should the intermediate result become a Delta data product?
  5. Is the lineage too long?
  6. Are downstream consumers actually querying different subsets?
  7. Would a single materialized Gold layer be a better architecture?

The correct answer depends on workload characteristics.

Compute scaling is not a substitute for eliminating unnecessary recomputation.

27. Enterprise decision matrix​

SituationPreferred thinking
One expensive actionOptimize the plan first
Same result reused several timesConsider cache/persist
Long lineage / iterative computationConsider checkpoint
Durable cross-job reuseConsider Delta materialization
Small diagnostic resultshow/take/limited collect
Large resultKeep distributed
Driver OOMRemove collect/toPandas or reduce result
Repeated expensive actionsInvestigate persistence/materialization
Slow writeAnalyze the complete upstream plan

28. Architect mental model​

Always translate:

df.filter(...).join(...).groupBy(...).count()

into:

CODE
 ↓
TRANSFORMATIONS
 ↓
LINEAGE
 ↓
OPTIMIZATION
 ↓
PHYSICAL PLAN
 ↓
ACTION
 ↓
JOB
 ↓
STAGES
 ↓
TASKS
 ↓
EXECUTORS
 ↓
RESULT / WRITE

Then ask:

Where is the data actually moving?

Where is it being recomputed?

Where does the result live?

Which boundary is distributed and which boundary is driver-side?

That is the architect-level way to reason about Spark execution.

29. Interview questions​

  1. What is lazy evaluation? Spark delays execution of transformations until an action requires a result.
  2. What is an action? An operation that requests a result or causes a side effect and therefore triggers execution.
  3. Does cache() immediately execute a DataFrame? No. It marks the dataset for persistence; computation happens when an action materializes it.
  4. Why can two actions be expensive? Without persistence or another reuse mechanism, each action can recompute upstream lineage.
  5. Why is collect() dangerous? It brings the result to driver memory and can cause driver OOM.
  6. Is show() an action? Yes.
  7. Is count() an action? Yes.
  8. Is write() an action? A write operation triggers distributed execution to produce the output.
  9. Cache vs checkpoint? Cache/persist primarily improves reuse; checkpoint establishes a durable lineage boundary.
  10. When should you materialize to Delta? When the result is a durable, reusable data boundary rather than merely a temporary performance cache.
  11. Why should you not always increase cluster size? The bottleneck may be recomputation, skew, shuffle, poor partitioning, driver-side collection or inefficient logic.
  12. How do you debug an unexpectedly slow action? Identify the action, inspect the physical plan, then use Spark UI to locate the expensive stage, shuffle, skew, spill or resource bottleneck.

30. Final cheat sheet​

TRANSFORMATION
     |
     | lazy
     v
PLAN / LINEAGE
     |
     | ACTION
     v
OPTIMIZE
     |
     v
PHYSICAL PLAN
     |
     v
JOB
     |
     +--> STAGES
             |
             +--> TASKS
                     |
                     v
                 EXECUTORS
                     |
                     v
              RESULT / WRITE

Remember​

  • Transformations build the plan.
  • Actions trigger execution.
  • Lazy evaluation enables optimization.
  • Repeated actions can cause repeated computation.
  • Cache/persist is for useful reuse, not decoration.
  • Checkpoint breaks lineage; it is not simply a faster cache.
  • collect() and toPandas() cross into driver memory.
  • Writes are execution triggers.
  • Physical plan + Spark UI are the source of truth.

Official references​

Next​

03.5 DAG — understand how Spark converts lineage into a Directed Acyclic Graph, how dependencies become stage boundaries, and how the DAG maps to actual execution.