03.3 Transformations

Phase 3 — Apache Spark Fundamentals

Spark transformations describe how a new distributed dataset is derived from an existing dataset. The key architectural idea is: a transformation builds a plan; an action causes Spark to execute that plan.

1. Transformation mental model

Source → Filter → Select → Derived columns → Logical/physical plan → Action → Jobs → Stages → Tasks

Example:

orders = spark.table('sales.orders')
result = (orders.filter("status = 'COMPLETE'")
          .select('customer_id', 'amount')
          .withColumn('tax', F.col('amount') * 0.18))

Writing the transformation chain does not mean the entire dataset has already been processed. Spark uses lazy evaluation and builds the required execution plan. The official Spark guide describes transformations as lazy and actions as operations that trigger computation. Apache Spark — RDD Programming Guide

2. Transformation vs action

Transformation	Action
Creates a new dataset/plan	Produces a result or side effect
Usually lazy	Triggers execution
Builds lineage	Executes lineage
filter	count
select	collect
withColumn	show
join	write

filtered = df.filter('amount > 100')
selected = filtered.select('customer_id', 'amount')
selected.count()  # action

3. Why transformations are lazy

Without lazy evaluation, every intermediate operation could force unnecessary work. Spark can instead reason about the complete chain and optimize it before execution.

filter → filter → projection → aggregation
              ↓
        optimized plan
              ↓
            action

4. Narrow transformations

A narrow transformation can generally execute without broad redistribution of data. Common examples include filter, select, withColumn, and drop.

Partition 0 → Partition 0'
Partition 1 → Partition 1'
Partition 2 → Partition 2'

Multiple narrow operations can often be pipelined into one stage.

5. Wide transformations

A wide transformation requires data to be redistributed across partitions. Common examples include groupBy, distinct, orderBy, repartition, and many joins.

P0 ──┐
P1 ──┼──> SHUFFLE ──> Q0
P2 ──┤              Q1
P3 ──┘              Q2

Shuffle can involve network I/O, serialization, disk I/O, memory pressure and additional stages. Spark documents shuffle as the mechanism for redistributing data across partitions. Apache Spark — RDD Programming Guide

6. Narrow vs wide

Dimension	Narrow	Wide
Redistribution	Usually no	Yes / potentially
Shuffle	Usually no	Usually
Stage boundary	Usually no	Usually
Network pressure	Lower	Potentially high
Main concern	CPU / scan efficiency	Shuffle / skew / memory / network
Examples	filter, select	groupBy, distinct, repartition

Narrow = data can stay local. Wide = data must move.

7. Core DataFrame transformations

select

Projects required columns. Selecting fewer columns can reduce scan, memory, network, shuffle and serialization costs.

df.select('customer_id', 'amount', 'order_date')

filter / where

Reduces rows. Filtering before expensive operations is often beneficial when semantics permit.

df.filter(F.col('amount') > 100)

withColumn

Creates a derived expression without inherently requiring a shuffle.

df.withColumn('net_amount', F.col('amount') - F.col('discount'))

drop

Removes unnecessary columns and keeps the dataset narrow.

8. Transformations that commonly require redistribution

groupBy

Grouping often requires keys to be colocated.

df.groupBy('customer_id').agg(F.sum('amount').alias('total_amount'))

distinct

A global distinct commonly requires redistribution by the values being deduplicated.

orderBy

Global ordering is expensive because Spark must coordinate records across partitions.

repartition

repartition() deliberately changes partition distribution and generally performs a shuffle.

coalesce

coalesce() is commonly used to reduce partition count without a full redistribution. It can be useful after a large filter, but excessive coalescing can reduce parallelism.

Feature	repartition	coalesce
Increase partitions	Yes	No
Decrease partitions	Yes	Yes
Redistribution	Generally full shuffle	Designed to reduce shuffle
Typical purpose	Redistribute / balance	Reduce partitions

Spark's documentation describes repartition as reshuffling data and coalesce as reducing partitions. Apache Spark — RDD Programming Guide

9. Join transformations

A join is logically a transformation, but its physical implementation can differ.

Join
 ├── BroadcastHashJoin
 ├── SortMergeJoin
 └── other physical strategies

If a small side can be broadcast, Spark can avoid shuffling the large side. Broadcasting something too large can create executor memory pressure.

Never infer the physical strategy from the API call alone. Inspect the physical plan.

10. Window transformations

w = Window.partitionBy('customer_id').orderBy('order_date')
result = df.withColumn('running_total', F.sum('amount').over(w))

Windows can be sensitive to partitioning, sorting, skew, large groups and memory. A heavily skewed window partition can become a major bottleneck.

11. explode and row multiplication

df.select('order_id', F.explode('items').alias('item'))

One input row can become many output rows. If 10 million rows each contain 50 items, the conceptual maximum is around 500 million output rows before downstream filtering.

Prefer filtering and column reduction before explosion when semantics allow.

12. map, flatMap and mapPartitions

RDD APIs include map, flatMap and mapPartitions. Modern Databricks workloads should generally prefer DataFrame/Spark SQL expressions for structured data.

mapPartitions processes an iterator for an entire partition and can be useful for carefully designed per-partition initialization, but it introduces risks around external connections, memory and fault handling.

13. Lineage

Source
  ↓
Filter
  ↓
Projection
  ↓
Aggregation

This chain is lineage. Spark can use lineage to recompute lost intermediate data, contributing to distributed fault tolerance.

14. Transformation fusion / pipelining

Read → Filter → Select → Expression → Task output

Spark can often pipeline compatible narrow operations instead of materializing every intermediate result. Therefore the number of transformations in source code is not the number of stages.

15. Python UDF warning

Python UDFs can introduce Python/JVM serialization boundaries and can reduce optimization opportunities compared with native Spark expressions.

Prefer built-in functions such as when, regexp_extract, transform, element_at, coalesce, explode and SQL expressions when they can express the requirement.

Approach	Typical preference
Native Spark expression	First choice
SQL expression	First choice
Pandas UDF	Targeted use
Python UDF	Use when genuinely necessary

16. Transformation performance review

When reviewing a pipeline, ask:

1. What is the input size?
2. Which transformations reduce rows?
3. Which reduce columns?
4. Where does data move?
5. Where can shuffle occur?
6. Where can rows multiply?
7. Where can sorting occur?
8. Where can Python execution occur?
9. What physical operators did Spark choose?

17. Common anti-patterns

Unnecessary repartition

Do not add repartition(1000) without a concrete reason. First inspect partition counts, skew and downstream parallelism.

Global sort without a business requirement

A global orderBy can introduce expensive redistribution and sorting. Sort at the presentation/query layer when global ordering is not part of the data contract.

Explode too early

Filter and project before exploding when semantics permit.

Repeated actions

Running count(), show() and write() independently can cause upstream work to be executed repeatedly. Consider persistence or materialization only when reuse justifies it.

Blind caching

Do not cache every intermediate DataFrame. Cache when repeated reuse makes the memory/storage trade-off worthwhile.

18. Hands-on lab — narrow pipeline

df = spark.range(0, 10_000_000)
result = (df.filter('id % 2 = 0')
          .select('id')
          .withColumn('double_id', F.col('id') * 2))
result.count()
result.explain('formatted')

Identify the scan, filter, projection, expression and whether an Exchange exists.

19. Hands-on lab — introduce a shuffle

result = (df.withColumn('bucket', F.col('id') % 100)
          .groupBy('bucket').count())
result.count()
result.explain('formatted')

Inspect Exchange, shuffle read/write, stage boundaries and task distribution in Spark UI.

20. Hands-on lab — repartition experiment

Compare:

df.count()
df.repartition(200).count()

Do not search for a universally optimal number. Prove how a transformation can change execution topology.

21. Hands-on lab — explode amplification

sample = spark.createDataFrame([(1, ['A','B','C']), (2, ['D','E'])], ['order_id','items'])
exploded = sample.select('order_id', F.explode('items').alias('item'))
sample.count(), exploded.count()

Observe the row multiplication.

22. Hands-on lab — native expression vs UDF

Implement equivalent logic using a native expression and a Python UDF. Compare explain('formatted') and then benchmark on a sufficiently large dataset. Do not treat a tiny benchmark as production evidence.

23. Architect scenario

Suppose a 1 TB pipeline is:

1 TB source → explode → join 500 GB dimension → groupBy → orderBy → write

Before increasing cluster size, ask:

• Can filtering happen before explode?
• Are unnecessary columns carried through?
• Why is the dimension joined this way?
• Is the join skewed?
• How much data is shuffled?
• Is global ordering actually required?
• Are Python UDFs involved?
• Which stage dominates runtime?

The architect's job is to reduce unnecessary work before simply adding compute.

24. Transformation decision framework

Need transformation
      ↓
Can native Spark expression do it?
   /             \
 YES             NO
  ↓               ↓
Use native      Consider UDF
  ↓
Will it reduce rows/columns?
  ↓
Do it before expensive operations when semantics allow
  ↓
Will it redistribute data?
  ↓
Check shuffle / skew / partitioning

25. Architect mental model

Translate code into:

Transformation
      ↓
Rows changed?
      ↓
Columns changed?
      ↓
Redistribution?
      ↓
Sort?
      ↓
Row multiplication?
      ↓
Python execution?
      ↓
Physical operator
      ↓
Partitions → stages → executors → cost

26. Interview questions

1. What is a transformation? An operation that derives a new distributed dataset/plan from an existing one.
2. Why are transformations lazy? To let Spark optimize the complete computation before execution.
3. What is a narrow transformation? One that generally avoids broad redistribution.
4. What is a wide transformation? One that requires redistribution across partitions.
5. Does every transformation create a stage? No. Multiple narrow operations can be pipelined.
6. Does groupBy cause a shuffle? Commonly, when grouping keys are not already suitably partitioned.
7. repartition vs coalesce? Repartition generally reshuffles; coalesce is designed to reduce partitions with less redistribution.
8. Why prefer native expressions over Python UDFs? Better optimizer visibility and less Python serialization overhead in many cases.
9. Why is explode dangerous? It can multiply row counts dramatically.
10. Why is global orderBy expensive? It requires coordination, redistribution and sorting.
11. How do you find an expensive transformation? Inspect physical plan and Spark UI for Exchange, shuffle, skew, spill, CPU and task duration.
12. Does more source code transformations mean more scans? No. Spark can optimize the transformation chain into a physical plan.

27. Final cheat sheet

TRANSFORMATION
      |
      +--> narrow → usually no shuffle → pipeline-friendly
      |
      +--> wide   → redistribution → shuffle / stage boundary

Transformation builds the plan.

Action triggers execution.

Narrow work can usually stay local.

Wide work moves data.

Reduce rows and columns before expensive operations.

Prefer native Spark expressions over UDFs when possible.

Physical plan + Spark UI are the source of truth.

Official references

• Apache Spark — RDD Programming Guide
• Apache Spark — SQL Performance Tuning
• Apache Spark — DataFrame API

Next

03.4 Actions & Lazy Evaluation — understand exactly when Spark executes work, how actions create jobs, why repeated actions can recompute data, and when caching or materialization makes architectural sense.