03.6 Partitions

Architect rule: A partition is a logical slice of distributed data and a fundamental unit of Spark parallelism. Partition design affects task concurrency, shuffle volume, memory pressure, file layout, runtime, and cost.

1. What is a partition?

A partition is a logical slice of a distributed dataset. Spark can process different partitions concurrently across executor cores.

Conceptually:

Dataset
   |
   +-- Partition 0 ----> Task 0 ----> Executor
   +-- Partition 1 ----> Task 1 ----> Executor
   +-- Partition 2 ----> Task 2 ----> Executor
   +-- Partition N ----> Task N ----> Executor

The important distinction is:

• Partition = data slice
• Task = one execution attempt for one partition in a stage
• Executor core = capacity to execute tasks concurrently

Therefore, more partitions can create more parallelism, but more partitions do not automatically mean better performance.

Apache Spark describes an RDD as a collection partitioned across nodes and notes that Spark runs one task for each partition. Its tuning guide also emphasizes sufficient parallelism to keep cluster resources utilized.

• Apache Spark RDD Programming Guide
• Apache Spark Tuning Guide

2. Why partitions matter

Area	Partition impact
Parallelism	Determines how much independent work can run concurrently
CPU utilization	Too few partitions can leave executor cores idle
Memory	Very large partitions can create memory pressure
Shuffle	Shuffle output is divided into partitions
Joins	Partitioning influences how records are colocated
Aggregations	Determines the number and size of reduce-side tasks
Skew	One oversized partition can become a straggler
File output	Output partition count influences file count
Cost	Poor parallelism can increase runtime and compute spend
AQE	Adaptive execution can coalesce or split shuffle partitions

Do not memorize a universal partition count. Tune from workload evidence.

3. Partitions → tasks → cores

Suppose a stage has 1,000 partitions and the cluster has 20 executor cores available for the stage.

Spark does not run all 1,000 tasks simultaneously. It schedules tasks in waves according to available execution slots.

1,000 partitions
       |
       v
1,000 tasks
       |
       +-------------------------------+
       | scheduler                      |
       v                                |
  20 tasks at a time  <-----------------+
       |
       v
  next wave of tasks
       |
       v
  ... until all partitions finish

If there are too few partitions, the opposite problem appears. For example, 8 partitions on a cluster with 100 usable cores cannot keep all cores busy during that stage.

The first sizing question

Ask:

1. How many cores can execute this stage concurrently?
2. How much data is processed by each partition?
3. How long does the typical task run?
4. Is there skew where a few partitions are much larger?
5. Is the operation before or after a shuffle?

These questions are more useful than blindly setting repartition(500).

4. Where partitions come from

Partitions can be introduced or changed by several mechanisms.

4.1 Input data

When Spark reads distributed files, the input source determines how the data is split into input partitions. File sizes, file boundaries, source configuration, and the execution engine all matter.

Do not assume that one physical file always equals exactly one Spark partition.

4.2 Existing dataset

A transformation often preserves the existing partitioning characteristics when it can do so without redistributing the data.

4.3 Shuffle

Operations such as joins, aggregations, grouping, and repartitioning can create a new partitioning boundary.

Before shuffle                 After shuffle
Partition 0                    Partition 0
Partition 1       ----->       Partition 1
Partition 2                    Partition 2
Partition 3                    ...
                                 Partition N

The shuffle topic is covered in 03.7 Shuffle.

4.4 Explicit repartitioning

You can intentionally change the partition count:

df2 = df.repartition(200)

This generally causes a full shuffle.

5. Inspecting partition counts

For RDD-backed inspection:

df.rdd.getNumPartitions()

For a production DataFrame workload, do not stop at this number. Also inspect the physical plan and Spark UI to understand:

• input size
• number of tasks
• task duration distribution
• shuffle read/write
• spill
• skew
• output file counts

Example:

print(df.rdd.getNumPartitions())
df.explain("formatted")

6. repartition()

repartition() redistributes data so that the resulting dataset has the requested number of partitions.

df2 = df.repartition(200)

You can also repartition by one or more columns:

df2 = df.repartition(200, "customer_id")

This is useful when you deliberately want records with the same partitioning key to be colocated for subsequent processing.

Cost

Repartitioning normally means network redistribution:

Executor A ----+
Executor B ----+----> Shuffle ----> New partitions
Executor C ----+

Therefore, repartition is a performance tool, not a free metadata operation.

When repartition is useful

• Increasing parallelism before a large operation
• Reducing severe imbalance
• Establishing useful partitioning for downstream work
• Controlling output file count when writing
• Recovering from a badly partitioned upstream stage

When it can hurt

• Repartitioning a small dataset unnecessarily
• Repartitioning repeatedly between transformations
• Creating a huge shuffle for a workload that did not need it
• Choosing a high partition count that creates many tiny tasks/files

7. coalesce()

coalesce() is commonly used to reduce the number of partitions without requiring a full redistribution.

df2 = df.coalesce(50)

Typical use case:

Large filtered dataset
        |
        v
Many partitions
        |
     filter
        |
        v
Much less data
        |
        v
coalesce to fewer partitions
        |
        v
Write output

coalesce() is particularly useful after a filter has dramatically reduced the amount of data.

coalesce() is not a universal replacement for repartition(). If you need balanced redistribution or want to increase partition count, use repartition().

8. repartition() vs coalesce()

Decision	repartition()	coalesce()
Increase partition count	Yes	No
Reduce partition count	Yes	Yes
Full redistribution	Generally yes	Designed to avoid a full shuffle when reducing
Good for balancing data	Yes	Limited
Good after aggressive filtering	Sometimes excessive	Often useful
Can increase network cost	Yes	Usually less when reducing

Example decision:

Need 20 -> 100 partitions?
        |
        +--> repartition(100)

Need 500 -> 50 partitions after filtering?
        |
        +--> consider coalesce(50)

Need 500 -> 50 balanced partitions by key?
        |
        +--> repartition(50, key)

9. Partition size is more important than partition count alone

Two jobs can both have 1,000 partitions and behave completely differently.

Job A

• 1 TB total data
• 1,000 partitions
• approximately 1 GB per partition

Job B

• 10 GB total data
• 1,000 partitions
• approximately 10 MB per partition

Job B may have excessive scheduling and file overhead, while Job A may have partitions large enough to create memory, spill, or straggler problems depending on the workload.

Think in terms of data per partition, not only partition count.

10. The small-partition problem

Too many tiny partitions can produce:

• large numbers of short-lived tasks
• scheduler overhead
• excessive task-launch overhead
• many small output files
• metadata pressure
• inefficient downstream reads

Example:

10 GB dataset
        |
        +-- 10,000 partitions
              |
              +-- ~1 MB each
              +-- many tiny tasks
              +-- many tiny output files

This is one form of the small-files problem.

Do not solve it by blindly setting a huge partition count.

11. The oversized-partition problem

Too few partitions can create the opposite failure mode:

1 TB dataset
        |
        +-- 8 partitions
              |
              +-- ~125 GB each
              +-- long-running tasks
              +-- memory pressure
              +-- spill
              +-- poor parallelism

One task processing dramatically more data than its peers can become the stage bottleneck.

12. Partition skew

Skew occurs when partitions are highly imbalanced.

Example:

Partition 0   2 GB
Partition 1   2 GB
Partition 2   1.8 GB
Partition 3   2.1 GB
Partition 4   95 GB   <-- straggler

Most tasks finish quickly, but one task continues running.

On the Spark UI this often appears as a long tail in task duration.

Common causes include:

• highly concentrated keys
• popular customer/product IDs
• geographic or temporal concentration
• bad partitioning strategy
• joins involving a skewed key

Skew mitigation is covered in depth in the performance-engineering phase and the shuffle module.

13. AQE changes the partition story

Modern Spark can adapt partitioning during execution.

Apache Spark's Adaptive Query Execution (AQE) can coalesce contiguous shuffle partitions to avoid excessive small tasks and can use runtime statistics to make better decisions. In Spark 4.3.0, adaptive coalescing includes controls such as spark.sql.adaptive.coalescePartitions.enabled, spark.sql.adaptive.coalescePartitions.initialPartitionNum, and spark.sql.adaptive.advisoryPartitionSizeInBytes.

• Spark SQL Performance Tuning
• Spark Configuration

Conceptually:

Initial shuffle partitions
        |
        v
Runtime statistics
        |
        v
Adaptive optimization
        |
        +--> coalesce small partitions
        +--> handle skew
        +--> adjust execution strategy

This means a modern architect should distinguish between:

1. static partitioning decisions made by your code/configuration, and
2. runtime adaptive decisions made by Spark.

14. spark.sql.shuffle.partitions

For Spark SQL/DataFrame shuffle operations, spark.sql.shuffle.partitions is a key configuration that defines the initial number of shuffle partitions when no more specific setting is supplied.

Example:

spark.conf.set("spark.sql.shuffle.partitions", 400)

Do not treat 200, 400, or any other number as a universal best practice.

The right value depends on:

• data volume
• cluster parallelism
• task duration
• shuffle size
• skew
• AQE behavior
• workload type

With AQE enabled, Spark can coalesce shuffle partitions at runtime.

15. A practical sizing approach

Use this workflow instead of memorizing a formula.

Step 1 — Measure input

Determine the actual data volume processed by the stage.

Step 2 — Measure cluster parallelism

Determine the usable executor cores for the workload.

Step 3 — Observe task duration

Very tiny tasks may indicate excessive partitioning. Extremely long tasks may indicate insufficient partitioning or skew.

Step 4 — Inspect shuffle

Look at shuffle read/write and partition-size distribution.

Step 5 — Check skew

Compare median task duration with the slowest tasks.

Step 6 — Change one variable

Test a partitioning change rather than simultaneously changing cluster size, join strategy, caching, and partition count.

Step 7 — Compare evidence

Compare wall-clock time, DBU/compute cost, shuffle, spill, task distribution, and output files.

16. Partitioning before joins

Suppose two large datasets are joined on customer_id.

orders = orders.repartition(400, "customer_id")
customers = customers.repartition(400, "customer_id")

joined = orders.join(customers, "customer_id")

Explicit repartitioning may sometimes help a workload, but it can also add expensive shuffles that Spark's optimizer could have handled differently.

Never add repartition calls just because a join exists. First inspect the physical plan and runtime metrics.

Broadcast joins, shuffle joins, AQE, and join optimization are covered later.

17. Partitioning before writes

Output partition count affects file count.

Example:

df.repartition(100).write.format("delta").mode("overwrite").save(path)

Conceptually, more partitions can produce more output files.

That can improve parallel write throughput, but too many partitions can create small files.

Therefore output partitioning is a trade-off between:

• write parallelism
• file size
• downstream read efficiency
• metadata overhead
• compaction requirements

Do not confuse Spark execution partitions with Delta Lake table partitioning such as PARTITIONED BY (date); they are related concepts but not the same thing.

18. Spark partitioning vs Delta table partitioning

Spark execution partition	Delta table partitioning
Runtime execution concept	Physical data-layout concept
Drives tasks	Organizes table data by partition columns
Can change during execution	Persists as directory/data layout characteristics
Controlled by Spark execution operations/configuration	Defined by table design
Important for parallelism	Important for data skipping/layout and maintenance

This distinction is critical for Databricks architects.

19. Hands-on lab — inspect partitions

Create a small dataset:

from pyspark.sql import functions as F

df = spark.range(0, 1_000_000)
print("Initial partitions:", df.rdd.getNumPartitions())

Now test different partition counts:

for n in [8, 32, 128, 512]:
    test_df = df.repartition(n)
    print(n, test_df.rdd.getNumPartitions())

Inspect the plan:

test_df.explain("formatted")

Look for an Exchange introduced by repartitioning.

20. Hands-on lab — compare repartition and coalesce

large_df = spark.range(0, 5_000_000).repartition(200)
filtered_df = large_df.filter(F.col("id") < 100_000)

print("Before:", filtered_df.rdd.getNumPartitions())

coalesced_df = filtered_df.coalesce(20)
repartitioned_df = filtered_df.repartition(20)

print("Coalesced:", coalesced_df.rdd.getNumPartitions())
print("Repartitioned:", repartitioned_df.rdd.getNumPartitions())

Compare the physical plans and execution behavior.

Questions:

1. Which operation introduces an Exchange?
2. Which operation can reduce partitions without a full redistribution?
3. Which produces more balanced partitions?
4. Which is better for your specific workload?

21. Hands-on lab — observe skew

Create intentionally skewed data:

from pyspark.sql import functions as F

skewed = (
    spark.range(0, 10_000_000)
    .withColumn('key', F.when(F.col("id") < 9_000_000, F.lit("HOT")).otherwise(F.col("id").cast("string"))))

Group by the key and inspect the Spark UI:

skewed.groupBy("key").count().show()

Look for a stage where one task takes dramatically longer than the others.

22. Production troubleshooting decision tree

Symptom: CPU is underutilized

Check:

• Too few input partitions?
• Too few shuffle partitions?
• A narrow stage with insufficient parallel work?
• Upstream source limiting parallelism?

Symptom: thousands of tiny tasks

Check:

• Too many partitions?
• Small files?
• Excessive repartition calls?
• AQE coalescing behavior?

Symptom: one task runs much longer

Check:

• Data skew?
• Hot key?
• Uneven input files?
• Partitioning mismatch?

Symptom: executor OOM

Check:

• Oversized partitions?
• Large aggregation state?
• Skew?
• Broadcast size?
• Per-task memory pressure?

Symptom: too many output files

Check:

• Excessive output partitions?
• Small upstream files?
• Missing compaction/optimization strategy?

23. Architect-level partition checklist

Before changing partition count, answer:

• What is the input volume?
• What is the post-filter volume?
• How many executor cores are available?
• Is this a narrow or wide stage?
• Is a shuffle involved?
• What is the median task size?
• What is the largest task size?
• Is there skew?
• How much shuffle is produced?
• Is AQE enabled?
• How many output files are produced?
• What is the cost impact?

Only then choose a partitioning strategy.

24. Common anti-patterns

Anti-pattern 1 — Always use 200 partitions

Historical defaults are not workload-specific engineering.

Anti-pattern 2 — Always use one partition for output

df.coalesce(1)

This can serialize the final write through a single partition and create a major bottleneck for large datasets.

Anti-pattern 3 — Repartition after every transformation

Each unnecessary redistribution can add network, CPU, and disk overhead.

Anti-pattern 4 — Ignore skew

Average task duration can look healthy while one straggler determines total stage runtime.

Anti-pattern 5 — Confuse table partitioning with Spark partitions

These solve different problems.

25. Interview questions

Beginner

1. What is a Spark partition?
2. How does a partition relate to a task?
3. Why can too few partitions hurt performance?
4. Why can too many partitions hurt performance?

Intermediate

5. Difference between repartition() and coalesce()?
6. What operations cause a shuffle?
7. How do you inspect partition count?
8. What is partition skew?
9. How can partition count affect output files?

Architect

10. How would you size partitions for a 5 TB workload?
11. How would you diagnose one task taking 20 minutes while the others finish in 2 minutes?
12. How does AQE change static partition-tuning decisions?
13. How do Spark execution partitions differ from Delta table partitions?
14. When would explicit repartitioning make a job slower?
15. How would you optimize a workload with both skew and small-file problems?

26. Cheat sheet

Concept	Mental model
Partition	Unit of distributed data
Task	Work performed for one partition in a stage
More partitions	More potential parallelism, but more overhead
Too few	Underutilization / large tasks
Too many	Tiny tasks / scheduling and file overhead
repartition()	Redistribute data; useful for balancing or increasing partitions
coalesce()	Reduce partitions with less redistribution in suitable cases
Skew	Uneven partition sizes/work
AQE	Runtime adaptation of query execution
Output files	Strongly influenced by write-side partitioning

27. The architect mental model

Remember this chain:

Data volume
    ↓
Partitioning
    ↓
Tasks
    ↓
Executor parallelism
    ↓
Shuffle / memory / skew
    ↓
Stage duration
    ↓
Job runtime + compute cost

The goal is not maximum partitions.

The goal is appropriate parallelism with balanced work and acceptable overhead.

Next

Continue to 03.7 Shuffle — where partitioning becomes especially important because Spark redistributes data across the cluster.