06.7 VACUUM & Delta File Lifecycle

Delta Lake separates logical table history from physical data-file lifecycle.

A Delta transaction can remove a file from the current snapshot without immediately deleting that physical file from object storage.

Animated Delta Lake VACUUM architecture

1. The problem VACUUM solves

Every write can create new files. Updates and deletes can make older files obsolete. Delta keeps those files available for a retention window because readers and historical snapshots can still depend on them.

Write v100 → file-A
UPDATE → v101 → file-B
DELETE → v102 → more files become stale
retention window → VACUUM → physical cleanup

Without lifecycle management, object storage can accumulate stale files.

2. Logical deletion vs physical deletion

Logical deletion

The transaction log says that a file is no longer part of the current table snapshot. The object can still exist in S3, ADLS or GCS.

Physical deletion

The object is actually removed from cloud storage.

Delta transaction
      ↓
logical removal
      ↓
retention window
      ↓
VACUUM
      ↓
physical deletion

UPDATE / DELETE / MERGE change table state; VACUUM cleans physical storage.

See Databricks — VACUUM.

3. Why Delta cannot delete files immediately

Consider a reader that started against version 100 while another job commits version 101. Snapshot isolation lets the reader continue using its original files while the new version becomes current.

Reader → snapshot v100 → file-A
Writer → snapshot v101 → file-B
                     ↘ file-A becomes stale

Deleting file-A immediately could break the reader. Retention creates a safety window.

4. What VACUUM removes

VACUUM targets unreferenced data files that are older than the applicable retention threshold.

It is not:

a transaction rollback
a table restore
a compaction operation
a replacement for backup
a way to delete rows from the current logical table
a mechanism for directly rewriting transaction history

Think of it as physical lifecycle cleanup.

5. The key retention property

The main table property controlling deleted-file retention is:

delta.deletedFileRetentionDuration

Example:

ALTER TABLE main.sales.orders
SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = '30 days'
);

Databricks documents a default deleted-file retention period of 7 days. Do not treat the default as your enterprise recovery policy.

See Databricks — Delta table properties.

6. Log retention and deleted-file retention are different

Concept	Controls
Transaction-log retention	How long Delta transaction history is retained
Deleted-file retention	How long stale data files remain eligible for cleanup
VACUUM	Physical cleanup of eligible unreferenced files

A historical read needs the required transaction history and the required physical files.

The usable historical window is bounded by the weakest required lifecycle component.

7. VACUUM does not rewrite the current table

VACUUM is primarily a storage lifecycle operation, not a data transformation. The current logical table remains unchanged.

v100 → v101 → v102 → v103
                    ↓
              current state

old physical files
       ↓ retention
     VACUUM
       ↓
   removed from storage

8. Basic syntax

VACUUM main.sales.orders;

A retention override can be specified when supported and when its safety implications are understood:

VACUUM main.sales.orders RETAIN 168 HOURS;

Production policy should be documented rather than driven by ad-hoc commands.

9. Time travel boundary

Before cleanup, an old version may still be readable because its required files exist. After physical cleanup, the transaction history may still show that version even though the data files required to reconstruct it are gone.

Time travel is bounded by file retention, not just the presence of version numbers.

This is why Delta Lake time travel and physical file lifecycle must be designed together.

10. VACUUM and RESTORE

RESTORE can make an earlier state current by creating a new Delta version, but it cannot recreate physical files that have already been permanently deleted.

v100 good → v101 good → v102 bad → v103 bad
                         ↓
                    RESTORE v101
                         ↓
             requires v101's files

If those files are gone, an independent backup or replication strategy may be required.

11. VACUUM is not disaster recovery

A mature architecture separates:

Requirement	Primary capability
Operational recovery	Delta history + RESTORE
Historical analysis	Time travel
Incremental processing	Change Data Feed / CDC
Disaster recovery	Independent backup / replication
Compliance	Retention + audit architecture

Do not collapse all of these into the word “time travel.”

12. Long-running readers and streaming

Retention must account for supported reader lifetimes and streaming restart scenarios.

Consider:

maximum reader lifetime
streaming checkpoint behavior
source retention
Delta retention
downstream restart scenarios
long outages
backfills

The correct question is not only “Can I vacuum this table?” but “Could a supported reader or recovery procedure still require these files?”

13. VACUUM after MERGE / UPDATE / DELETE

Delta mutations can rewrite files rather than modifying Parquet objects in place.

MERGE / UPDATE / DELETE
        ↓
replacement files
        ↓
new Delta snapshot
        ↓
old files become unreferenced
        ↓
retention
        ↓
VACUUM

High-frequency CDC workloads can therefore create substantial stale-file volume.

See Databricks — low shuffle MERGE.

14. Small files vs stale files

These are different problems.

Problem	Typical response
Too many tiny active files	OPTIMIZE / automatic maintenance
Unreferenced stale files	Retention + VACUUM

Do not use VACUUM to solve small-file problems, and do not use OPTIMIZE as a substitute for stale-file cleanup.

15. VACUUM vs OPTIMIZE

Concern	VACUUM	OPTIMIZE
Main purpose	Remove eligible stale files	Improve active file layout
Rewrites active data	No	Yes
Removes obsolete files	Yes, when eligible	Not as its primary purpose
Improves compaction	No	Yes
Directly improves data skipping	Not directly	Can

See Databricks — optimize data file layout.

16. Predictive optimization changes the operating model

For Unity Catalog managed tables with predictive optimization enabled, Databricks can automatically perform maintenance such as OPTIMIZE, VACUUM and ANALYZE.

That changes the operating model from:

schedule every maintenance command manually

to:

define governance + retention + workload requirements
                    ↓
          platform automates routine work
                    ↓
              monitor outcomes

See Databricks — predictive optimization.

Automation does not decide your legal retention, recovery objective, reader lifetime or independent DR requirements.

17. Storage-cost model

A rough planning model is:

stale storage ≈ daily obsolete data × retention window

Real systems vary because updates rewrite different numbers of files, compaction changes layout, compression varies and tables have different mutation rates. Use actual storage metrics for budgeting.

18. SAP/BW perspective

For replicated SAP purchasing data:

SAP CDC
  ↓
Bronze Delta
  ↓
MERGE / CDC processing
  ↓
new Delta versions
  ↓
old files become stale
  ↓
retention
  ↓
VACUUM

VACUUM does not delete business history from the current logical model. It removes physical files that are no longer required by retained Delta snapshots.

19. Production safety checklist

20. Hands-on Lab — observe stale files

Create a small Delta table, perform an update, record the versions, inspect history and identify files that are no longer part of the current snapshot where your environment permits it.

Then explain:

Why the old file became logically unreferenced.
Why it should not necessarily be deleted immediately.
Which retention requirement protects it.
What event ultimately makes it eligible for physical cleanup.

21. Hands-on Lab — retention design

Design policies for:

Table	Requirement	Design focus
Bronze telemetry	3-day replay	Operational replay
Finance fact	30-day recovery	Recovery + audit
Regulatory dataset	1-year historical access	Broader retention + independent architecture

Do not simply assign a VACUUM interval. Explain the recovery, reader, audit, DR and cost implications.

22. Troubleshooting workflow

Historical read fails

Check:

requested version
transaction-log retention
deleted-file retention
whether required files were vacuumed
table location
timestamp interpretation

Storage keeps growing

Check mutation rate, stale-file volume, OPTIMIZE frequency, VACUUM policy, predictive optimization and exceptional retention settings.

Query performance is poor

Do not immediately run VACUUM. Investigate active file count, small files, layout, clustering, statistics, query plans and data skipping.

23. Architect decision matrix

Requirement	Primary capability
Remove eligible stale files	VACUUM
Compact active files	OPTIMIZE
Adaptive data layout	Liquid clustering
Read old table state	Time travel
Make old state current	RESTORE
Process row changes	CDF
Long-term DR	Independent backup/replication
Regulatory evidence	Governance + audit architecture

24. Interview questions

Q: What does VACUUM do?

It removes eligible unreferenced data files from physical table storage.

Q: Does VACUUM delete rows from the current Delta table?

No. It removes stale physical files that are no longer required by retained table snapshots.

Q: Why can aggressive VACUUM break time travel?

Because historical snapshots can depend on physical files that become permanently unavailable after cleanup.

Q: How do you choose a retention period?

Start from recovery, reader, streaming, compliance and audit requirements, then balance those requirements against storage cost and independent DR.

25. Architect mental model

Remember this chain:

Transaction
   ↓
new snapshot
   ↓
old files become unreferenced
   ↓
retention window
   ↓
VACUUM
   ↓
physical deletion

Delta history tells you what happened; retained files determine what historical state can still be read.

1. The problem VACUUM solves​

2. Logical deletion vs physical deletion​

Logical deletion​

Physical deletion​

3. Why Delta cannot delete files immediately​

4. What VACUUM removes​

5. The key retention property​

6. Log retention and deleted-file retention are different​

7. VACUUM does not rewrite the current table​

8. Basic syntax​

9. Time travel boundary​

10. VACUUM and RESTORE​

11. VACUUM is not disaster recovery​

12. Long-running readers and streaming​

13. VACUUM after MERGE / UPDATE / DELETE​

14. Small files vs stale files​

15. VACUUM vs OPTIMIZE​

16. Predictive optimization changes the operating model​

17. Storage-cost model​

18. SAP/BW perspective​

19. Production safety checklist​

20. Hands-on Lab — observe stale files​

21. Hands-on Lab — retention design​

22. Troubleshooting workflow​

Historical read fails​

Storage keeps growing​

Query performance is poor​

23. Architect decision matrix​

24. Interview questions​

25. Architect mental model​

Official references​