Skip to main content

06.2 Delta Lake Transaction Log & Internals

If 06.1 answered why Delta Lake exists, this topic answers the more important architect question: How does Delta know what the table actually is?

The answer is the Delta transaction log.

Databricks describes Delta Lake as Parquet extended with a file-based transaction log for ACID transactions and scalable metadata handling. Each write creates a new table version, and the transaction log is the authoritative record used to reconstruct table state. Databricks — What is Delta Lake?

Delta transaction log architecture

1. The core mental model​

Do not think:

Delta table = folder full of Parquet files

Think:

Delta table = data files + transaction log + table protocol + snapshot

The Parquet files contain physical data. The transaction log describes the logical table state. The snapshot is the table state reconstructed from committed log actions.

2. Physical layout​

A simplified Delta table looks like:

sales/ ├── part-00000-....parquet ├── part-00001-....parquet ├── part-00002-....parquet └── _delta_log/ ├── 00000000000000000000.json ├── 00000000000000000001.json ├── 00000000000000000002.json └── 00000000000000000010.checkpoint.parquet

The exact physical layout can vary with Delta features and runtime versions. Never treat the directory listing as the table definition. The log is what tells Delta which data files are currently active.

Databricks explicitly recommends avoiding direct manipulation of Delta data and transaction-log files. Databricks Delta documentation

3. Every commit creates a table version​

First write creates Version 0. Second write creates Version 1, then Version 2, and so on.

v0 → v1 → v2 → v3 → v4 ↑ current

A Delta reader can ask what the table was at version 2. This is the basis of time travel.

4. The transaction log is an action log​

A Delta commit is composed of actions. Important action types include:

  • add
  • remove
  • metaData
  • protocol
  • commitInfo
  • txn

The exact set of actions evolves as Delta features evolve. For architecture purposes, remember that a commit records what data files became active, what became inactive, table metadata, protocol/features, and operational context.

The open Delta protocol defines the transaction-log structure so compatible engines can understand table state. Delta transaction log resources

5. The add action​

An add action says: This data file is now part of the table snapshot.

Conceptually:

add ├── file path ├── size ├── partition information └── additional file statistics/metadata

Real Delta metadata contains additional fields depending on table features. The key idea is that the log records the file as part of the logical table.

6. The remove action​

A remove action says: This previously active file should no longer be part of the current table snapshot.

Conceptually:

Version 5: A B C Version 6: remove B, add D

The physical file B may still exist in storage. It is simply no longer active in the latest logical snapshot.

The fundamental distinction is:

Physical existence ≠ logical table membership

7. Logical deletion vs physical deletion​

If a file is removed from the current snapshot, it may remain physically present for some period because historical versions may still need it.

Eventually, retention and VACUUM can remove obsolete files.

DELETE / UPDATE / MERGE ↓ logical table change ↓ obsolete files ↓ retention window ↓ VACUUM ↓ physical cleanup

Databricks documents separate retention controls for log history and deleted data files. Delta Lake batch documentation

8. The metaData action​

A Delta table needs to know things such as schema, partition columns, and table configuration. The metaData action represents table metadata in the transaction log.

metaData ├── schema ├── partition columns ├── configuration └── table metadata

This is why the table schema is not merely inferred from whichever Parquet file happens to be inspected.

9. The protocol action​

The Delta protocol defines compatibility requirements:

protocol ├── minimum reader capability └── minimum writer capability

Newer Delta features can require newer reader/writer capabilities. The Delta table-properties reference exposes properties such as delta.minReaderVersion, delta.minWriterVersion, and checkpoint policy. Delta table properties

Architect implication: a table is not just Parquet files that Spark can read. It is data + metadata + protocol contract.

10. The commitInfo action​

Commit information provides operational context around a write. It can capture information such as operation, parameters, user/job context, timestamps, and metrics.

For production troubleshooting, DESCRIBE HISTORY is the supported way to inspect table history rather than manually parsing transaction-log files.

11. The txn action​

Transaction identifiers can support idempotent processing patterns, particularly in streaming and foreachBatch designs.

application ID + transaction version ↓ unique transaction ↓ duplicate commit can be detected

Databricks documents transaction identifiers and retention behavior for idempotent writes. Delta table properties

12. One commit can contain multiple actions​

A transaction can change several things at once:

Version 27 ├── remove old_file_1 ├── remove old_file_2 ├── add new_file_1 ├── add new_file_2 └── commitInfo

The table moves from one consistent snapshot to another. Databricks documents that each successful transaction appears as a single entry in the Delta log while individual operations are represented as JSON metadata within the transaction entry. Databricks transactions

13. How an UPDATE can become file changes​

Suppose a table contains A.parquet, B.parquet and C.parquet, and an UPDATE changes rows located in B.

A simplified physical outcome can look like:

Before: A B C UPDATE B ↓ After commit: A C B-new

The log records the logical transition: remove B, add B-new.

This is why Delta can provide table-level UPDATE semantics even though Parquet files themselves are immutable objects.

14. MERGE can create complex commit actions​

A MERGE can simultaneously insert new records, update matched records, and delete matched records.

At the physical level, that can produce multiple file additions/removals. The transaction log commits the resulting state atomically.

Therefore: MERGE is a table operation; Parquet is a file format.

15. From log entries to a Snapshot​

A snapshot represents the table state at a particular version.

v0 actions + v1 actions + v2 actions + ... + vN actions ↓ Snapshot(vN)

The snapshot answers: which files are active, what is the schema, what table properties apply, and what protocol is required.

Delta's APIs describe a Snapshot as the state of a table at a specific version. Delta API reference

16. Why checkpoints are necessary​

Imagine a table with millions of commits. A reader should not need to replay every transaction from version 0.

Delta periodically creates checkpoints.

v0 v1 v2 ... v999 ↓ checkpoint @ 1000 ↓ v1001 v1002 ... v1099 ↓ checkpoint @ 1100

A checkpoint stores a compact representation of accumulated table state.

Databricks states that checkpoints speed up query planning by allowing Delta to reconstruct table state without replaying the full transaction log. Checkpoint V2

17. Checkpoint architecture​

To reconstruct version 149, conceptually:

Checkpoint 100 + v101 v102 ... v149 ↓ Snapshot 149

Instead of replaying v0 through v149, the reader starts from the checkpoint and replays only later commits.

18. Classic checkpoints vs Checkpoint V2​

Modern Delta implementations also support Checkpoint V2. Databricks documents V2 as a checkpoint format that can support more concurrent writers and reduce write conflicts on large or frequently updated Delta tables. Checkpoint V2

The architect-level point is: checkpoints are an acceleration mechanism for reconstructing table state; they are not a replacement for the transaction protocol.

19. Checkpoint vs transaction log​

ComponentPurpose
JSON commit filesIncremental table-state changes
CheckpointMaterialized accumulated state
Parquet data filesActual table data
Protocol metadataReader/writer compatibility
SnapshotLogical state used by readers

Think:

JSON log = changes Checkpoint = materialized state Snapshot = logical table Parquet = physical data

20. How a reader finds the current state​

A simplified reader algorithm is:

  1. Find latest table version ↓
  2. Find latest usable checkpoint ↓
  3. Load checkpoint state ↓
  4. Replay commits after checkpoint ↓
  5. Apply add/remove actions ↓
  6. Produce Snapshot ↓
  7. Read active Parquet files

This is the most important execution model in this topic.

21. Example: reconstructing a table​

Version 0: add A, add B → current A B

Version 1: add C → current A B C

Version 2: remove A, add D → current B C D

Version 3: remove C → current B D

The physical files may include A, B, C and D. The logical table only includes B and D.

The central Delta concept is: the active-file set is derived from the transaction log.

22. Why directory listing is not enough​

Suppose storage contains A.parquet, B.parquet, C.parquet, D.parquet and E.parquet.

The transaction log may say:

A → removed B → active C → removed D → active E → active

Therefore:

Physical files: A B C D E Logical table: B D E

This is why directly adding or deleting files underneath a Delta table is dangerous. Databricks explicitly warns against manually modifying Delta data files. Delta best practices

23. Table history vs transaction-log internals​

Use supported interfaces for normal operations:

DESCRIBE HISTORY sales;

Do not build production applications that parse internal JSON commit files unless you have a specific protocol-level engineering requirement.

Rule:

Business operations → SQL / Spark / supported APIs Protocol engineering → Delta transaction-log specification

24. Retention affects historical reconstruction​

A historical version is usable only while the required transaction-log entries and physical data files remain available.

Relevant concepts include delta.logRetentionDuration, delta.deletedFileRetentionDuration, VACUUM, and checkpoints.

The open-source Delta documentation describes default 30-day log retention and 7-day deleted-file retention, with configuration available for different requirements. Delta batch documentation

Therefore: historical accessibility is a function of both logical log retention and physical file retention.

25. Time travel through the internals​

Suppose current version is 100 and you request version 80.

checkpoint before 80 + commits through version 80 ↓ Snapshot(80) ↓ active files at v80 ↓ Parquet reads

This is why time travel is snapshot reconstruction, not simply opening an old folder.

26. Delta transaction log and streaming​

A Delta table can be a streaming source and sink.

As a source:

Delta versions → new commits → streaming micro-batches

As a sink:

streaming batch → transaction → Delta commit → new table version

Databricks documents the Delta transaction log as providing exactly-once semantics for Delta streaming sinks under the supported processing model. Delta streaming

27. Delta Change Data Feed relationship​

Change Data Feed tracks row-level changes between Delta table versions.

Version 10 → Version 11 → CDF → insert / update / delete changes

CDF is not the same thing as directly reading the transaction log. Use CDF when an application needs supported row-level change consumption; use the transaction log as the table-state protocol.

28. Advanced architecture: transaction protocol layers​

Business table semantics ↓ Snapshot / current table state ↓ Checkpoints + commit replay ↓ Delta transaction protocol ↓ Parquet data files ↓ Cloud object storage

Each layer solves a different problem.

29. Architect performance implications​

The transaction log is metadata. Metadata can become a performance bottleneck when tables have enormous numbers of files, extremely frequent commits, unnecessary history, inefficient checkpoints, or workloads that create excessive small files.

Therefore Delta architecture is not only about ACID. It is also about:

File layout + Commit frequency + Metadata scale + Checkpoint behavior + Retention

This connects directly to later topics on OPTIMIZE, data skipping, liquid clustering and file-size engineering.

30. Production anti-patterns​

Anti-pattern 1 — Manually deleting Parquet files​

This can make the transaction log reference files that no longer exist.

Anti-pattern 2 — Manually editing _delta_log​

This can corrupt table state.

Anti-pattern 3 — Treating every commit as free​

Very high commit frequency increases metadata activity.

Anti-pattern 4 — Excessive tiny writes​

Small files increase metadata and query overhead.

Anti-pattern 5 — Using raw log parsing for business CDC​

Use Change Data Feed or supported APIs where appropriate.

Anti-pattern 6 — Ignoring retention​

A streaming consumer can fall behind beyond available retention and lose access to required historical data. Databricks streaming guidance

31. Hands-on Lab — Observe versions​

Create a table:

CREATE TABLE demo.delta_internals ( id INT, amount DECIMAL(18,2) ) USING DELTA;

Insert data:

INSERT INTO demo.delta_internals VALUES (1, 100.00), (2, 200.00);

Then:

DESCRIBE HISTORY demo.delta_internals;

Insert again, inspect history again, and identify table version, operation, timestamp, operation parameters and metrics where available.

32. Hands-on Lab — Observe logical deletion​

Run:

DELETE FROM demo.delta_internals WHERE id = 2;

Then inspect history and reason about:

id=2 ↓ logical removal from current snapshot ↓ old physical data may remain ↓ historical version may still expose it ↓ retention/VACUUM eventually removes obsolete files

33. Hands-on Lab — Time travel​

Query an earlier version and compare it with the current version:

SELECT * FROM demo.delta_internals VERSION AS OF 0;

Your objective is to understand that each result corresponds to a different snapshot, not a different physical database.

34. Hands-on Lab — Predict the snapshot​

Before executing, predict the active files.

Initial: A B

Operation: replace B with C

Expected logical state: A C

Expected log concept: remove B + add C

The objective is to develop transaction-log thinking, not just SQL syntax.

35. Production checklist​

  • Treat the Delta log as the authoritative table-state protocol.
  • Never manually modify _delta_log.
  • Never manually delete active Parquet files.
  • Understand add/remove actions.
  • Understand logical vs physical deletion.
  • Understand table versions.
  • Understand snapshots.
  • Understand checkpoints.
  • Monitor excessive commit frequency.
  • Control small-file creation.
  • Design retention deliberately.
  • Use supported history interfaces.
  • Use CDF rather than raw-log parsing for supported row-level CDC use cases.
  • Validate streaming retention against consumer lag.
  • Understand protocol compatibility before enabling advanced features.

36. Interview questions​

Foundation​

  1. What is the Delta transaction log?
  2. What is an add action?
  3. What is a remove action?
  4. What is a snapshot?
  5. What is a checkpoint?
  6. Why does Delta need checkpoints?

Engineering​

  1. How does Delta reconstruct the current table?
  2. Why can't you determine table state by listing Parquet files?
  3. What happens physically after a DELETE?
  4. What is the difference between logical and physical deletion?
  5. What does metaData represent?
  6. What is the purpose of the protocol action?
  7. What is commitInfo?
  8. Why are transaction identifiers useful?

Architect​

  1. How would you explain Delta internals to a platform team?
  2. Why does frequent small-file creation affect metadata performance?
  3. How do checkpoints improve table-read performance?
  4. What happens if required log entries are removed?
  5. Why should business applications avoid parsing raw _delta_log files?
  6. How do Delta transaction logs relate to Change Data Feed?
  7. How would you troubleshoot a missing-file error in a Delta table?

37. Architect mental model​

Memorize this:

DELTA TABLE │ ┌────┴────┐ ↓ ↓ Transaction Data Files Log │ ┌─┼────┐ ↓ ↓ ↓ ↓ Parquet Add Remove Metadata └─┴────┘ ↓ Snapshot ↓ Current table

And for scale:

Commits ↓ Checkpoints ↓ Replay only recent commits ↓ Snapshot ↓ Read active files

That is the Delta Lake execution model to carry into every advanced topic.

38. Key takeaway​

Delta Lake does not determine table state by looking at files alone. It reconstructs a versioned logical table from a transactional history of actions, accelerated by checkpoints.

The complete mental model is:

Parquet stores the data → transaction-log actions describe changes → checkpoints accelerate reconstruction → snapshot represents table state → readers consume the active files.

Next: 06.3 — Delta Lake ACID Transactions, Concurrency & Isolation.

Official references​