Skip to main content

06.1 Why Delta Lake?

A data lake is excellent at storing huge amounts of inexpensive data. The hard problem is making that data behave like a reliable analytical data platform.

You need answers to questions such as:

  • What exactly is the current state of this table?
  • Can two writers modify it safely?
  • What happens if a job fails halfway through a write?
  • Can I update or delete individual records?
  • Can I reproduce yesterday's result?
  • Can batch and streaming use the same table?
  • How do I detect an incompatible schema?

Delta Lake addresses these problems by adding a transaction log and table protocol on top of cloud object storage and Parquet data files. Databricks describes Delta Lake as the optimized storage layer underlying the lakehouse; Delta is also the default table format in Databricks. Databricks — What is Delta Lake?

Delta Lake foundation architecture

1. What Delta Lake actually is​

Think of the lakehouse storage stack as:

SQL / PySpark / BI ↓ Spark / SQL ↓ Delta Lake table protocol + transaction log + rules ↓ Parquet data files ↓ S3 / ADLS / GCS / cloud storage

Delta Lake is not a replacement for cloud object storage and it is not simply another compute engine.

It is a storage-layer protocol and table format that coordinates data files and metadata so distributed reads and writes can have reliable table semantics.

The open-source Delta Lake project describes the core capabilities as ACID transactions, scalable metadata handling, streaming/batch unification, schema enforcement, time travel, and upserts/deletes. Delta Lake documentation

2. The problem with a plain Parquet data lake​

Parquet is an excellent columnar file format.

Imagine a directory:

s3://company-data/sales/ ├── part-0001.parquet ├── part-0002.parquet ├── part-0003.parquet └── ...

A query can read those files. But the directory itself does not automatically give you a database-style transaction protocol.

Failure scenario​

Suppose a job needs to replace 100 files:

Job starts ↓ writes file 1 ↓ writes file 2 ↓ writes file 3 ↓ job crashes

Without a transaction protocol, a reader needs some mechanism to determine which files belong to the valid table state. Delta introduces that coordination layer.

3. Delta = Parquet + transaction log + table protocol​

The most useful beginner mental model is:

DELTA TABLE ┌────────────────────────────┐ │ Transaction Log │ │ _delta_log/ │ ├────────────────────────────┤ │ Parquet files │ │ part-0000.parquet │ │ part-0001.parquet │ │ part-0002.parquet │ └────────────────────────────┘

The Parquet files contain the actual rows. The Delta transaction log records the committed table state and changes.

Do not manually edit or delete Delta transaction-log files. Databricks explicitly recommends avoiding direct manipulation of data and transaction-log files in Delta table directories. Databricks Delta documentation

4. Why the transaction log matters​

Suppose version 0 contains A.parquet and B.parquet. A write creates C.parquet and D.parquet. The transaction commits a new table version describing which files are active.

Conceptually:

Version 0 ├── A └── B ↓ transaction Version 1 ├── B ├── C └── D

The physical files do not have to be modified in place. Instead, Delta creates a new snapshot of the table state.

This is one of the key differences between file storage and a transactional table format.

5. ACID transactions​

Delta Lake provides ACID guarantees.

Atomicity​

A transaction either commits successfully or its changes do not become part of the table state.

Consistency​

Committed operations must satisfy the table's transactional and schema rules.

Isolation​

Concurrent readers and writers operate against consistent table snapshots, subject to Delta's concurrency semantics.

Durability​

Committed state is persisted in cloud object storage.

Databricks documents snapshot isolation for reads and write-serializable isolation by default for writes, with optimistic concurrency control for concurrent modifications. ACID guarantees on Databricks

6. Optimistic concurrency control​

A simplified write lifecycle is:

  1. Read current snapshot ↓
  2. Determine changes ↓
  3. Write new Parquet files ↓
  4. Validate conflicts ↓
  5. Commit new Delta version

If another writer has changed conflicting data between the initial read and commit, Delta can reject the transaction with a concurrency conflict rather than silently corrupting the table.

This is a critical architectural distinction: concurrency control is part of the table protocol, not something you should implement manually in every pipeline. Delta Lake concurrency control

7. What happens during a failed write?​

Suppose the current version is 42. A job writes three new Parquet files and fails before commit.

Those files are not automatically part of the committed table state. The transaction log determines the valid table snapshot. Later, retention/cleanup processes can remove untracked files when appropriate.

This gives you an important operational property: a partially completed physical write does not automatically become a partially committed logical table.

8. Why this is better than just Parquet​

CapabilityPlain Parquet filesDelta Lake
Columnar storageYesYes
Cloud object storageYesYes
ACID transactionsNo table protocolYes
Concurrent write coordinationLimitedYes
Schema enforcementExternal logicBuilt into table protocol
Time travelNo native table versioningYes
UPDATE / DELETENot native table operationsYes
MERGE / upsertExternal implementationYes
Batch + streaming tableExtra designNative integration
Transaction historyNoYes
Incremental change processingExternal logicSupported capabilities

Delta does not make Parquet obsolete. It makes Parquet-backed lake storage behave much more like a reliable table.

9. Why UPDATE and DELETE matter​

Traditional data lakes often treat files as immutable objects, making record-level changes awkward.

Example SQL:

UPDATE customers SET status = 'ACTIVE' WHERE customer_id = 1001;

Delta provides table-level commands for updating and deleting data while maintaining transaction semantics.

This matters for CDC, corrections, GDPR/data deletion workflows, SCD processing, late-arriving data, and operational reconciliation.

10. MERGE changes the lakehouse architecture​

One of the most important Delta operations is MERGE.

Conceptually:

Incoming CDC ↓ Match existing business key ↓ ┌───────────────┬────────────────┐ │ Match │ No match │ │ UPDATE/DELETE │ INSERT │ └───────────────┴────────────────┘ ↓ Transactional Delta commit

Example:

MERGE INTO target t USING source s ON t.customer_id = s.customer_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *;

We will study MERGE deeply later in this module.

11. Schema enforcement​

A data lake can easily become a dumping ground where every producer writes a slightly different shape.

Delta validates schema on write. For example, if the table expects customer_id STRING, amount DECIMAL(18,2), and order_date DATE, a producer attempting to write amount as STRING should not silently corrupt the table contract.

This is one of the most important differences between:

Data lake: Here are some files. Lakehouse table: Here is a governed, versioned table with a contract.

Schema evolution is a separate decision. Enforcement and evolution are not the same thing.

12. Time travel​

Every committed Delta write creates a new table version.

Version 0 → Version 1 → Version 2 → Version 3 ↑ current

Example:

SELECT * FROM sales VERSION AS OF 5;

Or with a timestamp:

SELECT * FROM sales TIMESTAMP AS OF '2026-11-01 10:00:00';

Time travel is useful for auditing, reproducing reports, debugging, recovering from logical mistakes, comparing historical states, and reproducible analytics/ML.

Delta's time-travel capability is powered by the transaction log and historical data files that remain available under retention rules. Delta Lake time travel

13. Time travel is not infinite history​

Two retention concepts matter:

Transaction log retention + Deleted-file retention ↓ Available historical snapshots

The default behavior commonly used by Delta includes a 7-day deleted-file retention and 30-day log retention, but production retention should be deliberately designed for the business requirement. Delta Lake batch documentation

If historical files have been physically removed, a logical table version may no longer be queryable.

Therefore: time travel is a retention-governed capability, not a permanent backup system. Do not design disaster recovery around time travel alone.

14. Batch and streaming use the same table​

Delta is designed to unify batch and streaming workloads.

Delta table /
Batch Streaming ↓ ↓ ETL ingest \ / same table

A Delta table can act as a streaming source and sink while also being queried by batch and interactive workloads. Delta Lake documentation

15. Exactly-once processing — understand the boundary​

Delta's transactional sink semantics are designed to support exactly-once processing with Structured Streaming.

But do not reduce this to “Delta makes every entire pipeline exactly once.” Exactly-once behavior depends on the source, checkpointing, transformation semantics, sink, and failure model.

Architects must reason about the whole pipeline, not just the storage format.

16. Delta and Unity Catalog​

Delta Lake and Unity Catalog solve different problems.

Lakehouse table │ ┌────┴────┐ ↓ ↓ Delta Unity Catalog │ │ table governance state access history lineage

Delta Lake answers​

How does the table behave?

Unity Catalog answers​

Who can access it, how is it governed, and how is it discovered?

You need both for enterprise architecture.

17. Delta and object storage​

A typical AWS architecture is:

Databricks │ Spark / SQL compute ↓ Delta table ↓ S3 /
Parquet _delta_log

The data remains in cloud object storage. Compute can scale independently from storage.

18. Why Delta scales better than directory-based metadata​

A massive data lake can contain millions or billions of files. Relying only on object-store directory listing and file inspection becomes increasingly expensive.

Delta maintains table metadata through its transaction log and checkpoints, allowing table state to be reconstructed without treating the entire storage directory as the database catalog.

This is why Delta is more than Parquet with a folder named _delta_log. It is a protocol for table state.

19. The transaction log — first look​

A Delta table contains a transaction-log directory:

_delta_log/ ├── 00000000000000000000.json ├── 00000000000000000001.json ├── 00000000000000000002.json ├── ... └── checkpoint...

Each committed version represents a table-state transition.

Version 0: add A, add B Version 1: add C, remove A Version 2: add D, remove B

The current snapshot is derived from the committed actions.

We will study the transaction log and checkpoint internals in 06.2.

20. What Delta does not solve automatically​

Delta is powerful, but it is not magic. It does not automatically solve bad business keys, incorrect data models, poor partition/layout decisions, duplicate source records, bad query design, expensive joins, uncontrolled schema evolution, weak access governance, poor retention strategy, bad streaming checkpoint design, or cloud-cost problems.

Think of Delta as a reliable foundation, not a substitute for architecture.

21. SAP/BW mental model​

For someone coming from SAP BW/4HANA, use this analogy carefully:

BW/4HANA conceptDelta/Lakehouse analogue
Persistent storage objectDelta table
Data load/changeDelta transaction
Request/version historyDelta table versions
Change/update logicMERGE / UPDATE / DELETE
Data model contractSchema + constraints
Process chainJobs / pipelines
Data governanceUnity Catalog
Physical persistenceCloud object storage

The analogy is useful for orientation, but Delta is not BW/4HANA and should not be modeled as a direct one-to-one replacement.

22. Hands-on lab — create your first Delta table​

Step 1 — Create data​

data = [(1001, 'A', 100.00), (1002, 'B', 250.00), (1003, 'C', 175.00)] df = spark.createDataFrame(data, ['customer_id', 'segment', 'amount'])

Step 2 — Write Delta​

df.write.format('delta').mode('overwrite').saveAsTable('demo.sales_delta')

Step 3 — Query it​

SELECT * FROM demo.sales_delta;

Step 4 — Inspect history​

DESCRIBE HISTORY demo.sales_delta;

Step 5 — Update a row​

UPDATE demo.sales_delta SET amount = 300 WHERE customer_id = 1002;

Step 6 — Inspect history again​

DESCRIBE HISTORY demo.sales_delta;

Step 7 — Query an older version​

SELECT * FROM demo.sales_delta VERSION AS OF 0;

What you should observe:

Initial write → Version 0 UPDATE → Version 1 Historical query → Version 0 snapshot

23. Lab — compare Parquet and Delta conceptually​

Create the same dataset as Parquet and Delta, then compare directory structure, transaction metadata, ability to update, table history, time travel, and concurrent-write semantics.

The point is not to prove that Delta always reads faster. The point is to understand the reliability contract Delta adds around data files.

24. Production architecture pattern​

Sources │ ├── SAP ├── APIs ├── Files ├── CDC └── Streaming ↓ Bronze Delta ↓ Silver Delta ↓ Gold Delta ├── BI ├── ML ├── APIs └── downstream systems

Delta provides the reliable table foundation across these layers. Unity Catalog provides governance around those tables.

25. Architect decision framework​

When someone asks, “Why are we using Delta instead of plain Parquet?” a strong enterprise answer includes:

  1. Transactional table state
  2. Concurrent-write protection
  3. Schema enforcement
  4. Version history
  5. UPDATE / DELETE / MERGE
  6. Batch + streaming unification
  7. Reliable incremental processing
  8. Scalable metadata handling
  9. Open table protocol
  10. Tight Spark/Databricks integration

Then ask whether the workload actually requires those capabilities.

26. Common mistakes​

Mistake 1 — Calling Delta just Parquet​

Delta uses Parquet data files, but the transaction log and table protocol are essential to its behavior.

Mistake 2 — Treating time travel as backup​

Retention can remove the files required for historical versions.

Mistake 3 — Editing _delta_log manually​

Do not manipulate Delta internals directly.

Mistake 4 — Assuming ACID means no conflicts​

Concurrent writes can still fail with conflict exceptions. ACID prevents corruption; it does not guarantee every concurrent transaction will succeed.

Mistake 5 — Confusing Delta with Unity Catalog​

Delta governs table reliability. Unity Catalog governs access, discovery, and governance.

Mistake 6 — Assuming schema evolution is always good​

Uncontrolled evolution can silently damage downstream contracts.

27. Interview questions​

Foundation​

  1. What is Delta Lake?
  2. Why does Delta use Parquet?
  3. What is the _delta_log?
  4. What problem does the transaction log solve?
  5. What are ACID guarantees?
  6. What is time travel?

Engineering​

  1. How does Delta handle concurrent writes?
  2. What happens if a write fails before commit?
  3. How does MERGE work conceptually?
  4. How does schema enforcement differ from schema evolution?
  5. Why can Delta support both batch and streaming?
  6. What determines how far back time travel works?

Architect​

  1. Why would you choose Delta over a plain Parquet lake?
  2. How would you design Delta retention for regulated data?
  3. How do Delta and Unity Catalog complement each other?
  4. How would you design Bronze/Silver/Gold using Delta?
  5. How would you diagnose a concurrent write conflict?
  6. Why should Delta not be treated as a backup mechanism?

28. Production checklist​

  • Delta is the intentional table format.
  • Table ownership is defined.
  • Schema contract is defined.
  • Schema evolution policy is explicit.
  • MERGE/update/delete patterns are reviewed.
  • Concurrent writers are understood.
  • Retention requirements are documented.
  • Time-travel expectations are realistic.
  • VACUUM/retention policy is governed.
  • Unity Catalog governance is configured.
  • Storage permissions are controlled.
  • Data quality is validated before commit.
  • Batch/streaming interactions are understood.
  • Direct manipulation of Delta internals is prohibited.

29. Key takeaway​

Delta Lake turns cloud-object-storage files into a reliable, versioned, transactional table foundation for the lakehouse.

The mental model to retain is:

Cloud storage + Parquet → Delta transaction protocol → reliable table → governed lakehouse

In the next topic: 06.2 — Delta Lake Transaction Log & Internals, where we will dissect commits, JSON actions, checkpoints, snapshots, add/remove actions, table versions, and how Delta reconstructs the current table state.

Official references​