08.12 Lakeflow Declarative Pipelines
1. Why Lakeflow Declarative Pipelines matter
Lakeflow Declarative Pipelines are Databricks' declarative framework for building batch and streaming data pipelines with SQL and Python.
The architectural idea is simple:
Declare the datasets and transformations you want. Let the pipeline engine infer dependencies and manage execution.
A traditional procedural pipeline often says:
RUN STEP A
WAIT
RUN STEP B
WAIT
RUN STEP C
A declarative pipeline says:
TABLE A = SOURCE + TRANSFORMATION
TABLE B = TRANSFORMATION OF A
TABLE C = AGGREGATION OF B
The engine can then construct the dependency graph, determine execution order and perform incremental processing where supported.
Lakeflow pipelines are built on Apache Spark Declarative Pipelines and add Databricks production capabilities such as data-quality expectations, AUTO CDC and a queryable event log.
2. The architectural problem they solve
A mature data platform has to solve more than transformation logic.
It must manage:
- dependencies;
- incremental processing;
- streaming;
- batch refresh;
- data quality;
- CDC;
- orchestration;
- observability;
- retries;
- schema changes;
- security;
- cost;
- deployment;
- operational ownership.
If every team implements these concerns independently, the organization accumulates incompatible pipeline patterns.
Lakeflow provides a common programming and operating model.
The high-level architecture is:
DATA SOURCES
|
+-------------+-------------+
| | |
Files CDC Events
| | |
+-------------+-------------+
|
v
LAKEFLOW PIPELINE
|
+-----------+-----------+
| |
STREAMING TABLES MATERIALIZED VIEWS
| |
+-----------+-----------+
|
QUALITY RULES
|
v
SILVER / GOLD
|
+-----------+-----------+
| | |
BI ML APPS
3. Declarative vs procedural processing
This distinction is fundamental.
Procedural
You explicitly control the sequence.
extract()
transform()
load()
validate()
publish()
The developer owns the orchestration logic.
Declarative
You describe the desired data state.
silver_orders =
clean(bronze_orders)
gold_revenue =
aggregate(silver_orders)
The engine derives the dependency graph.
Why this matters
Declarative processing reduces the amount of orchestration code that engineers have to maintain.
Instead of writing:
if A completed:
start B
if B completed:
start C
if C failed:
retry B
the dependency graph is represented by the data definitions themselves.
This is one of the major architectural advantages of Lakeflow pipelines.
4. Pipeline architecture
A Lakeflow pipeline is the primary unit of development and execution.
It contains:
- source code;
- dataset definitions;
- flows;
- pipeline configuration;
- execution settings;
- data-quality rules;
- operational metadata.
A pipeline can contain SQL and Python source files.
The important rule is:
A single pipeline can mix SQL and Python, but each source file uses one language.
This allows an enterprise team to use SQL where SQL is the clearest expression and Python where programmatic logic is required.
5. Pipeline dependency graph
Consider:
raw_orders
|
v
bronze_orders
|
+------> gold_customer_sales
|
+------> gold_product_sales
|
+------> gold_daily_revenue
The engine can infer the dependencies from dataset references.
The important architectural principle is:
Do not create artificial orchestration dependencies when the data dependency already expresses the relationship.
6. Dataset types
The most important dataset choices are:
| Dataset type | Primary use |
|---|---|
| Streaming table | Incremental and streaming data |
| Materialized view | Derived state, joins and aggregations |
| Temporary view | Intermediate logic without persisted output |
The choice affects:
- correctness;
- latency;
- compute;
- storage;
- refresh behavior;
- operational complexity.
Databricks currently recommends streaming tables for ingestion and low-latency incremental workloads, materialized views for complex transformations and analytical queries, and temporary views for intermediate logic that does not need persisted storage.
The correct question is:
What execution and correctness semantics does this dataset require?
7. Streaming tables
A streaming table is designed for incremental processing.
Typical workloads include:
- append-heavy ingestion;
- cloud object storage;
- event streams;
- incremental transformations;
- CDC targets.
Conceptually:
NEW DATA
|
v
STREAMING TABLE
|
v
PROCESS NEW / RELEVANT INPUT
Good fit:
- the workload is naturally incremental;
- latency matters;
- new records arrive continuously or in frequent batches;
- the table represents an incrementally maintained data state.
Architect caution
Streaming does not automatically mean faster or cheaper.
A streaming architecture can become expensive when:
- data is extremely small;
- latency is unnecessary;
- state grows excessively;
- the pipeline is kept continuously running without a business requirement.
8. Materialized views
A materialized view represents derived data that is maintained from its defining query.
Example:
``sql CREATE MATERIALIZED VIEW gold_daily_sales AS SELECT order_date, region, SUM(net_amount) AS revenue, COUNT(*) AS orders FROM silver_orders GROUP BY order_date, region;
This is particularly useful for:
- aggregations;
- joins;
- analytical serving;
- dashboard acceleration;
- derived business datasets.
A materialized view is not merely a cache. It represents a managed data product whose contents are derived from its definition.
---
## 9. Temporary views
Temporary views are useful for intermediate logic.
Example:
``sql
CREATE TEMPORARY VIEW prepared_orders AS
SELECT *
FROM bronze_orders
WHERE order_id IS NOT NULL;
Use them when:
- the intermediate result has no independent consumer;
- persisting another table would add unnecessary storage and maintenance;
- the view improves readability.
Do not materialize every intermediate transformation.
10. Streaming table vs materialized view
| Requirement | Preferred pattern |
|---|---|
| Append-heavy ingestion | Streaming table |
| Low-latency incremental processing | Streaming table |
| CDC target | Streaming table + AUTO CDC |
| Large aggregation | Materialized view |
| Complex join | Materialized view |
| Dashboard acceleration | Materialized view |
| Intermediate logic only | Temporary view |
The deeper decision is:
``text Incremental row-oriented state vs Derived analytical state
Choose according to the semantics of the data product.
---
## 11. SQL authoring
Lakeflow SQL uses declarative statements.
Typical constructs include:
``sql
CREATE STREAMING TABLE ...
CREATE MATERIALIZED VIEW ...
CREATE TEMPORARY VIEW ...
CREATE FLOW ...
AUTO CDC INTO ...
A simple pipeline might look like:
``sql CREATE STREAMING TABLE bronze_orders AS SELECT * FROM STREAM read_files('/Volumes/raw/orders');
CREATE MATERIALIZED VIEW silver_orders AS SELECT order_id, customer_id, CAST(order_date AS DATE) AS order_date, CAST(amount AS DECIMAL(18,2)) AS amount FROM STREAM(bronze_orders) WHERE order_id IS NOT NULL;
The important point is that the SQL expresses the desired dataset, while the platform manages the pipeline graph.
---
## 12. Python authoring
The modern Python interface uses the pipelines module.
``python
from pyspark import pipelines as dp
@dp.table
def bronze_orders():
return spark.readStream.table("raw.orders")
A materialized view can be expressed as:
``python from pyspark import pipelines as dp
@dp.materialized_view def gold_daily_sales(): return ( spark.read.table("silver.orders") .groupBy("order_date") .sum("net_amount") )
The modern interface is based on:
``python
from pyspark import pipelines as dp
rather than starting new development with the older:
``python import dlt
The older DLT terminology remains important when reading legacy projects, but architects should understand the current Lakeflow/Spark Declarative Pipelines model.
---
## 13. When to use SQL vs Python
### Prefer SQL when
- transformations are relational;
- joins dominate;
- filters dominate;
- aggregations dominate;
- the team is SQL-heavy;
- the transformation should be easy for analysts to understand.
### Prefer Python when
- custom programmatic logic is needed;
- reusable Python libraries are required;
- many datasets need to be generated programmatically;
- specialized processing is easier to express in Python;
- complex control logic is unavoidable.
Do not choose Python because it looks more advanced. Use the language that makes the data contract easiest to understand and maintain.
---
## 14. Flows
Flows describe how data moves into a target.
Conceptually:
```text
SOURCE
|
+---- FLOW A ----+
| |
+---- FLOW B ----+----> TARGET
| |
+---- FLOW C ----+
Flows are particularly useful for:
- incremental ingestion;
- append operations;
- CDC;
- update processing;
- more complex pipeline topologies.
The architect should distinguish the dataset from the flow that populates it.
15. AUTO CDC
CDC is one of the most important enterprise features of Lakeflow pipelines.
Modern Lakeflow pipelines provide:
AUTO CDC
AUTO CDC FROM SNAPSHOT
These APIs simplify:
- SCD Type 1;
- SCD Type 2;
- ordered CDC processing;
- database replication patterns;
- snapshot-based change detection.
The current APIs replace the older APPLY CHANGES naming for new designs.
Typical architecture:
SOURCE DATABASE
|
v
CDC FEED
|
v
AUTO CDC FLOW
|
v
STREAMING TARGET
|
+------ SCD1
|
+------ SCD2
16. AUTO CDC sequencing
CDC events may arrive out of order.
Example:
customer_id = 100
sequence 101 -> address = A
sequence 103 -> address = C
sequence 102 -> address = B
The target must use an appropriate sequencing column.
Conceptually:
KEY
+
SEQUENCE
+
CHANGE OPERATION
|
v
CORRECT CURRENT STATE / HISTORY
Never assume arrival order is the same as business change order.
17. AUTO CDC and SCD Type 1
SCD Type 1 retains the current state.
Example:
Before:
Customer 100 -> Mumbai
After:
Customer 100 -> Bengaluru
The target contains the latest value and does not retain the prior attribute version.
Use SCD1 when historical attribute changes are not required by consumers.
18. AUTO CDC and SCD Type 2
SCD Type 2 preserves history.
Conceptually:
CUSTOMER 100
Mumbai
2024-01-01 -> 2026-03-10
Bengaluru
2026-03-11 -> current
SCD2 is important when reporting must answer:
What did we know at that point in time?
19. AUTO CDC FROM SNAPSHOT
Not every source provides a CDC stream.
Some systems only provide periodic snapshots.
Example:
Snapshot 1
|
Snapshot 2
|
Snapshot 3
The platform can compare ordered snapshots to infer changes.
Use:
- AUTO CDC when a reliable CDC feed exists;
- AUTO CDC FROM SNAPSHOT when only ordered snapshots are available.
Do not build a custom comparison framework unless source semantics require it.
20. Data quality expectations
Lakeflow pipelines support executable quality expectations.
Conceptually:
``sql CONSTRAINT valid_order EXPECT (order_id IS NOT NULL)
An expectation can be used to:
- warn;
- drop invalid records;
- fail the update.
Quality architecture:
```text
RAW
|
v
INGEST
|
v
QUALITY GATE
| PASS FAIL
| |
v v
TRUSTED QUARANTINE
|
v
GOLD
Quality enforcement should occur close to the point where bad data would cross into a trusted data boundary.
For a deeper treatment, see 08.11 Data Quality & Expectations.
21. Expectations are not a complete quality platform
Expectations are powerful, but they are not the entire data-quality architecture.
You still need to consider:
- dataset-level volume checks;
- distribution anomalies;
- freshness;
- reconciliation;
- referential integrity;
- business controls;
- incident management;
- ownership;
- alerting.
For example:
order_id IS NOT NULL
does not detect:
Yesterday = 1,000,000 orders
Today = 300 orders
Row-level validation and dataset-level observability solve different problems.
22. Pipeline execution modes
Triggered
The pipeline runs and stops.
Use this as the default starting point for many workloads.
Advantages:
- compute is consumed only when needed;
- easier cost control;
- simpler operations.
Continuous
The pipeline remains active to process new data with minimal latency.
Use only when the business genuinely requires continuously running processing.
The architect must quantify:
Required latency
vs
Continuous compute cost
Do not choose continuous execution merely because the source is a stream.
23. Serverless vs classic compute
Serverless pipelines reduce infrastructure management and are the preferred starting point when the workload is compatible.
Classic compute can still be appropriate when you need:
- specific instance types;
- custom cluster policies;
- specialized initialization;
- environment-specific infrastructure controls.
Prefer serverless when the requirement is simply:
Run this pipeline reliably.
Consider classic compute when the requirement is:
Run this pipeline with this specific infrastructure behavior.
24. Dependency management
One of the strongest benefits of declarative pipelines is automatic dependency resolution.
Example:
bronze_orders
|
v
silver_orders
|
+------> gold_sales
|
+------> gold_customer
The engine understands the graph from dataset references.
Pipeline design should focus on:
- clear dataset boundaries;
- correct references;
- meaningful ownership;
- appropriate dataset types.
Do not reproduce the graph manually in a giant orchestration script.
25. Orchestration vs transformation
Lakeflow pipelines are not a replacement for every orchestration requirement.
Separate:
Data transformation
SOURCE
|
v
BRONZE
|
v
SILVER
|
v
GOLD
from:
Enterprise workflow orchestration
INGESTION
|
v
QUALITY
|
v
PIPELINE
|
v
TESTS
|
v
NOTIFICATION
|
v
DOWNSTREAM JOB
Use broader workflow orchestration when the process crosses independent systems or operational tasks.
26. Observability and the event log
Production pipelines need operational evidence.
The pipeline event log can provide information about:
- updates;
- flows;
- progress;
- data-quality metrics;
- failures;
- execution state.
This enables operational questions such as:
Did the pipeline run?
Did the update succeed?
Which flow failed?
How many records failed expectations?
How long did the update take?
What changed?
A mature platform turns this information into dashboards, alerts, SLA monitoring and incident workflows.
27. Failure handling
A production pipeline should classify failures.
| Failure class | Example | Response |
|---|---|---|
| Transient | service/network issue | retry |
| Data | invalid record | quarantine or fail |
| Code | unresolved column | fix, test, deploy |
| Architecture | wrong grain or CDC model | redesign |
Retrying every failure is not an error strategy.
28. Schema evolution
Possible schema events include:
ADD COLUMN
REMOVE COLUMN
RENAME COLUMN
TYPE CHANGE
NESTED STRUCTURE CHANGE
The architect must distinguish:
APPROVED EVOLUTION
vs
UNEXPECTED DRIFT
A safe platform should define compatibility rules before accepting producer changes.
29. Performance architecture
Analyze pipeline performance at multiple levels.
Level 1 — Data volume
How much data is processed?
Level 2 — Incrementality
How much history is recomputed?
Level 3 — File layout
Are files appropriately sized and clustered?
Level 4 — Query plan
Are joins, filters and aggregations efficient?
Level 5 — State
How much state must streaming computation retain?
Level 6 — Pipeline topology
Are unnecessary transformations or materializations present?
A common mistake is optimizing Spark code before deciding whether the architecture is processing too much data.
30. Incremental processing
The largest performance improvement often comes from reducing data processed.
Compare:
FULL RECOMPUTE
10 TB every run
with:
INCREMENTAL
50 GB changed data
Before tuning code:
- check whether the dataset can be incremental;
- check whether a materialized view can maintain derived state;
- check whether CDC can eliminate repeated full comparisons;
- check whether the source can provide change information.
31. Cost architecture
Pipeline cost is driven by:
- compute duration;
- compute size;
- execution frequency;
- data volume;
- state;
- unnecessary recomputation;
- continuous execution;
- inefficient joins;
- poor pipeline topology.
A useful mental model:
COST
=
COMPUTE
×
TIME
×
FREQUENCY
×
INEFFICIENCY
Optimize all four dimensions.
32. Security and Unity Catalog
Lakeflow pipelines operate within the broader Databricks governance architecture.
Use Unity Catalog to govern:
- catalogs;
- schemas;
- tables;
- volumes;
- permissions;
- lineage;
- external locations;
- storage credentials.
The pipeline should not become a security bypass.
A production architecture should define:
IDENTITY
|
v
UNITY CATALOG
|
+--> DATA ACCESS
+--> STORAGE ACCESS
+--> LINEAGE
+--> AUDIT
Pipeline identity should follow least-privilege principles.
33. Development environments
A mature pipeline should move through environments:
DEVELOPMENT
|
v
TEST
|
v
UAT
|
v
PRODUCTION
Environment-specific values should be parameterized rather than hard-coded.
Examples:
- catalog;
- schema;
- storage location;
- source location;
- notification endpoint;
- pipeline configuration.
34. CI/CD architecture
Treat pipeline source as software.
Recommended flow:
GIT
|
v
PULL REQUEST
|
+--> LINT
|
+--> UNIT TEST
|
+--> VALIDATION
|
v
DEPLOY
|
v
TEST ENVIRONMENT
|
v
PRODUCTION
Version control provides:
- history;
- review;
- rollback;
- traceability;
- reproducibility.
35. Testing strategy
Testing should exist at multiple levels.
Unit tests
Validate transformation logic.
Data-quality tests
Validate expectations.
Integration tests
Validate source-to-target behavior.
Reconciliation tests
Validate business totals.
Example:
SOURCE REVENUE
=
GOLD REVENUE
within an approved tolerance.
Regression tests
Protect existing behavior when code changes.
A pipeline is production-ready when it has evidence, not merely when it runs successfully once.
36. Data contracts
A data contract defines what producers and consumers can expect.
Example:
DATASET: sales_order
GRAIN:
one row per order line
KEY:
(order_id, line_number)
REQUIRED:
order_id
line_number
order_date
MEASURES:
quantity
net_amount
FRESHNESS SLA:
15 minutes
OWNER:
Sales Data Product
QUALITY:
no duplicate keys
no negative quantity
valid currency
The pipeline implements the contract.
37. Lakeflow pipeline design for a complete medallion architecture
A practical architecture:
SOURCE SYSTEMS
|
+------------+------------+
| | |
FILES CDC EVENTS
| | |
+------------+------------+
|
v
BRONZE STREAMING
|
v
QUALITY GATE
/ PASS FAIL
| |
v v
SILVER QUARANTINE
|
+-----+------+
| |
v v
DIMENSIONS FACTS
| |
+-----+------+
|
v
GOLD MATERIALIZED
VIEWS
|
+-------+-------+
| | |
BI ML APPS
This is a common enterprise pattern, not a mandatory architecture.
38. Example enterprise pipeline
Suppose an organization receives customer orders from an operational database.
Step 1 — CDC ingestion
ERP / OMS
|
v
CDC
|
v
Bronze
Step 2 — SCD and current state
Bronze customer changes
|
v
AUTO CDC
|
v
Customer Silver
Step 3 — Business validation
Customer Silver
|
+--> validity
+--> completeness
+--> referential integrity
Step 4 — Business facts
Orders + Customer + Product
|
v
Gold Sales Fact
Step 5 — Business metrics
Gold Sales Fact
|
v
Daily Revenue
Monthly Revenue
Customer Revenue
Product Revenue
The resulting architecture is a complete data product rather than a collection of disconnected tables.
39. Common anti-patterns
Anti-pattern 1 — Everything is a streaming table
Not every dataset needs continuous incremental processing.
Anti-pattern 2 — Everything is a materialized view
Not every ingestion workload needs recomputed derived state.
Anti-pattern 3 — Continuous mode by default
Continuous execution can create unnecessary cost.
Anti-pattern 4 — Custom CDC logic everywhere
Use native AUTO CDC capabilities where they fit.
Anti-pattern 5 — Quality only in BI
Bad data should be controlled before it reaches consumers.
Anti-pattern 6 — No business grain
A Gold table without a declared grain becomes dangerous.
Anti-pattern 7 — Giant pipeline
One pipeline containing unrelated domains creates operational coupling.
Anti-pattern 8 — Manual orchestration of every dependency
Let the declarative dependency graph do its job.
Anti-pattern 9 — Hard-coded environments
Development and production should not require source-code rewrites.
Anti-pattern 10 — Treating successful execution as successful data
A green pipeline can still publish wrong business results.
40. Production operating model
An enterprise Lakeflow platform should define ownership.
| Area | Owner |
|---|---|
| Source contract | Source domain |
| Pipeline code | Data engineering |
| Data quality | Data product owner + engineering |
| Unity Catalog access | Data platform / governance |
| Cost | Platform + product team |
| SLA | Product owner |
| Incident response | Data engineering |
| Business KPI definition | Business owner |
This avoids the common failure mode where everyone assumes someone else owns data quality.
41. Architect decision framework
When designing a Lakeflow pipeline, ask these questions in order.
- What is the business data product?
- What is the grain?
- Is the source append-only, CDC, snapshot or event-based?
- What latency is actually required?
- Should the target be a streaming table or materialized view?
- Which quality rules are hard constraints?
- What happens when a record fails?
- How is historical change handled?
- What is the data contract?
- What is the SLA?
- How will the pipeline be observed?
- How will it be deployed and rolled back?
- What is the expected cost?
- Who owns the data product?
This sequence prevents technology-first architecture.
42. Lakeflow vs notebooks and jobs
Lakeflow and notebooks are not mutually exclusive.
Use notebooks or standard Spark jobs when:
- procedural logic is dominant;
- one-off processing is required;
- complex custom code is easier to express imperatively.
Use Lakeflow pipelines when:
- dataset dependencies are central;
- incremental processing matters;
- quality is part of the pipeline;
- declarative transformations fit the problem.
Use broader job orchestration around them when you need cross-system workflows.
A mature Databricks platform uses each appropriately.
43. Lakeflow vs dbt
dbt remains valuable when an organization has an established SQL transformation operating model.
Lakeflow is particularly strong when the workload needs:
- native streaming;
- managed pipeline semantics;
- expectations;
- AUTO CDC;
- Spark-based processing;
- integrated Databricks operations.
dbt can still run on Databricks.
The architect should choose based on:
TEAM SKILLS
+
WORKLOAD
+
STREAMING REQUIREMENTS
+
GOVERNANCE
+
OPERATING MODEL
rather than tool popularity.
44. Lakeflow vs custom Spark jobs
Custom Spark jobs provide maximum procedural flexibility.
Lakeflow provides more standardized data-pipeline semantics.
Use custom jobs when:
- procedural control is essential;
- specialized libraries dominate;
- external side effects are central;
- the workload does not map naturally to declarative datasets.
Use Lakeflow when:
- the workload is fundamentally a dataflow;
- dependencies are data-driven;
- incremental processing is important;
- quality and CDC should be integrated.
45. Hands-on lab — Build a Bronze-to-Gold pipeline
Objective
Build:
raw_orders
|
v
bronze_orders
|
v
silver_orders
|
v
gold_daily_sales
Step 1 — Create Bronze
Use a streaming table for incremental ingestion.
Step 2 — Add Silver
Clean:
- null keys;
- invalid amounts;
- malformed dates.
Step 3 — Add expectations
Example:
order_id IS NOT NULL
amount >= 0
Step 4 — Create Gold
Aggregate:
order_date
region
revenue
order_count
Step 5 — Observe
Inspect:
- execution;
- quality metrics;
- failed records;
- refresh behavior;
- runtime.
Step 6 — Break it intentionally
Inject invalid data and observe whether the expected action occurs.
This is more valuable than testing only the happy path.
46. Hands-on lab — CDC customer dimension
Build:
CUSTOMER CDC
|
v
AUTO CDC
|
v
CUSTOMER SCD2
Test:
- INSERT customer.
- UPDATE customer.
- UPDATE customer again.
- DELETE customer.
- Send events out of order.
- Verify sequencing.
- Verify historical versions.
The objective is to understand state transitions rather than memorize syntax.
47. Hands-on lab — Architect the right dataset type
Given these workloads:
| Workload | Decision |
|---|---|
| Kafka-like event ingestion | Streaming table |
| Daily customer revenue | Materialized view |
| Complex monthly aggregation | Materialized view |
| Intermediate cleansing step | Temporary view |
| CDC target | Streaming table + AUTO CDC |
| Dashboard acceleration | Materialized view |
Then explain why each choice was made.
The explanation is the architect skill.
48. Production checklist
Architecture
- Business grain defined.
- Source semantics documented.
- Dataset types intentionally selected.
- Pipeline boundaries defined.
- CDC strategy defined.
Quality
- Expectations defined.
- Severity defined.
- Quarantine strategy defined.
- Reconciliation defined.
- Freshness SLA defined.
Performance
- Incremental processing evaluated.
- State growth evaluated.
- Joins reviewed.
- File layout reviewed.
- Compute sizing reviewed.
Governance
- Unity Catalog objects defined.
- Access model reviewed.
- Ownership assigned.
- Lineage available.
- Audit requirements addressed.
Operations
- Event monitoring defined.
- Alerts configured.
- Retry strategy defined.
- Incident process defined.
- Rollback strategy defined.
Delivery
- Source controlled.
- Tests automated.
- Environment configuration separated.
- CI/CD implemented.
- Production deployment reviewed.
49. Enterprise architect exercise
Design a Databricks platform for a global retailer.
Sources
- ERP order database;
- CRM customer database;
- product master;
- e-commerce events;
- supplier files.
Requirements
- 15-minute sales freshness;
- customer history for five years;
- daily finance reconciliation;
- executive dashboards;
- data science consumption;
- strict governance;
- multiple business domains.
Your architecture should answer
- Where does each source land?
- Which sources use CDC?
- Which datasets are streaming tables?
- Which are materialized views?
- Where do expectations run?
- What gets quarantined?
- How is SCD2 implemented?
- What is the Gold grain?
- What is the KPI contract?
- How are pipelines deployed?
- How are failures monitored?
- How is cost controlled?
- How are domains isolated?
- How does Unity Catalog govern access?
- What happens when a producer changes its schema?
If the architecture cannot answer these questions, it is not yet enterprise-ready.
50. Architect mental model
Do not memorize Lakeflow syntax first.
Memorize this architecture:
SOURCE SEMANTICS
|
v
INGESTION MODEL
|
v
DATASET TYPE
|
v
QUALITY CONTRACT
|
v
INCREMENTAL / CDC STRATEGY
|
v
BUSINESS MODEL
|
v
OBSERVABILITY
|
v
GOVERNANCE
|
v
OPERATING MODEL
The syntax is the implementation detail.
The architecture is the skill.
51. What you should be able to explain after this chapter
You should now be able to explain:
- what Lakeflow Declarative Pipelines are;
- why declarative processing matters;
- how pipeline dependency graphs work;
- streaming tables;
- materialized views;
- temporary views;
- flows;
- SQL authoring;
- Python authoring;
- modern pipelines APIs;
- expectations;
- AUTO CDC;
- AUTO CDC FROM SNAPSHOT;
- SCD1;
- SCD2;
- sequencing;
- triggered vs continuous execution;
- serverless vs classic compute;
- observability;
- event logs;
- performance;
- cost;
- Unity Catalog governance;
- CI/CD;
- testing;
- production operating models;
- when Lakeflow should and should not be used.
52. The enterprise takeaway
Lakeflow Declarative Pipelines should not be viewed as simply a nicer way to write ETL.
They represent a shift from:
"Engineer writes every execution step"
to:
"Engineer declares the desired data system"
The platform can then manage:
DEPENDENCIES
INCREMENTAL PROCESSING
QUALITY
CDC
EXECUTION
OBSERVABILITY
The architect's responsibility therefore moves upward.
Instead of spending most of the design effort on:
"How do I execute step 7 after step 6?"
the architect should focus on:
"What is the correct data product, what guarantees does it provide, and what operating model keeps those guarantees true in production?"
That is the difference between knowing a Databricks feature and designing an enterprise Databricks platform.
References
- Databricks — Choose an ETL approach
- Databricks — What are pipelines?
- Databricks — Lakeflow pipeline best practices
- Databricks — Lakeflow pipeline Python reference
- Databricks — Pipeline SQL reference
- Databricks — Pipeline expectations
- Databricks — AUTO CDC
- Databricks — Apache Spark Declarative Pipelines