Skip to main content

10. Structured Streaming — Real-Time Data Engineering

Goal: understand streaming from first principles, then design production-grade Databricks streaming systems without treating streaming as “batch with a loop.”

Structured Streaming is the execution model for incremental, continuously arriving data in Spark. Current Databricks guidance recommends Lakeflow pipelines for new ETL, ingestion, and Structured Streaming workloads, while the underlying Structured Streaming concepts remain essential for understanding state, checkpoints, watermarks, triggers, and failure recovery.

1. The mental model​

A batch pipeline asks:

“What data should I process now?”

A streaming pipeline asks:

“What new data has arrived since my last safe progress point, and what state must I retain to produce the correct result?”

The core architecture is:

Source → Stream Read → Transform → State → Sink

Four operational concepts sit underneath it:

  • Offset — how far the source has been consumed.
  • Checkpoint — durable progress and recovery metadata.
  • State — information retained across micro-batches.
  • Watermark — a bound that lets the engine eventually discard old state.

Animated Structured Streaming architecture

Architect takeaway​

A streaming design is not complete until you can answer:

  1. Where is progress stored?
  2. What happens after a worker failure?
  3. How much state can accumulate?
  4. How is late data handled?
  5. What is the delivery guarantee at the sink?
  6. What is the latency target?
  7. What is the cost of keeping the pipeline running?

2. Micro-batch: the default mental model​

Spark Structured Streaming commonly processes data in micro-batches.

Imagine events arriving continuously:

time ───────────────────────────────────────────────>

events:  e1 e2 e3   e4 e5     e6 e7 e8       e9 ...

batch 1: [e1 e2 e3]
batch 2:           [e4 e5]
batch 3:                     [e6 e7 e8]
batch 4:                                   [e9 ...]

Each batch becomes an incremental computation.

This is why Structured Streaming can use the same DataFrame-style transformations you already learned in Spark.

Do not confuse these​

ConceptMeaning
Event timeTime carried by the business event
Processing timeTime the platform processes the event
OffsetSource position/progress
CheckpointDurable recovery information
TriggerDetermines when the engine processes available data
WatermarkBound used to control late data and state

3. Event time vs processing time​

Suppose an order was created at:

10:00:03

but the network was slow and Databricks received it at:

10:02:41

Then:

  • event_time = 10:00:03
  • processing_time ≈ 10:02:41

If you build a five-minute business window using processing time, you can produce incorrect business results.

Architect rule​

Use event time for business semantics.

Use processing time for operational behavior.

4. Stateless vs stateful streaming​

This is one of the most important distinctions.

Stateless​

Each record can be processed independently.

Examples:

  • filter
  • select
  • cast
  • rename
  • simple projection
  • simple parsing

Conceptually:

input row → transformation → output row

No historical information is required.

Stateful​

The engine needs information from previous records or batches.

Examples:

  • Aggregations
  • Deduplication
  • Stream-stream joins
  • Sessionization
  • Running counts
  • Windowed analytics

Conceptually:

new event
    │
    ├── current state
    │
    └── update state
           │
           ▼
        new result

State is powerful — but it is also one of the biggest production risks.

5. Why state can become dangerous​

Suppose you calculate:

count(*) GROUP BY customer_id

The engine needs to remember customer state.

If millions of customers exist, the state store can become large.

Now consider:

GROUP BY customer_id, device_id, campaign_id, session_id

The state cardinality can explode.

Architect question​

Do not ask only:

“Will this query work?”

Ask:

“How much state will this query retain at production scale?”

6. Watermarks​

A watermark provides a logical boundary for old data.

Example:

withWatermark("event_time", "10 minutes")

The intent is approximately:

“I will tolerate events arriving up to 10 minutes behind the observed event-time progress, subject to the operator's semantics.”

This lets the engine eventually remove state that can no longer contribute to valid results.

Without a bounded state strategy​

event → state
event → state
event → state
event → state
...
             ↓
       state keeps growing

With a watermark​

old state ───────────► eligible for cleanup
                     ↑
                 watermark

Important​

A watermark is not a guarantee that every late record will be accepted.

It is a mechanism for bounding state and defining late-data behavior.

7. Windowed aggregation​

A common architecture is:

events
   │
   ▼
event-time window
   │
   ▼
aggregation
   │
   ▼
Delta table

For example:

  • 5-minute windows
  • 15-minute windows
  • hourly windows
  • daily windows

Business example:

Count orders per region every five minutes based on event time.

The important design choices are:

  1. Window size
  2. Allowed lateness
  3. Grouping cardinality
  4. Output mode
  5. State retention
  6. Sink behavior

8. Checkpointing​

A checkpoint is the recovery mechanism that allows a streaming query to continue from durable progress rather than starting from zero.

Think of it as:

SOURCE
  │
  ▼
processed offsets
  │
  ▼
CHECKPOINT
  │
  ├── progress
  ├── state metadata
  └── recovery information

Production rule​

A streaming query should have a stable, dedicated checkpoint location.

Do not casually reuse a checkpoint directory between unrelated streaming queries.

Because the checkpoint represents the identity and progress of a particular streaming computation.

9. What happens when the stream crashes?​

Imagine:

events 1 ───── 100
             │
             ▼
        checkpoint
             │
             X  ← worker failure

On restart, the engine uses its checkpoint and source progress to recover.

The goal is:

restart ≠ start from zero

This is one of the fundamental differences between a carefully designed streaming pipeline and a script that simply polls a source.

10. Trigger strategy​

A streaming workload must define when processing happens.

Triggered execution​

Process available data and stop.

This is attractive when:

  • low latency is not mandatory
  • cost matters
  • data arrives in bursts
  • scheduled incremental processing is sufficient

Continuous operation​

Keep the streaming workload running.

Use this when the business genuinely requires continuous processing.

Architect trade-off​

lower latency
      ↑
      │       cost
      │        ↑
      │        │
      └────────┴────────►
             runtime

Do not choose continuous execution simply because it sounds more “real time.”

Choose it because the business latency requirement demands it.

11. Auto Loader + Structured Streaming​

Auto Loader is particularly important for cloud-file ingestion.

The mental model is:

S3 / ADLS / GCS
       │
       ▼
   Auto Loader
       │
       ▼
Structured Streaming
       │
       ▼
Bronze Delta
       │
       ▼
Silver

Auto Loader handles incremental discovery of files while Structured Streaming provides the incremental processing model.

This combination is one of the most important Databricks ingestion patterns.

12. Kafka and message-bus ingestion​

For event-driven architectures, the source may be Kafka, Kinesis, Event Hubs, or another message system.

Conceptually:

Application
    │
    ▼
Message Bus
    │
    ▼
Databricks Streaming
    │
    ├── validation
    ├── parsing
    ├── deduplication
    └── enrichment
          │
          ▼
      Delta Lake

Important design questions​

  • What is the partitioning strategy?
  • What is the event key?
  • Can events arrive more than once?
  • How much out-of-order data exists?
  • What is the retention period?
  • What is the target latency?
  • What happens if downstream processing is unavailable?

13. Deduplication​

Real systems frequently produce duplicate events.

Example:

order_id = 1001
event_id = abc123

The same event may appear twice because of retries.

A robust streaming design therefore needs an event identity strategy.

Typical key:

event_id

or a carefully defined composite key.

Architect rule​

Never assume:

“The source won't duplicate messages.”

Instead ask:

“How will the pipeline behave if the same event arrives twice?”

14. Stream-stream joins​

Joining two unbounded streams is much harder than joining two static tables.

Example:

Orders Stream ─────┐
                   ├── join ──► enriched events
Payments Stream ───┘

The engine may need to retain records from both sides while waiting for matching events.

That means:

  • state grows
  • latency matters
  • event-time constraints matter
  • watermarks become important

Example business scenario​

An order may arrive first:

10:00:01 ORDER

and its payment later:

10:00:47 PAYMENT

The system must define how long it is willing to wait.

15. Exactly-once: understand the boundary​

“Exactly-once” is often misunderstood.

A managed Delta-to-Delta pipeline can provide strong exactly-once processing semantics through coordinated checkpointing and transactional commits.

But the guarantee does not automatically extend across arbitrary external systems.

For example:

Kafka
  │
  ▼
Databricks
  │
  ▼
Delta

is very different from:

Databricks
  │
  ▼
custom REST API

For external sinks, you must reason about:

  • retries
  • duplicate writes
  • idempotency
  • transaction boundaries
  • acknowledgements

Enterprise principle​

Exactly-once is a system-level property, not a magic label attached to every streaming query.

16. Output modes​

Streaming aggregations can have different output semantics.

Append​

Only newly finalized rows are emitted.

Update​

Rows whose results changed are emitted.

Complete​

The full result table is emitted.

Do not choose a mode by habit.

Choose based on:

  • query type
  • downstream sink
  • state size
  • latency
  • update semantics

17. Structured Streaming vs Lakeflow pipelines​

Modern Databricks architecture requires understanding both.

Structured Streaming​

You directly work with the Spark streaming API.

Useful for:

  • learning streaming fundamentals
  • specialized transformations
  • advanced custom streaming workloads
  • understanding the execution model

Lakeflow pipelines​

You define the desired data pipeline declaratively.

The platform can manage:

  • dependency resolution
  • incremental processing
  • orchestration
  • checkpointing
  • recovery
  • data quality
  • monitoring

Current Databricks guidance recommends Lakeflow pipelines for most new ETL and streaming workloads.

Architect decision​

RequirementPreferred direction
Learn core streaming mechanicsStructured Streaming
New production ETLLakeflow pipelines
File ingestionAuto Loader + Lakeflow
Complex custom Spark logicSpark / Structured Streaming
Declarative production pipelineLakeflow
External streaming sinkEvaluate Lakeflow sink / custom streaming carefully

18. Lakeflow streaming tables​

A streaming table represents incrementally processed data.

The conceptual model is:

SOURCE
  │
  ▼
FLOW
  │
  ▼
STREAMING TABLE
  │
  ▼
DOWNSTREAM TABLES

Lakeflow pipelines automatically reason about dependencies between datasets and execute the graph accordingly.

This is a major architectural shift from manually orchestrating every transformation step.

Instead of saying:

“Run job A, then job B, then job C.”

you increasingly describe:

“This dataset depends on that dataset.”

The platform can derive the execution graph.

19. Data quality in streaming​

Streaming quality must be continuous.

A batch mentality says:

run → validate → finish

A streaming system says:

event → validate → process
event → validate → process
event → validate → process
...

Quality rules should therefore define explicit outcomes:

  • warn
  • drop/quarantine
  • fail

This connects directly to the previous chapter on data quality.

Enterprise pattern​

Bronze
  │
  ▼
quality gate
  ├── PASS ──► Silver
  │
  └── FAIL ──► quarantine / remediation

20. Production observability​

Never monitor only:

“Is the job green?”

A production streaming platform needs deeper signals.

Pipeline health​

  • run status
  • failures
  • restart count
  • execution duration

Streaming health​

  • input rate
  • processing rate
  • backlog
  • batch duration
  • trigger latency

State health​

  • state size
  • state growth
  • watermark progress
  • late-event volume

Data health​

  • records processed
  • rejected records
  • duplicates
  • schema changes
  • null/error rates

Business health​

  • orders/hour
  • revenue/hour
  • active customers
  • payment success rate

The strongest platforms correlate technical health with business health.

21. Backpressure and lag​

Suppose:

Incoming = 100,000 events/sec
Processing = 70,000 events/sec

Then backlog grows:

100k - 70k = 30k events/sec

If this continues:

lag ↑
latency ↑
state pressure ↑
cost ↑
risk ↑

A streaming architect therefore monitors the relationship between:

input rate
vs
processing rate

not just CPU utilization.

22. Performance engineering​

22.1 Avoid unnecessary state​

Stateful operations are more expensive than simple projections and filters.

22.2 Control cardinality​

A high-cardinality grouping can create enormous state.

22.3 Use watermarks deliberately​

Watermarks help bound state, but they must match the business tolerance for late data.

22.4 Avoid unnecessary wide transformations​

Reduce columns early.

22.5 Choose the correct trigger​

Do not pay continuous-compute cost for a workload that can run incrementally on a trigger.

22.6 Optimize the sink​

A fast stream feeding a poorly designed Delta layout can still become a production bottleneck.

22.7 Watch small files​

High-frequency micro-batches can produce many small files.

That affects downstream read performance and maintenance.

23. Streaming architecture patterns​

Pattern A — File ingestion​

Cloud Storage
     │
     ▼
Auto Loader
     │
     ▼
Bronze
     │
     ▼
Silver
     │
     ▼
Gold

Best for:

  • SaaS exports
  • application files
  • data lake landing zones

Pattern B — Event-driven analytics​

Application
     │
     ▼
Kafka / Kinesis / Event Hubs
     │
     ▼
Streaming
     │
     ▼
Delta
     │
     ├── BI
     ├── ML
     └── downstream applications

Best for:

  • telemetry
  • clickstream
  • transactions
  • IoT

Pattern C — CDC​

Operational DB
     │
     ▼
CDC
     │
     ▼
Bronze
     │
     ▼
Silver / SCD
     │
     ▼
Gold

Best for:

  • ERP
  • CRM
  • transactional databases
  • dimensional history

24. Streaming architecture anti-patterns​

Anti-pattern 1 — No checkpoint strategy​

If the pipeline cannot recover safely, it is not production ready.

Anti-pattern 2 — Infinite state​

A stateful query with no practical state bound can become a slow-motion outage.

Anti-pattern 3 — Processing-time business logic​

Using arrival time where event time matters can create incorrect analytics.

Anti-pattern 4 — Continuous mode everywhere​

“Real time” is not automatically better architecture.

Anti-pattern 5 — External sink without idempotency​

Retries can create duplicates.

Anti-pattern 6 — No late-data policy​

Every serious event pipeline needs an answer for delayed events.

Anti-pattern 7 — No replay strategy​

You need to know how to reconstruct data after corruption or logic changes.

25. Replay and recovery architecture​

A mature architecture separates:

raw landing data from derived business state.

That allows:

Raw events
    │
    ├── normal processing ──► Silver
    │
    └── replay ─────────────► rebuilt Silver

This is one reason the Bronze layer is so valuable.

It provides a durable source for:

  • debugging
  • backfills
  • reprocessing
  • new transformations
  • disaster recovery

26. Security architecture​

Streaming pipelines should run with controlled identities.

Prefer:

Human developer
      │
      ▼
Git / deployment
      │
      ▼
Service identity
      │
      ▼
Databricks pipeline
      │
      ▼
Unity Catalog governed data

Avoid embedding credentials in notebooks or source code.

Security design should cover:

  • identity
  • data permissions
  • secrets
  • network access
  • auditability
  • sensitive columns
  • external sinks

27. Cost architecture​

Streaming cost is driven by more than row count.

Consider:

cost ≈ compute
      + state management
      + storage
      + data transfer
      + downstream processing

The biggest architectural cost mistake is often unnecessary always-on compute.

Ask:

“What is the minimum freshness SLA that satisfies the business?”

If the answer is 15 minutes, a continuously running sub-second pipeline may be economically irrational.

28. Hands-on lab — build a streaming Bronze pipeline​

Step 1 — Create a streaming source​

For a learning environment, use a small file-based source.

from pyspark.sql import functions as F

source_path = "/Volumes/main/landing/events"

events = (
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load(source_path)
)

Step 2 — Normalize the schema​

clean_events = (
    events
    .select(
        F.col("event_id").cast("string"),
        F.col("customer_id").cast("string"),
        F.col("event_type").cast("string"),
        F.to_timestamp("event_time").alias("event_time"),
        F.col("amount").cast("decimal(18,2)")
    )
)

Step 3 — Write to Delta​

query = (
    clean_events.writeStream
        .format("delta")
        .option("checkpointLocation", "/Volumes/main/checkpoints/events")
        .outputMode("append")
        .toTable("main.bronze.events")
)

Step 4 — Observe​

Measure:

  • input rows
  • processed rows
  • batch duration
  • source lag
  • checkpoint health
  • output files

Do not stop at “the table has data.”

29. Hands-on lab — watermark + aggregation​

Conceptually:

from pyspark.sql import functions as F

aggregated = (
    events
    .withWatermark("event_time", "10 minutes")
    .groupBy(
        F.window("event_time", "5 minutes"),
        F.col("event_type")
    )
    .count()
)

Now ask:

  1. What state is retained?
  2. Why is the watermark needed?
  3. What happens to a record that arrives very late?
  4. How does the choice of window size affect state?
  5. What happens when event volume increases 100x?

These are architect questions, not syntax questions.

30. Architect exercise​

Design a platform for:

5 million events/minute, 2-minute freshness SLA, 24-hour replay capability, 30-day analytical retention, multiple downstream consumers.

Your design should specify:

Ingestion​

  • source technology
  • partition strategy
  • schema strategy

Processing​

  • stateless vs stateful operations
  • watermark
  • deduplication
  • enrichment

Storage​

  • Bronze
  • Silver
  • Gold
  • retention

Reliability​

  • checkpoint
  • replay
  • retry
  • idempotency

Governance​

  • Unity Catalog
  • service identity
  • sensitive data handling

Operations​

  • backlog monitoring
  • alerting
  • cost controls

Business SLA​

  • freshness
  • correctness
  • availability

31. Structured Streaming architect checklist​

Data​

  • Event-time semantics defined
  • Schema evolution strategy defined
  • Duplicate-event strategy defined
  • Late-data policy defined

State​

  • Stateful operations identified
  • State cardinality estimated
  • Watermark strategy defined
  • State growth monitored

Reliability​

  • Checkpoint location defined
  • Restart behavior tested
  • Replay strategy defined
  • External sink idempotency tested

Performance​

  • Input rate measured
  • Processing rate measured
  • Backlog alerts configured
  • Trigger selected from SLA
  • Small-file impact considered

Governance​

  • Unity Catalog permissions defined
  • Service identity used
  • Secrets externalized
  • Audit requirements satisfied

Cost​

  • Freshness SLA agreed
  • Continuous compute justified
  • Storage retention justified
  • Scaling behavior tested

32. Interview questions you should be able to answer​

Beginner​

  1. What is Structured Streaming?
  2. What is a micro-batch?
  3. What is a checkpoint?
  4. What is a trigger?
  5. What is event time?

Intermediate​

  1. Why do streaming aggregations require state?
  2. What is a watermark?
  3. Why can state grow indefinitely?
  4. How do you handle duplicate events?
  5. What is the difference between event time and processing time?

Advanced​

  1. How do stream-stream joins work?
  2. How would you design a replay strategy?
  3. What causes streaming lag?
  4. How do you make an external sink idempotent?
  5. How do you size a streaming workload?

Architect​

  1. When would you choose Lakeflow pipelines over direct Structured Streaming?
  2. How would you design a multi-region streaming platform?
  3. How would you guarantee business correctness with late-arriving data?
  4. How would you design for 10x traffic growth?
  5. How would you balance a 30-second SLA against platform cost?

If you can answer these with architecture diagrams and trade-offs rather than memorized definitions, you are moving toward architect-level thinking.

33. The architect mental model​

Remember these five questions:

1. WHAT ARRIVED?
        ↓
2. HOW FAR HAVE I SAFELY PROCESSED?
        ↓
3. WHAT STATE MUST I REMEMBER?
        ↓
4. HOW LATE CAN DATA ARRIVE?
        ↓
5. HOW DO I COMMIT THE RESULT SAFELY?

That is the core of production streaming.

Final takeaway​

Structured Streaming is not primarily about making data move faster.

It is about building a system that can continuously process changing data while remaining:

  • correct
  • recoverable
  • observable
  • governable
  • scalable
  • economically justified

Databricks' current platform direction places Lakeflow pipelines around these same concerns: declarative batch/streaming development, automatic dependency management, incremental processing, data quality, monitoring, and production operations.

Architect principle: If you cannot explain the checkpoint, state, watermark, replay path, sink guarantee, SLA, and cost model, you do not yet have a production streaming architecture.