08.11 Data Quality & Expectations

1. Why data quality is an architecture concern

A lakehouse can be technically healthy while the data is still wrong.

A successful pipeline may finish on time, consume reasonable compute and produce valid Delta tables while still publishing incorrect business results.

Enterprise data engineering therefore needs two separate questions:

1. Did the pipeline run?
2. Did it produce trustworthy data?

The second question is data quality.

Animated flow: records move from source through ingestion into the quality gate. Passing records continue to trusted/gold data; failures are isolated in quarantine for investigation, correction and replay.

A production platform must make bad data visible, isolated and recoverable rather than silently propagating it.

2. Quality dimensions

Data quality is multidimensional.

Accuracy

Does the value represent reality?

Completeness

Are required records and attributes present?

Validity

Does a value satisfy the allowed domain?

Uniqueness

Does the business key occur only as often as expected?

Consistency

Do related datasets agree?

Timeliness

Did the record arrive within its expected time window?

Freshness

How recently was the dataset updated?

Integrity

Do relationships and business constraints hold?

These dimensions should become measurable rules.

3. Data quality is not just null checking

A weak implementation checks only for nulls.

Enterprise quality includes:

QUALITY
  |
  +--> schema
  +--> completeness
  +--> validity
  +--> uniqueness
  +--> referential integrity
  +--> business rules
  +--> reconciliation
  +--> freshness
  +--> volume
  +--> distribution
  +--> timeliness
  +--> security

The objective is not to maximize the number of checks. The objective is to identify the checks that protect the business contract.

4. Expectations as executable contracts

An expectation turns a quality rule into executable logic.

Conceptually:

EXPECT
    order_id IS NOT NULL

or:

EXPECT
    amount >= 0

or:

EXPECT
    status IN ('OPEN', 'CLOSED', 'CANCELLED')

Databricks Lakeflow Declarative Pipelines supports expectations for defining data quality constraints and controlling what happens when records violate them.

The architectural principle is:

A quality rule should be executable, observable and owned.

A rule documented only in a wiki is not enough.

5. Three possible outcomes

A quality rule should have an intentional response.

Accept

The record is valid and continues.

Quarantine

The record is suspicious or invalid but unrelated good records can continue.

Fail

The defect is severe enough that publishing the dataset would be unsafe.

Do not use one response for every rule.

6. Severity model

A useful enterprise model:

Severity	Example	Action
INFO	minor distribution drift	monitor
WARNING	optional attribute missing	publish + alert
ERROR	invalid business value	quarantine
CRITICAL	broken financial reconciliation	fail

Severity should be based on business impact, not technical inconvenience.

7. Hard vs soft constraints

A hard constraint makes the dataset unsafe when violated.

A soft constraint is useful to know but does not necessarily make the dataset unusable.

Example:

posting_date IS NOT NULL

may be a hard rule for regulatory finance data.

An optional phone number may be a soft rule.

This distinction prevents over-engineering.

8. Quality architecture

A mature architecture can look like:

                  SOURCE
                    |
                    v
                BRONZE
                    |
                    v
               VALIDATION
             /     |      \
            /      |       \
        ACCEPT   QUARANTINE  FAIL
          |          |         |
          v          v         v
       SILVER     DQ STORE   ALERT
          |
          v
        GOLD
          |
          v
      CONSUMERS

Quality should be integrated into the pipeline rather than bolted on after BI discovers bad data.

9. Schema quality

Schema quality starts before business rules.

Check:

• required columns;
• data types;
• nullability;
• allowed structural changes;
• nested structures;
• unexpected columns;
• renamed columns.

Example contract:

order_id       STRING
order_date     DATE
customer_id    STRING
amount         DECIMAL
status         STRING

A producer changing amount from DECIMAL to STRING may require controlled compatibility handling.

10. Schema evolution vs schema drift

These are different.

Schema evolution

An intentional, approved change.

Example:

ADD delivery_date

Schema drift

An unexpected change.

Example:

amount

suddenly arriving as:

"10,000 INR"

A mature platform should distinguish the two.

Do not automatically accept every incoming schema change.

11. Completeness checks

Completeness can be measured at multiple levels.

Record completeness

Received records divided by expected records.

Column completeness

Non-null values divided by total values.

Partition completeness

Example:

Expected:
2026-11-01 ... 2026-11-07

Received:
2026-11-01 ... 2026-11-06

Business completeness

Example:

Every posted invoice must have a company code.

Business completeness is often more valuable than simple row counts.

12. Uniqueness

Use the business key, not necessarily the entire row.

Example:

customer_id

may need to be unique in a current-state dimension.

For an order-line fact:

(order_id, line_number)

may be the correct key.

Always ask:

What makes two records the same business entity?

13. Referential integrity

Example:

orders.customer_id
        |
        v
customers.customer_id

A missing dimension row may be a temporary timing problem rather than a permanent defect.

The architecture should distinguish:

• temporary missing reference;
• permanently invalid reference;
• late-arriving dimension.

14. Business-rule validation

Technical validity is not business validity.

Examples:

quantity > 0

return_quantity <= shipped_quantity

invoice_total = SUM(invoice_lines)

delivery_date >= order_date

Business rules are where data quality becomes closely connected to domain knowledge.

15. Range and domain validation

Examples:

discount_pct BETWEEN 0 AND 100

currency_code IN ('INR', 'USD', 'EUR', 'GBP')

quantity >= 0

The rule should be explicit rather than hidden inside downstream SQL.

16. Distribution quality

A dataset can pass row-level validation while its distribution is wrong.

Example:

Yesterday: 1,000,000 orders
Today:            12 orders

Every row may be valid. The pipeline is still probably broken.

Monitor:

• row count;
• null percentage;
• distinct count;
• min/max;
• averages;
• percentiles;
• category distribution.

This is anomaly detection at the dataset level.

17. Freshness

Freshness asks:

How recently was the data successfully updated?

Example:

Expected freshness: < 15 minutes
Actual freshness:    47 minutes

The data may be perfectly accurate but operationally unusable.

Freshness should therefore be an explicit SLA.

18. Timeliness

Freshness and timeliness are related but different.

Freshness asks how recent the dataset is.

Timeliness asks whether a particular record arrived within its expected window.

Example:

Business event: 10:00
Received:       10:17
SLA:             5 minutes
Result:          FAIL

19. Volume anomaly detection

Monitor expected volumes.

Example:

Expected range: 900,000 - 1,100,000
Actual:           120,000

But do not blindly fail every anomaly. Holidays, planned shutdowns and source-system maintenance can legitimately change volumes.

Quality monitoring needs business context.

20. Quarantine architecture

A quarantine dataset should preserve enough information to investigate the defect.

Recommended fields:

• record payload;
• rule ID;
• rule name;
• failure reason;
• source system;
• ingestion timestamp;
• pipeline run ID;
• dataset name;
• quality severity.

Architecture:

              SOURCE
                 |
                 v
              BRONZE
                 |
          +------+------+
          |             |
        VALID         INVALID
          |             |
          v             v
       SILVER       QUARANTINE
                        |
                        v
                  DQ OPERATIONS

Never simply discard bad records unless there is a documented retention policy.

21. Quarantine is not a garbage dump

A useful quarantine system should answer:

• Why did this record fail?
• Which rule failed?
• When did it fail?
• Which source produced it?
• Which pipeline run processed it?
• Can it be replayed?
• Has the source corrected it?
• Who owns remediation?

Quarantine should be operationally useful.

22. Error record vs rejected record

A rejected record is data that cannot enter the target dataset.

A pipeline error is a failure of the processing system itself.

Example:

Malformed customer record

is a data-quality rejection.

But:

Unity Catalog permission denied

is a platform failure.

Do not classify infrastructure failures as data-quality failures.

23. Data quality metrics

Track quality quantitatively.

Examples:

valid_records / total_records
quarantined_records / total_records
freshness_minutes
reconciliation_variance
null_rate(column)
duplicate_rate

These metrics can feed a quality dashboard.

24. Quality score

A simplified quality score might combine completeness, validity, accuracy, freshness and integrity.

But do not create arbitrary scores merely to produce a green dashboard.

A critical finance reconciliation failure should not become “92% quality” because several low-impact checks passed.

Use both:

• quantitative metrics;
• critical-rule gates.

25. Data contracts

A data contract defines what a producer promises to consumers.

A contract can contain:

DATASET
  |
  +--> schema
  +--> keys
  +--> semantics
  +--> quality rules
  +--> freshness SLA
  +--> ownership
  +--> classification
  +--> change policy

This turns data quality from an informal expectation into an engineering interface.

26. Producer vs consumer responsibility

The producer should guarantee:

• schema;
• required fields;
• documented semantics;
• minimum quality;
• delivery SLA.

The consumer should not assume that every downstream transformation can repair producer defects.

PRODUCER
   |
   | contract
   v
DATA PRODUCT
   |
   | consumption
   v
CONSUMER

27. Quality gates by layer

Different layers need different expectations.

Bronze

Focus on:

• ingestion completeness;
• schema;
• source metadata;
• raw record preservation.

Silver

Focus on:

• canonical types;
• deduplication;
• business validity;
• referential integrity;
• CDC correctness.

Gold

Focus on:

• KPI correctness;
• reconciliation;
• semantic consistency;
• freshness;
• consumer SLAs.

Do not impose the exact same rules on every layer.

28. Bronze quality philosophy

Bronze should generally preserve source evidence.

If a source sends:

amount = "UNKNOWN"

do not necessarily destroy the evidence by silently converting it to zero.

Prefer preserving the raw value alongside parsing status:

raw_value = "UNKNOWN"
parsed_amount = NULL
parse_status = INVALID

This preserves recoverability.

29. Silver quality philosophy

Silver is where data becomes trustworthy enough for broad internal reuse.

Typical operations:

• type normalization;
• deduplication;
• standardization;
• CDC handling;
• invalid-record isolation;
• reference resolution;
• business-rule validation.

Silver should be substantially cleaner than Bronze.

30. Gold quality philosophy

Gold should enforce the strongest consumer-facing contracts.

Examples:

Revenue must reconcile to the approved finance source.

Daily sales must arrive by 06:00 UTC.

Customer KPI must use the approved customer definition.

Gold quality is therefore strongly tied to business ownership.

31. Expectations and pipeline behavior

A quality rule can conceptually use several behaviors:

EXPECT
  -> record remains
  -> violation counted

EXPECT OR DROP
  -> violating record removed

EXPECT OR FAIL
  -> pipeline fails

The correct behavior depends on the rule.

For example:

• optional email missing -> observe;
• invalid status -> quarantine;
• corrupted financial reconciliation -> fail.

Use the least disruptive response that still protects the contract.

32. Quality observability

A production platform should expose:

                  DATA QUALITY
                       |
       +---------------+---------------+
       |               |               |
    RULES           RESULTS          TRENDS
       |               |               |
   definitions      pass/fail       degradation
       |               |               |
       +---------------+---------------+
                       |
                       v
                   OPERATIONS

Operators need to see trends, not only today's failures.

33. Quality dashboard

A useful dashboard might show:

Metric	Current	SLA
Completeness	99.8%	>99.5%
Valid records	99.2%	>99%
Duplicate rate	0.04%	<0.1%
Freshness	8 min	<15 min
Reconciliation	0.01%	<0.1%
Critical failures	0	0

Add drill-down from:

metric
  -> dataset
     -> rule
        -> failed records

34. Alerting

Do not alert on every minor violation.

A good alert includes:

• dataset;
• rule;
• severity;
• current value;
• threshold;
• expected value;
• pipeline run;
• first occurrence;
• trend;
• owner;
• remediation link.

Example:

CRITICAL
Finance Revenue Reconciliation

Expected variance: <= 0.10%
Actual variance:     4.72%

Pipeline: finance_gold_daily
Run: 2026-11-08T05:00Z
Owner: Finance Data Product

That is actionable.

35. Quality incident lifecycle

Treat severe data defects like production incidents.

DETECTED
   |
   v
CLASSIFIED
   |
   v
CONTAINED
   |
   v
ROOT CAUSE
   |
   v
REMEDIATED
   |
   v
REPLAYED
   |
   v
VALIDATED
   |
   v
CLOSED

This is stronger than simply rerunning the failed job.

36. Root-cause analysis

When a Gold KPI is wrong, investigate upstream.

Gold
 |
 v
Silver
 |
 v
Bronze
 |
 v
Source

Ask:

1. Did the source change?
2. Did ingestion lose records?
3. Did schema interpretation change?
4. Did deduplication change?
5. Did CDC ordering change?
6. Did business logic change?
7. Did a dimension arrive late?
8. Did a downstream aggregate fail to refresh?

This prevents fixing symptoms instead of causes.

37. Reconciliation

For critical datasets, reconcile independently.

Example:

S/4HANA
   |
   +---- expected revenue
   |
   v
Databricks Gold
   |
   +---- calculated revenue

Then:

variance = expected - actual

Set a documented tolerance.

For finance, reconciliation should be treated as a control, not merely a monitoring metric.

38. Quality and CDC

CDC introduces additional quality risks:

• duplicate changes;
• missing deletes;
• out-of-order events;
• duplicate keys;
• invalid operation codes;
• late-arriving changes.

A CDC quality contract should validate the operation domain and ensure that the resulting current state is consistent.

39. Quality and SCD2

For SCD2 dimensions, test:

• one current row per business key;
• no overlapping validity intervals;
• valid start/end dates;
• correct current flag;
• deterministic ordering.

Example:

customer_id = C1001

2024-01-01 -> 2025-03-01
2025-03-01 -> 9999-12-31

Overlapping intervals indicate a quality defect.

40. Quality and late-arriving data

Late data is not automatically bad data.

Example:

Business event: 10:00
Received:       10:17
Allowed SLA:    30 minutes
Result:         VALID

Quality rules require contextual thresholds.

41. Quality and replay

A quality architecture should support replay.

Preferred:

immutable/raw evidence
        |
        v
reprocess
        |
        v
corrected Silver
        |
        v
corrected Gold

If Bronze is overwritten destructively, recovery becomes much harder.

42. Quality and idempotency

A replay should not create duplicates.

If the same source batch is processed twice, the target should still contain the correct business records.

Quality and idempotency are tightly connected.

43. Quality ownership model

Define:

Role	Responsibility
Data producer	upstream correctness
Data engineer	implementation
Data owner	business definition
Data steward	governance
Platform team	tooling and observability
Consumer	consumption contract

Without ownership, quality becomes everyone's problem and therefore nobody's problem.

44. Quality SLAs and SLOs

Examples:

Freshness SLO

99% of daily loads complete within 15 minutes.

Validity SLO

At least 99.5% valid records.

Critical reconciliation SLO

100% within approved tolerance.

SLOs should be measurable and reviewed.

45. Quality anti-patterns

Silent cleansing

Changing bad values without recording the change.

Converting unknown to zero

Can create false financial results.

Dropping invalid records without audit

Destroys evidence.

One severity for every rule

Creates noisy operations.

Quality only in Gold

Allows bad data to travel too far.

Quality only in Bronze

Misses business semantics.

No quarantine

Makes investigation difficult.

No replay strategy

Makes recovery expensive.

No ownership

Creates recurring defects.

Green dashboard with critical failure

A weighted score must never hide a critical control failure.

46. Lab 1 — Expectations

Create a Silver pipeline with rules for:

• non-null business key;
• valid status;
• non-negative amount;
• valid date;
• duplicate detection.

Classify each as:

• observe;
• quarantine;
• fail.

Document why.

47. Lab 2 — Quarantine framework

Build a quarantine dataset containing:

• original record;
• rule ID;
• rule name;
• failure reason;
• source;
• pipeline run ID;
• timestamp;
• severity.

Create a query that shows the top failing rules.

48. Lab 3 — Quality dashboard

Build a quality dashboard containing:

• pass rate;
• failure rate;
• quarantine volume;
• freshness;
• duplicate rate;
• completeness;
• reconciliation variance.

Add a seven-day trend.

49. Lab 4 — Enterprise reconciliation

Build a simplified SAP-to-Databricks scenario.

Compare:

source revenue
vs
Silver revenue
vs
Gold revenue

Introduce:

• duplicate records;
• missing records;
• currency mismatch;
• late posting.

Detect and explain each variance.

50. Architect exercise

Design a quality operating model for:

• 500+ datasets;
• Finance;
• Procurement;
• Sales;
• Supply Chain;
• 15-minute operational analytics;
• daily executive reporting;
• SAC and Tableau;
• SAP source systems;
• CDC;
• SCD2 dimensions;
• regulatory reporting.

Define:

1. quality dimensions;
2. rule ownership;
3. severity model;
4. expectation behavior;
5. quarantine strategy;
6. reconciliation;
7. freshness SLAs;
8. alerting;
9. replay;
10. incident management;
11. quality dashboard;
12. governance model;
13. critical-data controls;
14. cost strategy.

Your architecture should explicitly answer:

Which failures stop the pipeline, which failures isolate records, and which failures only alert?

51. Production checklist

• Quality dimensions are defined.
• Critical datasets have explicit contracts.
• Expectations are executable.
• Rule severity is documented.
• Critical failures can stop publication.
• Invalid records can be quarantined.
• Rejected records are auditable.
• Schema drift is detected.
• Completeness is measured.
• Uniqueness is measured.
• Referential integrity is tested.
• Business rules are tested.
• Freshness is monitored.
• Timeliness is monitored where required.
• Volume anomalies are monitored.
• Critical data is reconciled.
• Quality metrics are trendable.
• Alerts are actionable.
• Ownership is defined.
• Replay is possible.
• Pipelines are idempotent.
• Quality incidents have an operating process.

52. Architect mental model

Think of data quality as a control system:

             DATA SOURCES
                  |
                  v
             DATA CONTRACT
                  |
                  v
              PIPELINE
                  |
          +-------+-------+
          |               |
       QUALITY          DATA
       RULES           PROCESSING
          |               |
          +-------+-------+
                  |
          +-------+-------+
          |       |       |
        PASS   QUARANTINE FAIL
          |       |       |
          v       v       v
       TRUSTED   DQ     ALERT /
        DATA     OPS    STOP
          |
          v
      DATA PRODUCTS

The mature architect does not ask:

How do I check whether this table has nulls?

The architect asks:

What does trustworthy mean for this data product, how is that definition executable, who owns it, what happens when it fails, and how do we recover?

---

Further reading

• Databricks — Data quality with expectations
• Databricks — Lakeflow Declarative Pipelines
• Databricks — Data engineering best practices