13.4 — Freshness, Quality & Reconciliation

13.30 Freshness monitoring

Freshness should be modeled as a first-class operational KPI.

Example:

Expected refresh: every 30 min Maximum acceptable age: 45 min

If:

current_time - latest_successful_watermark > 45 min

then the dataset is outside its freshness SLO.

Track freshness separately from task status.

A task can be green while the business dataset is stale because the source itself stopped changing.

---

13.31 Completeness monitoring

A pipeline can be fresh but incomplete.

Examples:

• expected 10 million rows;
• received 8 million rows.

Or:

• expected all company codes;
• one company code is missing.

Completeness checks should be based on business controls, not only technical execution status.

---

13.32 Reconciliation controls

For important pipelines, compare:

• source row count;
• target row count;
• control totals;
• min/max dates;
• distinct business keys;
• rejected-record counts;
• watermark;
• load timestamp.

Example:

Source count = 1,250,000 Target count = 1,250,000 Revenue source = 875,420,100 Revenue target = 875,420,100

A successful technical run plus reconciliation gives stronger evidence than either alone.

---

13.33 Data-quality failure classification

Classify data-quality incidents:

Class	Example
Completeness	Missing records
Uniqueness	Duplicate key
Validity	Invalid code
Consistency	Header/item mismatch
Timeliness	Stale data
Accuracy	Incorrect measure
Referential integrity	Missing dimension
Semantic	Wrong unit/currency
Security	Unauthorized exposure

This classification improves triage and ownership.

---

13.34 Data-quality gates

A production pipeline can use gates:

Acquire | v Validate | PASS / FAIL / PASS FAIL | | v v Publish Quarantine | v Consume

Do not publish known-bad data simply because the ingestion task succeeded.

---

13.35 Incident severity

A useful operational model:

Severity	Meaning
Sev 1	Critical business outage or major data risk
Sev 2	Important production degradation
Sev 3	Limited impact or workaround exists
Sev 4	Non-urgent defect or improvement

Severity should depend on business impact, not technical complexity.

---

13.36 Incident lifecycle

Use:

Detect | v Triage | v Contain | v Diagnose | v Recover | v Validate | v Communicate | v RCA | v Prevent recurrence

Do not close an incident immediately after the task turns green.

---

13.37 Triage checklist

When a production incident arrives, capture:

• affected business process;
• affected object;
• affected space;
• first known bad timestamp;
• last known good timestamp;
• expected freshness;
• actual freshness;
• task status;
• error message;
• recent changes;
• source health;
• downstream impact;
• security impact;
• current workaround.

---

13.38 Last-known-good analysis

The most useful question is often:

When was this data last definitely correct?

Record:

Last known good = T1 First known bad = T2

Then inspect:

• deployments between T1 and T2;
• source changes;
• schema changes;
• schedule changes;
• credential changes;
• data-volume changes;
• task configuration changes.

This narrows the search space.

---

13.39 Change correlation

Operational troubleshooting should correlate incidents with:

• model releases;
• connection changes;
• source upgrades;
• transports;
• task-chain changes;
• schedule changes;
• authorization changes;
• infrastructure maintenance.

A failure that starts immediately after a change deserves priority investigation of that change.

---