13.6 — Recovery, Backfill & Operations
13.61 RCA methodology
Use five layers:
Symptom
What users saw.
Failure
What technically failed.
Cause
Why the failure occurred.
Contributing factors
Why detection or recovery was difficult.
Prevention
What changes reduce recurrence.
Example:
Symptom: Finance dashboard stale.
Failure: Persistence task failed.
Cause: Upstream transformation exceeded memory.
Contributing factor: No freshness alert until morning.
Prevention: Reduce intermediate volume and add freshness SLO monitoring.
13.62 Five Whys example
Why was the dashboard stale?
Because the persistence task failed.
Why did persistence fail?
Because memory was exhausted.
Why was memory exhausted?
Because a join processed a much larger dataset than baseline.
Why did volume increase?
A source filter was removed.
Why was the change not detected?
No performance regression and data-volume control existed.
The RCA should address the control-system failure, not only the immediate technical error.
13.63 Evidence hierarchy
Prefer evidence in this order:
- Runtime and task logs.
- Monitor status/history.
- Source-system logs.
- Data reconciliation.
- Model lineage.
- Recent changes.
- Operator hypothesis.
Never turn a hypothesis into a root cause before evidence confirms it.
13.64 Incident timeline
Create a timeline:
| Time | Event |
|---|---|
| 08:00 | Normal run |
| 08:30 | Source change |
| 08:35 | Load starts |
| 08:41 | Transformation warning |
| 08:43 | Persistence fails |
| 09:05 | Alert fires |
| 09:15 | Recovery starts |
| 09:22 | Data reconciled |
Timeline reconstruction often reveals causal relationships.
13.65 Dependency-first troubleshooting
Do not troubleshoot alphabetically.
Troubleshoot according to dependency direction:
Consumer | v Semantic model | v View | v Persistence | v Flow | v Remote table | v Source
For data absence, follow upstream until the first divergence.
For performance, follow the execution path and identify the most expensive boundary.
13.66 Monitoring architecture
A mature architecture looks like:
+-----------------------------+ | Business Consumers | +-------------+---------------+ | v +-----------------------------+ | Semantic / Analytic Models | +-------------+---------------+ | v +-----------------------------+ | Runtime + View Analyzer | +-------------+---------------+ | v +-----------------------------+ | Pipeline + Data Integration | | Monitor + Task Logs | +-------------+---------------+ | v +-----------------------------+ | Sources / Agents / Network | +-----------------------------+
Across every layer:
freshness correctness availability performance security
13.67 Operational ownership model
Assign ownership at the boundary.
| Boundary | Typical owner |
|---|---|
| Source system | Source team |
| Connection | Integration/platform team |
| Replication | Data engineering |
| Transformation | Data engineering/model owner |
| Semantic model | Analytics/model owner |
| Security | Data governance/security |
| Consumption | BI/product team |
| Platform | Datasphere platform team |
Ownership should be explicit.
13.68 RACI for incidents
A practical model:
| Activity | Platform | Data Engineer | Model Owner | Business |
|---|---|---|---|---|
| Platform availability | A/R | C | I | I |
| Pipeline recovery | C | A/R | C | I |
| Semantic correction | C | C | A/R | C |
| Data acceptance | I | C | R | A |
| Security incident | A/R | C | C | I |
Adapt this to the organization.
13.69 Production change control
Monitor changes to:
- connections;
- source schemas;
- transformations;
- graphical views;
- SQL views;
- analytic models;
- schedules;
- task chains;
- security;
- persistence;
- partitioning.
Every high-risk change should have:
- expected impact;
- rollback;
- validation;
- owner;
- change window.
13.70 Safe recovery principles
Prefer the smallest safe action.
Order:
- Observe.
- Contain.
- Correct.
- Retry.
- Reconcile.
- Validate.
- Resume downstream processing.
Avoid broad destructive actions such as dropping and rebuilding production objects unless the recovery plan explicitly requires it.
13.71 Retry versus rerun
A retry usually means:
continue from a failed execution boundary
A rerun may mean:
execute the whole workload again
The distinction matters for:
- duplicates;
- source load;
- runtime;
- downstream side effects;
- idempotency.
Use the narrowest recovery mechanism that restores correctness.
13.72 Backfill incidents
When a dataset has a gap:
normal data 2026-01-01 ... 2026-03-10 | GAP | 2026-03-11 | normal data resumes
Design a controlled backfill:
- Identify exact gap.
- Determine source availability.
- Isolate affected period.
- Run controlled load.
- Reconcile.
- Validate downstream.
- Resume normal schedule.
Do not reload the entire historical dataset automatically.
13.73 Late-arriving data
Late data can create:
- temporary freshness gaps;
- incorrect aggregates;
- missing dimension mappings;
- reconciliation differences.
Define whether the architecture supports:
- correction;
- replay;
- incremental update;
- periodic reconciliation.
The business freshness contract should distinguish delayed source data from pipeline failure.
13.74 Partial success
A pipeline can partially succeed.
Example:
Company A -> loaded Company B -> loaded Company C -> failed
The correct recovery depends on task semantics.
Track:
- successful partitions;
- failed partitions;
- target state;
- retry behavior;
- reconciliation scope.
Do not assume the whole dataset is invalid.
13.75 Monitoring and partitioning
Partition monitoring should connect technical size to workload behavior.
Track:
- partition growth;
- skew;
- record count;
- storage;
- query patterns;
- pruning effectiveness.
A partition that grows rapidly can become a future incident even if today's queries are healthy.
13.76 Monitoring storage
Storage monitoring should identify:
- rapid growth;
- unexpected persistence;
- redundant copies;
- large staging objects;
- unused data;
- file/object-store growth where applicable.
Capacity incidents are often predictable.
Use trend analysis rather than waiting for an allocation failure.
13.77 Capacity trend
Track:
Month 1 2 TB Month 2 2.3 TB Month 3 2.7 TB Month 4 3.2 TB
Calculate:
- absolute growth;
- percentage growth;
- growth by space;
- growth by object;
- forecast exhaustion date.
This is an architecture responsibility, not merely an operations task.
13.78 Monitoring concurrency
Monitor overlapping schedules.
Example:
01:00 Finance 01:05 Procurement 01:10 Sales 01:15 Inventory
If all workloads peak together, resource contention can occur.
Stagger workloads where business freshness permits.
Current SAP documentation recommends considering staggered scheduling when many tasks compete for resources.
13.79 Schedule design
Avoid:
every important task at 00:00
Prefer:
dependency-aware windows
Example:
Source replication | +30m | Transformation | +15m | Persistence | +10m | Consumption-ready
The exact cadence must be based on business requirements and observed runtime.
13.80 Monitoring automation
Where APIs or command-line capabilities are appropriate, operational monitoring can be integrated with external tooling.
Current SAP documentation exposes task-management capabilities through REST APIs and command-line interfaces for supported task operations.
Use automation for:
- status collection;
- operational reporting;
- controlled task execution;
- log retrieval;
- notification;
- integration with incident platforms.
Do not automate destructive recovery without guardrails.