Skip to main content

13.5 — Incident Response & Troubleshooting

13.40 Error taxonomy​

Use a consistent taxonomy:

Connectivity​

Cannot reach source

Authentication​

Credential or token failure

Authorization​

User/service lacks required privilege

Schema​

Column/table/type changed

Data quality​

Invalid or unexpected data

Runtime​

Memory, timeout, statement limit, resource issue

Modeling​

Invalid logic or dependency

Scheduling​

Task did not run when expected

Orchestration​

Dependency or chain failure

Consumption​

Model or client behavior problem

13.41 Connectivity incident​

Symptoms:

  • connection test fails;
  • replication cannot start;
  • remote queries fail;
  • agent is disconnected.

Investigate:

  1. Datasphere connection status.
  2. Source availability.
  3. Network path.
  4. Agent status if applicable.
  5. Credentials.
  6. Recent firewall or certificate changes.
  7. Source maintenance.

Recovery should restore the smallest broken boundary first.

13.42 Authentication incident​

Symptoms:

  • expired credential;
  • invalid client;
  • token failure;
  • authentication rejection.

Do not immediately rebuild the connection.

Check:

  • credential expiry;
  • authentication method;
  • technical-user status;
  • secret rotation;
  • certificate validity;
  • OAuth configuration where applicable.

After repair, test connectivity before restarting production loads.

13.43 Authorization incident​

Symptoms:

  • object visible but execution fails;
  • source access denied;
  • task cannot be run;
  • monitoring details are unavailable.

Separate:

  • global role;
  • scoped space role;
  • object privilege;
  • runtime privilege;
  • source-system authorization.

Current SAP documentation defines distinct privileges for monitoring and executing different task types. Use least privilege rather than broad administrator access.

13.44 Schema-change incident​

Example:

Source column: CUSTOMER_ID INTEGER

becomes:

CUSTOMER_ID VARCHAR

or a source column is renamed.

Potential impact:

  • replication;
  • transformations;
  • joins;
  • calculated columns;
  • semantic models;
  • downstream consumption.

Recovery:

  1. Identify exact schema change.
  2. Determine compatibility.
  3. Update the correct contract.
  4. Validate dependent objects.
  5. Reconcile data.
  6. Document the change.

13.45 Duplicate-data incident​

Symptoms:

  • row count doubled;
  • duplicate business keys;
  • repeated financial amounts.

Possible causes:

  • non-idempotent retry;
  • duplicate source extraction;
  • incorrect join;
  • incorrect incremental watermark;
  • task executed twice;
  • late-arriving data mishandled.

Do not use DISTINCT as the first fix.

First identify why duplicates exist.

13.46 Missing-data incident​

Symptoms:

  • expected records absent;
  • one date missing;
  • one company code missing;
  • target count lower than expected.

Use:

Consumer | v Target | v Transformation | v Acquisition | v Source

Find the first boundary where expected records disappear.

Data Pipeline Analyzer can accelerate this dependency-oriented investigation.

13.47 Wrong-value incident​

When values are wrong but records exist, investigate:

  • join duplication;
  • filter;
  • currency conversion;
  • unit conversion;
  • aggregation;
  • semantic measure definition;
  • source correction;
  • hierarchy mapping.

A row-count check alone will not detect this class of defect.

13.48 Slow-dashboard incident​

Use this sequence:

  1. Confirm whether all users are affected.
  2. Test the same analytic model directly.
  3. Inspect Runtime Metrics.
  4. Inspect View Analyzer.
  5. Check federation and Remote Queries if remote data is involved.
  6. Check persistence state.
  7. Compare current data volume with baseline.
  8. Check concurrency.
  9. Validate recent model changes.

Do not begin by rebuilding the dashboard.

13.49 Slow-pipeline incident​

Check:

  • source latency;
  • row volume;
  • column width;
  • transformation complexity;
  • joins;
  • aggregation;
  • persistence;
  • partitioning;
  • schedule overlap;
  • source throttling.

Compare:

normal runtime versus current runtime

Then identify the first stage whose runtime changed materially.

13.50 Stale-data incident​

A stale dataset can have a green pipeline.

Possible causes:

  • source stopped changing;
  • schedule disabled;
  • replication paused;
  • task completed against an unchanged source;
  • persistence refresh failed;
  • watermark did not advance;
  • downstream view remains persisted from an earlier run.

Always inspect the data watermark, not only status.

13.51 Alerting philosophy​

Alerts should be actionable.

Bad alert:

"Task failed."

Better:

"Finance daily sales dataset has exceeded its 60-minute freshness SLO; latest successful watermark is 08:42; task chain FIN_DAILY_LOAD failed at persistence step."

A good alert contains:

  • object;
  • symptom;
  • threshold;
  • timestamp;
  • impact;
  • likely boundary;
  • runbook reference.

13.52 Alert thresholds​

Use business-aware thresholds.

Examples:

Freshness > 60 min Failure count >= 2 Runtime > 2x baseline Row count deviation > 15% Reconciliation difference != 0

Do not create hundreds of low-value alerts.

13.53 Alert fatigue​

If operators ignore alerts, monitoring has failed.

Reduce:

  • duplicate alerts;
  • transient noise;
  • informational notifications;
  • alerts without owners;
  • alerts without recovery actions.

Prefer:

condition + owner + action + escalation

13.54 SLO design​

An SLO is a target for service reliability.

Example:

SLOTarget
Daily finance freshness99.5% within 60 min
Pipeline success99%
Critical analytic availability99.9%
Reconciliation accuracy100% for controlled totals

These are examples, not universal SAP defaults.

Define targets with business owners.

13.55 SLA versus SLO​

An SLA is a commitment, often contractual.

An SLO is an internal reliability objective.

An SLI is the measured indicator.

Relationship:

SLI = measurement SLO = target SLA = external commitment

Example:

SLI: dataset age SLO: age less than or equal to 60 minutes SLA: finance reporting available by 09:00

13.56 Monitoring scorecard​

A production scorecard can contain:

KPICurrentTargetStatus
Freshness
Pipeline success
Failed tasks
Mean runtime
P95 runtime
Reconciliation failures
Data-quality failures
Remote-source latency
Storage growth

The scorecard should be reviewed regularly.

13.57 Mean time to detect​

MTTD measures:

incident detected time​

incident start time

Lower is generally better.

Improve MTTD with:

  • freshness monitoring;
  • failure alerts;
  • quality gates;
  • clear ownership;
  • operational dashboards.

13.58 Mean time to recover​

MTTR measures the time required to restore service.

Reduce MTTR with:

  • runbooks;
  • safe retries;
  • clear ownership;
  • dependency maps;
  • known-good baselines;
  • automation;
  • rollback procedures.

13.59 Runbook design​

Every critical object should have a short runbook.

Minimum structure:

  1. Purpose.
  2. Owner.
  3. SLO.
  4. Dependencies.
  5. Monitoring location.
  6. Common failures.
  7. Safe recovery.
  8. Validation.
  9. Escalation.
  10. Prevention notes.

A runbook should be executable by an on-call engineer who did not build the object.

13.60 Example runbook — failed finance load​

Incident | v Check task chain | v Identify first failed child | v Open task log | v Classify error | +--> transient -> controlled retry | +--> source -> source escalation | +--> schema -> change review | +--> data quality -> quarantine/reconcile | +--> runtime -> performance investigation | v Validate watermark + totals | v Close incident