Skip to main content

18.4 Replication & Persistence

1. Why replicate​

Replication moves the runtime dependency from the source toward a local persisted representation.

Source
  |
replication
  |
local table
  |
semantic model
  |
consumer

This can improve:

  • source protection,
  • query consistency,
  • analytics performance,
  • recovery,
  • reuse.

2. Full versus delta​

A replication design must define:

  • initial load,
  • delta mechanism,
  • deletion handling,
  • restart behavior,
  • retention,
  • reconciliation.

SAP Datasphere replication flows support delta loads for supported source connections.

3. Freshness model​

Define freshness as a measurable contract.

Example:

Source commit
    |
    +-- extraction delay
    |
    +-- transfer delay
    |
    +-- target commit
    |
    +-- semantic availability
    |
consumer freshness

4. Persistence placement​

Do not persist every intermediate dataset.

Persist where it provides:

  • performance,
  • reuse,
  • isolation,
  • recovery,
  • governance,
  • historical value.

5. Storage optimization​

For large datasets evaluate:

  • unnecessary columns,
  • filters,
  • partitioning where supported,
  • in-memory acceleration where appropriate,
  • retention,
  • incremental processing.

SAP documentation explicitly recommends filtering unnecessary columns and data during acquisition and transformation.

6. Reconciliation​

At minimum compare:

  • source record count,
  • target record count,
  • business totals,
  • latest source timestamp,
  • latest target timestamp,
  • rejected records.

Exercise​

Design a daily 2 TB replication pipeline with a 15-minute freshness SLA.

Explain initial load, delta, recovery, reconciliation and cost controls.