Skip to main content

Phase 11 — Data Flows, Replication & Task Chains

SAP Datasphere data movement and orchestration architecture

11.1 What this phase solves​

A Datasphere model becomes an enterprise data product only when data reaches the right place with predictable correctness, freshness, recoverability, security, and cost.

This phase turns the modeling knowledge from Phases 01–10 into data engineering. You will learn how to reason about Data Flows, Replication Flows, Transformation Flows, and Task Chains as one operating architecture.

CapabilityPrimary purposeTypical outcome
Data FlowMove and transform source data into a local tableLocal persisted dataset
Replication FlowReplicate supported data assets with strong connectivity and delta capabilitiesSynchronized target
Transformation FlowTransform local data and write the result to a local tableCurated persisted dataset
Task ChainOrchestrate data integration tasksRepeatable pipeline execution

SAP's current documentation positions Replication Flows and Transformation Flows as the preferred modern ways to load and write local tables when their capabilities fit the requirement. Data Flows remain relevant for supported scenarios and existing implementations.

Official SAP documentation:

11.2 The architect mental model​

Do not begin by asking which flow to click. Begin with the architecture questions:

  1. Where does the data originate?
  2. Is the source already available inside Datasphere?
  3. Do I need replication, transformation, or both?
  4. Is the target local or external?
  5. Is the workload initial, recurring, scheduled, or continuously delta-oriented?
  6. What freshness SLA is required?
  7. Where should transformation occur?
  8. How will failures be detected?
  9. How will a failed run be recovered?
  10. How will data correctness be reconciled?
  11. What happens when the source schema changes?
  12. What is the expected volume and concurrency?
  13. Who owns the pipeline?

Logical data path:

Source | v Acquisition / Replication | v Raw or Landing Persistence | v Transformation | v Curated Persistence | v Semantic / Analytical Layer | v Consumption

Operational control path:

Pipeline | +--> Schedule +--> Parameters +--> Dependencies +--> Monitoring +--> Logging +--> Alerts +--> Retry / Recovery +--> Reconciliation +--> Audit +--> Ownership

That second path is what turns a developer flow into an enterprise pipeline.

11.3 Data movement, transformation, and orchestration​

Data movement​

The question is: how do records move from A to B?

Examples include S/4HANA to Datasphere, BW to Datasphere, files to Datasphere, and Datasphere local table to another supported target.

Transformation​

The question is: how does the data shape or meaning change?

Examples include joins, standardization, cleansing, derivation, aggregation, and conformance.

Orchestration​

The question is: in what order should independent and dependent tasks execute?

Replicate Customer | +----------------+ | | v v Replicate Sales Replicate Material | | +--------+-------+ | v Transform Sales | v Validate | v Publish

Do not force all three responsibilities into one object.

11.4 Current capability positioning​

Current SAP documentation describes Replication Flows as offering improved connectivity and delta-load processing compared with Data Flows. Transformation Flows provide additional transformation capabilities and can use applicable HANA or Spark runtime options depending on the scenario.

A practical decision tree is:

Need to load data? | +-- External or SAP source? | | | +-- Need replication / delta? | | | +--> Replication Flow | +-- Data already local? | +-- Need transformation? | +--> Transformation Flow

Use Data Flow where its supported capabilities are the right fit. Do not automatically reproduce an older Data Flow pattern when a current Replication Flow plus Transformation Flow architecture is more appropriate.

11.5 Data Flow​

11.5.1 What a Data Flow is​

A Data Flow is a data integration object used to move and transform source data into a local Datasphere table.

Source objects | v Data Flow | | | | | +--> Calculation | +----> Filter / Projection +------> Join / Union | v Local Table

11.5.2 When a Data Flow makes sense​

Consider it when:

  • the source and target scenario is supported;
  • required transformations fit the Data Flow capability set;
  • an existing implementation is operationally sound;
  • migration to newer flow types does not justify the change;
  • the target is a local Datasphere table.

Do not choose it merely because it is familiar.

11.5.3 Data Flow as a legacy/current decision​

A Data Flow becomes a questionable abstraction when you need richer replication behavior, broad delta processing, clearer separation of acquisition and transformation, or capabilities provided by current Replication and Transformation Flows.

The architect question is not: can a Data Flow do this?

The architect question is: is Data Flow the most appropriate lifecycle and runtime abstraction for this requirement?

11.6 Replication Flow​

11.6.1 What a Replication Flow is​

A Replication Flow is primarily a data replication and synchronization mechanism. It is designed to copy supported data assets from source systems to supported targets and supports delta-oriented scenarios.

Source | | initial load | delta changes v Replication Flow | v Target

Think: preserve source data availability at the target.

Do not think: implement every business transformation during replication.

11.6.2 Why replication deserves its own layer​

A strong architecture can first replicate source-aligned datasets and then transform them:

S/4HANA | +--> Sales Header ----+ | | +--> Sales Item ------+--> Landing | | +--> Customer --------+ | +--> Material

Landing | v Transformation | v Curated | v Semantic

This separation improves restartability, observability, source reconciliation, reuse, troubleshooting, and change isolation.

11.6.3 Replication Flow source discipline​

Current SAP documentation supports multiple source and target combinations depending on connection type. ABAP-oriented scenarios can include extraction-enabled CDS views and appropriately keyed tables, subject to current prerequisites.

Do not generalize that every connected source is automatically replicable. Connectivity and replication capability are separate questions.

Always validate the current SAP support documentation and relevant SAP Notes for the exact source scenario before production implementation.

11.6.4 Initial load versus delta​

Initial load establishes the target baseline:

Source state at T0 | v Full extraction | v Target baseline

Delta processing then handles changes after the baseline:

Source changes after T0 | v Delta capture | v Replication | v Target changes

The delta mechanism is source-specific. ODP, SLT, connector-specific change capture, and other mechanisms must not be treated as interchangeable.

11.7 Delta correctness​

A first successful load does not prove that delta processing is correct.

Validate at least:

  • insert;
  • update;
  • delete where supported;
  • late-arriving change;
  • repeated execution;
  • restart after failure;
  • source outage;
  • target maintenance;
  • reconciliation after recovery.

Delta is a business correctness problem.

Example:

Order 100 = 1,000 later Order 100 = 1,200

If the target treats both values as additive facts, analytics can incorrectly produce 2,200 instead of the intended current value 1,200.

Every delta design therefore needs an explicit semantic contract:

  • append;
  • update;
  • delete;
  • upsert;
  • snapshot;
  • history.

11.8 Replication projections and column selection​

Projection and mapping capabilities can reduce unnecessary data movement.

Instead of:

300 source columns --> target

you may need:

300 source columns | v Replication projection | v 40 required columns

Benefits include lower data movement, smaller targets, less downstream processing, and clearer contracts.

Do not remove a column simply because the current consumer does not use it. First decide whether it belongs to the enterprise source contract.

11.9 Replication parallelism​

Current Replication Flow documentation exposes source and target thread-limit settings for applicable flows. These settings influence parallel processing, especially during initial loads.

More parallelism does not automatically mean more throughput.

Approximate throughput is constrained by the minimum of:

source extraction network runtime target write partition parallelism

If the source can deliver only 200 MB/s, increasing target parallelism cannot create 500 MB/s of source capacity.

Measure before tuning.

11.10 Transformation Flow​

11.10.1 What a Transformation Flow is​

A Transformation Flow loads data from one or more supported local sources, applies transformations, and writes the result to a target local table.

Local sources | v Transformation Flow | +--> Graphical View transform +--> SQL View transform +--> other supported transformations | v Target local table

11.10.2 Transformation Flow versus Graphical View​

A Graphical View primarily describes modeling and query logic.

A Transformation Flow describes an execution and loading process that writes transformed data to a target table.

Therefore:

Graphical View = modeling / query logic Transformation Flow = execution / loading process

They can be used together.

11.11 Transformation Flow load types​

Transformation Flows support initial-only and, for eligible scenarios, initial-and-delta patterns.

Initial only:

Source --> Full read --> Target

Initial and delta:

First run --> Full baseline --> Target Later runs --> Delta --> Target

The important question is not whether the UI says delta. The important question is what makes a source row eligible for delta processing and whether the source and target are configured accordingly.

11.12 Transformation runtime​

Current SAP documentation identifies HANA and applicable Spark runtime choices depending on the space and storage scenario.

Do not choose Spark because Spark is fashionable.

Choose the runtime based on:

  • supported transformation features;
  • data location;
  • scale;
  • transformation complexity;
  • runtime behavior;
  • operational requirements;
  • skills;
  • cost.

HANA runtime​

Typically attractive for relational transformations and HANA-oriented processing.

Spark runtime​

Potentially attractive for distributed processing and applicable large-scale transformation scenarios.

This is not a universal performance ranking. Measure the actual workload.

11.13 Transformation design rules​

Rule 1 — Define grain first​

Example:

Sales Item = one order item Customer = one customer

The join is then reasoned about at a known grain.

Rule 2 — Control row multiplication​

If one side has multiple rows per key, a join can multiply records. Never use DISTINCT as a blind repair.

Rule 3 — Project deliberately​

Remove columns that are definitely unnecessary for the next contract.

Rule 4 — Filter early when semantically safe​

Reducing rows before expensive operations can reduce work.

Rule 5 — Avoid accidental many-to-many joins​

Document expected cardinality before implementation.

Rule 6 — Reconcile the target​

Flow completion is not data correctness.

11.14 Modern two-stage ingestion pattern​

A common enterprise pattern is:

SOURCE SYSTEM | v Replication Flow | v LANDING TABLE | v Transformation Flow | v CURATED TABLE | v Semantic Model | v Analytic Model | v SAC / Consumption

This separates source synchronization from business transformation and analytical semantics.

11.15 Why landing persistence can be valuable​

Landing persistence is not mandatory.

Use it when it provides operational or performance value:

  • replay;
  • reconciliation;
  • source-load isolation;
  • decoupling;
  • reusable source-aligned data;
  • recovery after downstream failure.

Costs include storage, additional processing, freshness latency, and lifecycle management.

Architectural rule: persist because the boundary provides value, not because every pipeline needs another table.

11.16 Data Flow versus Replication Flow versus Transformation Flow​

QuestionData FlowReplication FlowTransformation Flow
Primary purposeMove + transformReplicateTransform + load
External source acquisitionSupported scenariosStrong focusNot primary purpose
Delta capabilityScenario-dependentStrong focusSupported for eligible local sources
Transformation richnessModerateProjection/mapping orientedTransformation focused
Local targetYesSupported scenariosYes
Source already localPossibleUsually not the reason to choose itStrong fit
OrchestrationSupported scheduling/task patternsSupported scheduling/task patternsSupported scheduling/task patterns

Always check the current SAP support matrix for exact source and target combinations.

11.17 Task Chains​

A Task Chain is an orchestration object. It coordinates data integration tasks according to dependencies and supported execution semantics.

Start | v Replication | v Transformation | v Validation | v Publish

A Task Chain is not a transformation engine.

11.18 Sequential orchestration​

Use sequential dependencies when downstream work depends on upstream success.

Customer Replication | v Sales Replication | v Sales Transformation | v Analytical Publish

Examples include referential completeness, source readiness, and transformations that consume upstream output.

11.19 Parallel orchestration​

Use parallel branches when tasks are genuinely independent.

Start | +-------+-------+ | | v v Customer Material Replication Replication | | +-------+-------+ | v Sales Transformation

Current SAP documentation supports parallel task-chain branches. Parallelism should reduce elapsed time without creating resource contention or violating dependencies.

11.20 ANY versus ALL completion semantics​

When multiple tasks run in parallel, the continuation condition matters.

ALL means every required branch must satisfy the continuation condition.

ANY means the chain can continue when the configured qualifying branch condition is satisfied.

Choose deliberately. A data pipeline must never publish incomplete business data because a parallel-branch condition was modeled incorrectly.

11.21 Task-chain parameters​

Task chains can pass input parameters to supported tasks.

Example parameter contract:

Business Date Load Window Region Source System

Then:

Task Chain | +--> Replication / supported task +--> Transformation Flow +--> Validation

Rules:

  1. define ownership;
  2. use stable technical names;
  3. document defaults;
  4. avoid hidden dependencies;
  5. test scheduled execution;
  6. test manual execution;
  7. test retry behavior.

11.22 Scheduling architecture​

A schedule is part of the operating model.

A weak design is to start everything at the same clock time.

A stronger design models dependencies:

Source ready | v Replication complete | v Transformation | v Quality gate | v Publish

Clock schedules remain useful, but dependency-aware execution is usually more robust.

11.23 Continuous versus scheduled replication​

Applicable replication configurations can process delta data continuously or at scheduled times.

Continuous pattern:

Start -> Check delta -> Process -> Wait -> Check again

Scheduled pattern:

Schedule -> Process available delta -> Complete

Choose based on freshness SLA, source change rate, connection capacity, downstream dependency model, and operational simplicity.

Continuous execution is not automatically better.

11.24 Freshness engineering​

Define freshness as an end-to-end property.

Let:

  • S = source waiting time;
  • R = replication duration;
  • T = transformation duration;
  • Q = validation duration;
  • P = publishing duration.

Then:

End-to-end latency = S + R + T + Q + P

If the business requires 30 minutes and the pipeline consumes 20 + 15 + 5 + 5 minutes, the design violates the SLA.

The solution is architectural, not merely operational.

11.25 Failure domains​

LayerExample failureRecovery focus
SourceS/4 unavailableSource recovery / retry
ConnectionAuthentication failureConnection repair
ReplicationDelta failureFlow/object recovery
LandingTarget unavailableTarget/runtime recovery
TransformationModel errorCorrect logic and redeploy
Task ChainDependency failureRetry / branch recovery
ConsumptionAnalytical issueSemantic correction

Separate failure domains so operators know where to act.

11.26 Idempotency​

A production pipeline should answer: what happens if the same operation runs twice?

Example:

Run 1 --> 1,000 rows Run 2 --> same logical input

The target should follow the intended semantics rather than accidentally creating duplicates.

Common strategies include deterministic keys, merge/upsert semantics, controlled replace-and-reload, partition overwrite, source change markers, and explicit history models.

Idempotency is particularly important for retries.

11.27 Retry architecture​

Use:

Task failed | v Safe to retry? | +--> Yes --> Retry | +--> No --> Repair / Reinitialize / Reconcile

Typical classification:

FailureTypical action
Temporary source outageRetry
Temporary connection issueRepair then retry
Deterministic SQL/model errorFix design
Schema incompatibilityAssess contract and impact
Corrupt delta stateFollow supported recovery/reinitialization procedure
Data-quality failureCorrect or quarantine data

Never use infinite retries for deterministic failures.

11.28 Replication recovery and source maintenance​

Current SAP documentation describes pause/resume and stop behavior for applicable replication-flow configurations. The exact runtime-state behavior depends on configuration.

Before source maintenance:

Identify affected flows | v Pause / stop according to runbook | v Source maintenance | v Source validation | v Resume | v Reconcile

Never reduce a maintenance runbook to 'stop the interface'. Document the exact flow, objects, state behavior, resume process, and reconciliation.

11.29 Schema evolution​

Common changes include:

  • column added;
  • column removed;
  • data type changed;
  • key changed;
  • field renamed;
  • extraction artifact changed.

Model the change decision:

Schema change | v Compatible? | | Yes No | | v v Apply Impact analysis | v Change pipeline

Some current Replication Flow scenarios support controlled schema-change handling. Availability depends on source and target configuration. Never assume automatic schema evolution is universal.

11.30 Reconciliation​

Pipeline success is not equivalent to data correctness.

Use several control dimensions:

ControlExample
CountSource count versus target count
AmountSource sum versus target sum
KeysSource keys minus target keys
ChangesExpected changes versus applied changes
FreshnessLatest target timestamp versus SLA
NullsCritical-field null count
DuplicatesBusiness-key duplicate count

Control totals are especially valuable for finance and procurement.

11.31 Monitoring architecture​

Monitor at three levels:

Enterprise Pipeline | +--> Task Chain | +--> Flow | +--> Object | +--> Data / Error detail

Track at least:

  • run status;
  • duration;
  • rows processed;
  • rows changed;
  • error count;
  • last successful run;
  • data freshness;
  • throughput;
  • retry count;
  • failure frequency.

Trend these metrics. A single successful run proves little.

11.32 Operational SLOs​

Define measurable objectives.

SLOExample target
Daily pipeline success99.5 percent
Freshnessless than 30 minutes
Recovery timeless than 60 minutes
Critical reconciliation variance0 unexplained
Failure alert latencyless than 5 minutes

These values are examples, not SAP defaults.

11.33 Performance engineering​

Analyze the entire chain:

Source | Extraction | Network | Replication | Persistence | Transformation | Target write | Consumption

Find the bottleneck before changing settings.

Blind parallelism​

More threads can help initial-load throughput, but excessive concurrency can overload source systems, networks, connection pools, runtimes, or targets.

Test controlled levels and record duration, throughput, source impact, and target impact.

Unnecessary volume​

Ask whether every row and every column is actually required.

Transformation before reduction​

Where semantically safe, reduce rows before expensive joins and aggregations.

Unnecessary materialization​

Persist when reuse, recovery, source isolation, or performance justifies it. Do not create layers automatically.

11.34 Security​

Separate these questions:

  • who can design the flow;
  • who can run it;
  • who can monitor it;
  • who can access the source;
  • who can access the target;
  • who can see sensitive data;
  • who can manage schedules;
  • who administers the space.

Current SAP documentation distinguishes modeling and data-integration privileges. Use scoped roles and least privilege.

Sensitive data can be moved at high volume by a pipeline. Security therefore starts at acquisition, not only at the analytical model.

11.35 Space architecture​

A possible enterprise pattern is:

INGESTION SPACE Replication

PREPARATION SPACE Transformation

SEMANTIC SPACE Reusable models

CONSUMPTION SPACE Analytics

Do not create spaces merely for visual organization. Use boundaries where they provide meaningful security, ownership, lifecycle, workload, cost, or data-product separation.

11.36 Cross-space sharing​

A shared dataset is an interface.

Define:

  • owning space;
  • sharing authority;
  • change ownership;
  • security behavior;
  • consumer notification;
  • compatibility expectations.

Treat important shared datasets like APIs: stable contract, explicit owner, controlled change.

11.37 Naming and ownership​

Example naming convention:

RF_S4_SALES_HEADER RF_S4_SALES_ITEM TF_SALES_CURATED TC_DAILY_SALES_LOAD

A generic pattern is:

OBJECT_TYPE_SOURCE_OR_DOMAIN_PURPOSE

Every production pipeline should also have:

  • business owner;
  • technical owner;
  • support owner;
  • escalation path;
  • SLA;
  • criticality;
  • source dependency;
  • downstream dependencies.

A flow without an owner is an operational risk.

11.38 Development lifecycle​

Use an environment-aware lifecycle:

DEV -> TEST -> QA -> PROD

Validate at promotion:

  • object dependencies;
  • connections;
  • parameters;
  • schedules;
  • privileges;
  • target mappings;
  • runtime settings;
  • monitoring;
  • alerts.

Treat complex flow definitions as code-like assets even when designed graphically.

11.39 Transport, versioning, and rollback​

A production release should follow:

Change | v Review | v Test | v Transport | v Deploy | v Validate | v Monitor

Before production change, capture the known-good version, dependencies, and recovery plan.

Rollback may require restoring an earlier object version, reverting a transported design, redeploying a known-good version, or reinitializing data according to the supported procedure.

Rollback is not simply 'undo in the editor'.

11.40 Lineage and impact​

A mature pipeline should answer:

Where did this field come from?

and:

What breaks if this source field changes?

Example dependency chain:

S/4 CDS | v Replication Flow | v Landing Table | v Transformation Flow | v Curated Table | v Graphical View | v Analytic Model | v SAC

Use lineage and impact analysis during design and change review.

11.41 Troubleshooting methodology​

Use a fixed sequence rather than random setting changes.

Step 1 — Identify the failing layer​

Source, connection, flow, target, transformation, orchestration, or consumer.

Step 2 — Establish the last known good run​

Ask when it last succeeded and what changed afterward.

Step 3 — Inspect runtime evidence​

Check status, timestamps, processed volume, errors, warnings, and object-level results.

Step 4 — Reproduce safely​

Use simulation or controlled execution where supported.

Step 5 — Isolate​

Reduce the problem to source, object, operation, and target.

Step 6 — Correct​

Fix configuration or logic.

Step 7 — Re-run​

Use a controlled retry.

Step 8 — Reconcile​

Prove the business result is correct.

11.42 Troubleshooting: zero rows​

Check:

  1. source actually contains data;
  2. source filter;
  3. projection mapping;
  4. join conditions;
  5. parameter values;
  6. date/time window;
  7. delta state;
  8. target mapping;
  9. source authorization;
  10. whether the source is supported for the selected flow.

Execution status 'completed' does not mean the business result is correct.

11.43 Troubleshooting: duplicate rows​

Check source duplicates, business keys, one-to-many joins, many-to-many joins, retry behavior, delta semantics, task-chain re-execution, and target merge behavior.

Do not immediately add DISTINCT. Find the multiplication source.

11.44 Troubleshooting: missing delta records​

Trace the record through the chain:

Missing record | +--> Exists in source? +--> Eligible for delta? +--> Captured by source mechanism? +--> Received by flow? +--> Applied to target? +--> Visible downstream?

This converts a vague problem into a traceable problem.

11.45 Troubleshooting: slow initial load​

Measure source extraction, source query behavior, network, thread settings, target writes, concurrent workloads, partitioning, storage, and transformation complexity.

Do not assume the flow engine is the bottleneck.

11.46 Troubleshooting: task chain appears stuck​

Check:

  • upstream task still running;
  • upstream task failed;
  • dependency condition;
  • parallel branch completion;
  • long-running replication;
  • cancellation state;
  • schedule overlap;
  • source outage.

Determine whether the chain is actually stuck or correctly waiting.

11.47 Production pattern — SAP source to analytics​

S/4HANA | v Replication Flow | v Source-aligned tables | v Transformation Flow | v Conformed tables | v Semantic / Graphical | v Analytic Model | v SAC

Task-chain orchestration:

Start | +--> Replicate Customer --+ +--> Replicate Material --+--> Transform --> Validate --> Publish +--> Replicate Sales -----+

Use parallel branches only when the dependencies are genuinely independent.

11.48 Production pattern — multiple domains​

Enterprise | +--> Finance | +--> pipelines | +--> Procurement | +--> pipelines | +--> Sales +--> pipelines | v Shared semantic products | v Consumption

Do not build one giant task chain for the entire enterprise. Prefer domain ownership and smaller dependency graphs.

11.49 Production pattern — late-arriving source​

Suppose Procurement arrives before Material Master.

Do not automatically fail Procurement if Material is not required for the landing step.

Possible designs:

Procurement Landing | v Wait for Material readiness | v Curate

or dependency-driven orchestration where the transform begins only after required inputs are ready.

Choose based on business SLA and data dependency.

11.50 Architect exercise — Procurement​

Design a pipeline for:

  • 20 million purchase-order items per day;
  • S/4HANA source;
  • hourly freshness;
  • supplier and material master;
  • purchase-order header and item;
  • deletion handling;
  • reconciliation;
  • SAC consumption.

Your design must specify:

  1. source acquisition;
  2. replication;
  3. landing;
  4. transformation;
  5. task-chain dependencies;
  6. parallelism;
  7. reconciliation;
  8. failure recovery;
  9. security;
  10. monitoring;
  11. SLA.

Explain why each boundary exists.

11.51 Architect exercise — Finance​

Design:

FI source | v Replication | v Ledger landing | v Conformance | v Financial semantic layer | v Analytical model

Requirements include daily close, late postings, reconciliation, auditability, historical reporting, and restricted access.

Explain delta strategy, period-close strategy, restartability, control totals, and audit evidence.

11.52 Architect exercise — 100x volume increase​

Today:

10 million rows/day

Tomorrow:

1 billion rows/day

Do not immediately increase thread limits.

Re-evaluate source extraction, replication strategy, partitioning, target storage, transformation runtime, retention, incremental processing, task concurrency, workload isolation, downstream consumption, and cost.

The correct architecture at 1 billion rows may not be a 100x-scaled version of today's architecture.

11.53 Common anti-patterns​

One giant flow​

Hard to debug, reuse, own, and change.

One giant task chain​

Creates an opaque dependency graph and large blast radius.

Business logic inside replication​

Couples acquisition to business semantics and source change.

All transformation at consumption time​

Can cause repeated expensive computation and poor reuse.

Blind retries​

Can duplicate data or repeatedly overload a failing source.

Green status equals correctness​

Execution success does not prove business correctness.

Maximum parallelism​

Can create source and target contention.

No reconciliation​

Allows silent data loss.

11.54 Hands-on Lab 1 — First Data Flow​

Build:

Source -> Projection -> Filter -> Local Target

Tasks:

  1. create the flow;
  2. add source;
  3. project required fields;
  4. filter records;
  5. map target;
  6. save;
  7. deploy;
  8. run;
  9. inspect output;
  10. reconcile counts.

11.55 Hands-on Lab 2 — Replication Flow​

Build a replication scenario using a supported source.

Validate initial load, target keys, row counts, column mapping, run status, and runtime evidence.

Document source, target, load type, delta design, and ownership.

11.56 Hands-on Lab 3 — Delta test​

Test:

ChangeSource actionExpected targetActual target
InsertNew business keyNew row
UpdateExisting key changedNew state
DeleteSource record removedConfigured delete behavior

Repeat the test after a controlled restart.

11.57 Hands-on Lab 4 — Transformation Flow​

Build:

Landing A + Landing B | v Transformation | v Curated Target

Include projection, filter, join, calculated field, and aggregation.

Validate target grain before and after each major operation.

11.58 Hands-on Lab 5 — Task Chain​

Build:

Start | +--> Source A +--> Source B | +----> Transform | v Validate

Make A and B parallel only if genuinely independent.

Compare sequential and parallel elapsed time and explain the result.

11.59 Hands-on Lab 6 — Failure injection​

Create a controlled failure in a test environment.

Practice:

  1. identify;
  2. isolate;
  3. recover;
  4. reconcile;
  5. document root cause.

11.60 Hands-on Lab 7 — Production reconciliation​

Create a control record containing run ID, source count, target count, source amount, target amount, distinct keys, error count, start time, end time, freshness timestamp, and reconciliation status.

Use it as the foundation of an operational dashboard.

11.61 Hands-on Lab 8 — Performance experiment​

Run the same workload with controlled parallelism.

SettingDurationThroughputSource impactTarget impact
Low
Medium
High

Find the point of diminishing returns.

11.62 Hands-on Lab 9 — Schema evolution​

In a test pipeline:

  1. add a compatible source column;
  2. observe behavior;
  3. assess target;
  4. update the pipeline if required;
  5. deploy;
  6. validate;
  7. document.

Repeat with a breaking change and compare the recovery process.

11.63 Hands-on Lab 10 — Enterprise pipeline review​

Review one pipeline against correctness, freshness, performance, security, recovery, observability, ownership, lifecycle, and cost.

Score each dimension from 1 to 5. Anything below 4 requires an architectural action.

11.64 Interview preparation​

Data Flow versus Replication Flow?​

Data Flow combines movement and transformation for supported scenarios. Replication Flow is primarily designed for copying and synchronizing supported data assets with stronger replication and delta capabilities.

Why separate replication and transformation?​

To decouple source synchronization from business transformation, improving restartability, observability, reuse, and change isolation.

When would you use a Transformation Flow?​

When suitable data is already available in local sources and the requirement is to transform and load a local target table.

Why use Task Chains?​

To orchestrate multiple data integration tasks using dependencies and, where appropriate, parallel branches.

Does a green flow guarantee correct data?​

No. Reconciliation and business validation are required.

How do you optimize replication?​

Find the bottleneck first: source, network, runtime, target, concurrency, or partitioning.

How do you handle schema changes?​

Assess compatibility, impact, source/target contracts, deployment requirements, and downstream consumers.

Why not one enterprise task chain?​

Blast radius, maintainability, ownership, and dependency complexity.

Why not maximum parallelism?​

Because the bottleneck may be upstream or downstream and excessive concurrency can increase contention.

What makes a pipeline production-ready?​

Correctness, SLA, monitoring, recovery, reconciliation, security, ownership, lifecycle management, and documented operations.

11.65 Architect decision framework​

DecisionAnswer
Source
Source type
Target
Freshness
Initial load
Delta strategy
Replication Flow?
Transformation Flow?
Data Flow?
Task Chain?
Runtime
Persistence boundaries
Expected volume
Parallelism
Reconciliation
Security
Monitoring
Recovery
Owner
SLA
Cost considerations

If you cannot complete this table, the architecture is not ready.

11.66 Production readiness checklist​

Functional​

  • Source validated
  • Target validated
  • Grain documented
  • Mapping documented
  • Initial load tested
  • Delta tested
  • Deletes tested where applicable
  • Reconciliation implemented

Performance​

  • Volume estimated
  • Throughput measured
  • Bottleneck identified
  • Parallelism tested
  • Transformation complexity reviewed
  • Storage strategy reviewed

Reliability​

  • Retry behavior documented
  • Recovery procedure documented
  • Source outage procedure documented
  • Target outage procedure documented
  • Schema-change procedure documented
  • Reinitialization procedure documented where relevant

Operations​

  • Monitoring configured
  • Failure alerts configured
  • SLA defined
  • Freshness monitored
  • Ownership documented

Security​

  • Least privilege
  • Source authorization reviewed
  • Target authorization reviewed
  • Sensitive-data handling reviewed
  • Space boundaries reviewed

Lifecycle​

  • DEV/TEST/PROD strategy
  • Transport strategy
  • Versioning
  • Rollback plan
  • Dependency review
  • Change approval

11.67 Architect takeaway​

The key lesson is not how to create a flow. It is how to design a reliable data movement system.

REPLICATE | v PERSIST | v TRANSFORM | v VALIDATE | v SEMANTIC MODEL | v CONSUME

Surround the pipeline with security, observability, reconciliation, recovery, governance, ownership, and cost control.

The architect's decision is not which flow should I use?

It is: what data movement, transformation, orchestration, and operational architecture produces the required business outcome with predictable correctness, freshness, recoverability, security, and cost?

That is the foundation for the enterprise Datasphere architecture phases that follow.

Official SAP documentation anchors​