Phase 03 — Data Integration Architecture
Architectural objective: learn how to decide how data should enter, leave, or be accessed by SAP Datasphere before building the individual pipelines.
Phase 02 established the platform and space boundaries.
This phase establishes the integration architecture.
The central question is not:
"Which button creates a connection?"
It is:
"What is the right data movement pattern for this workload?"
That distinction separates a Datasphere developer from a Datasphere architect.
3.1 What this phase teaches
By the end of this phase, you should be able to design integration for:
- SAP S/4HANA
- SAP ECC
- SAP BW
- SAP BW/4HANA
- SAP HANA
- SAP Datasphere
- SAP BW bridge
- SAP Business Data Cloud data products
- Oracle
- SQL Server
- Snowflake
- cloud object storage
- REST/OData sources
- files
- streaming/event sources
- other enterprise platforms
You will learn to decide between:
- federation
- remote tables
- replication
- replication flows
- data flows
- transformation flows
- task chains
- ODP-based extraction
- CDS-based extraction
- SAP-specific real-time patterns
- batch ingestion
- micro-batch ingestion
- file ingestion
- API ingestion
The detailed implementation of Data Flow, Replication Flow, Transformation Flow, and Task Chain is intentionally reserved for Phase 11.
Here, we focus on the architecture decision.
3.2 The integration problem
A real enterprise rarely has one source.
Consider:
ENTERPRISE
│
┌──────────────────────┼──────────────────────┐
│ │ │
SAP CORE NON-SAP CLOUD
│ │ │
S/4HANA Oracle Snowflake
ECC SQL Server S3
BW/4HANA Salesforce BigQuery
HANA Files APIs
│ │ │
└──────────────────────┼──────────────────────┘
▼
SAP DATASPHERE
│
┌──────────┼──────────┐
│ │ │
Federate Replicate Transform
│ │ │
└──────────┼──────────┘
▼
Data Models
│
▼
SAC / AFO / BI / APIs
The architecture must determine:
- where data lives
- where it is processed
- where it is persisted
- how often it moves
- who owns the connection
- how deltas are captured
- what happens when the source is unavailable
- how source-system load is controlled
- how failures are recovered
- how the integration scales
3.3 The first architectural decision: move or access?
Every integration architecture starts with two fundamental patterns.
Pattern A — Access remotely
Source
│
│ query
▼
Datasphere
│
▼
Consumer
Data remains at the source.
Datasphere accesses it when required.
This is the conceptual foundation of federation.
Pattern B — Move data
Source
│
│ extract / replicate
▼
Datasphere
│
▼
Persisted data
│
▼
Consumer
Datasphere owns a copy of the data.
This is the foundation of replication and ingestion.
3.4 Federation versus replication
| Dimension | Federation | Replication |
|---|---|---|
| Data copy | Usually no | Yes |
| Source dependency at query time | High | Lower |
| Storage in Datasphere | Low | Higher |
| Query latency | Depends on source/network | Usually more predictable |
| Historical independence | Limited | Strong |
| Source workload | Query-dependent | Load-dependent |
| Transformation before storage | Limited by pattern | Stronger |
| Resilience to source outage | Lower | Higher after successful load |
| Freshness | Potentially very high | Depends on replication frequency |
| Best use | Live access / selective data | Analytics / repeated access |
Neither pattern is universally better.
The architecture depends on workload.
3.5 The integration decision triangle
Think about three competing objectives:
FRESHNESS
▲
/ \
/ \
/ \
/ \
/ \
▼-----------▼
PERFORMANCE COST
Increasing freshness may require:
- real-time replication
- more frequent loads
- source-side change capture
- more compute
- more monitoring
Increasing performance may require:
- local persistence
- optimization
- pre-aggregation
- partitioning
- materialization
Reducing cost may favor:
- federation
- object-store staging
- lower-frequency loads
- selective persistence
The architect balances all three.
3.6 The eight integration questions
Before creating a connection, answer:
1. What is the source?
SAP, database, SaaS, file, API, stream, or another platform?
2. What is the latency requirement?
Seconds, minutes, hourly, daily, or on demand?
3. What is the volume?
Thousands, millions, billions, or continuously growing events?
4. Is delta extraction available?
Can the source identify inserts, updates, and deletes?
5. Does the data need transformation?
If yes, where should transformation occur?
6. Who owns the source?
Source team or Datasphere team?
7. What happens if the source is unavailable?
Can the business continue using the last persisted copy?
8. What is the cost of duplication?
Storage, compute, network, source load, and operational overhead?
These questions should be answered before implementation.
3.7 Datasphere integration architecture
A useful conceptual architecture is:
SOURCE SYSTEMS
│
┌──────────────┼──────────────┐
│ │ │
SAP NON-SAP CLOUD
│ │ │
└──────────────┼──────────────┘
│
CONNECTIONS
│
┌────────────┼────────────┐
│ │ │
FEDERATION REPLICATION INGESTION
│ │ │
│ ▼ │
│ LOCAL DATA │
│ │ │
└────────────┼────────────┘
▼
PREPARATION
│
▼
MODELING
│
▼
CONSUMPTION
A connection is the technical bridge.
It is not the integration architecture by itself.
3.8 Connection versus integration pattern
A connection answers:
"How can Datasphere communicate with this source?"
An integration pattern answers:
"What should Datasphere do with the data?"
For example:
Oracle Connection
│
├── Remote Table → Federation
│
├── Remote Table → Replication
│
├── Data Flow → Local Table
│
└── Replication Flow → Local Table
Therefore:
Never choose a connection type and assume the architecture is finished.
3.9 Current Datasphere integration toolbox
SAP Datasphere provides multiple acquisition and integration mechanisms.
The major architectural categories are:
| Mechanism | Primary purpose |
|---|---|
| Remote table | Access source data and, where supported, replicate it |
| Federation | Query remote source without maintaining a local analytical copy |
| Data Flow | Graphical ETL/data movement |
| Replication Flow | Efficient loading and replication, including delta scenarios where supported |
| Transformation Flow | Transform existing Datasphere data into target tables |
| Task Chain | Orchestrate integration tasks |
| File ingestion | Load file-oriented datasets |
| API/custom connection patterns | Integrate application-oriented sources |
| Data products | Consume governed data supplied through SAP Business Data Cloud |
SAP currently recommends replication flows and transformation flows over data flows for eligible loading scenarios because they provide stronger connectivity, delta processing, transformation capabilities, or recovery behavior.
3.10 Important current-state rule
A historical tutorial may teach:
"Use Data Flow for ETL."
That is incomplete for current Datasphere.
A better rule is:
Use the integration object whose capabilities match the workload, with Replication Flow and Transformation Flow preferred for eligible modern loading patterns.
Data Flow remains important and is not obsolete.
It is especially useful when its supported transformation operators and source/target capabilities fit the requirement.
3.11 Data Flow — architectural role
A Data Flow moves and transforms data through a graphical pipeline.
Typical pattern:
Source
│
▼
Data Flow
│
├── Projection
├── Filter
├── Join
├── Union
├── Calculated Column
├── Aggregation
└── Python Script
│
▼
Local Table
Data Flow is therefore an ETL-oriented mechanism.
Detailed node behavior belongs to Phase 11.
3.12 Replication Flow — architectural role
Replication Flow is designed for efficient movement and replication.
Conceptually:
Source
│
├── Initial Load
│
└── Delta Changes
│
▼
Replication Flow
│
▼
Target
Current SAP documentation highlights improved connectivity, delta load processing, parallelized initial loading, and restart/resume behavior compared with traditional Data Flow patterns.
This makes Replication Flow particularly important for production ingestion architectures.
3.13 Transformation Flow — architectural role
Transformation Flow is primarily a transformation layer between existing data objects.
Conceptually:
Local / Supported Source
│
▼
Transformation Flow
│
┌─────┼─────┐
│ │ │
Join Filter Aggregate
│ │ │
└─────┼─────┘
▼
Target
SAP documents Transformation Flows as a mechanism for loading data from one or more supported sources, applying transformations, and writing the result to a target table.
3.14 Task Chain — architectural role
A Task Chain is orchestration.
It should not be confused with a transformation engine.
Think:
Task Chain
│
├── Replication Flow
│
├── Transformation Flow
│
├── Data Flow
│
└── Other Tasks
Its purpose is sequencing and operational control.
Deep orchestration design comes in Phase 11 and Phase 20.
3.15 Remote tables
A remote table represents source data accessible through a connection.
Depending on the connection and configuration, remote tables can support:
- federation
- replication
- real-time replication in supported scenarios
SAP explicitly documents that remote tables can provide live access to connected source data or be used to copy data into Datasphere, depending on the supported connection features.
3.16 Remote table is not automatically federation
This distinction is critical.
REMOTE TABLE
│
├── Query remotely
│ ↓
│ Federation
│
└── Copy locally
↓
Replication
The object is similar.
The runtime behavior is different.
3.17 Federation architecture
A federated pattern looks like:
Datasphere
│
Remote Table
│
▼
Remote Source
│
▼
Database
When a query executes, part or all of the work may depend on the remote system and connectivity path.
Therefore federation should be evaluated against:
- source workload
- network latency
- query complexity
- pushdown
- concurrency
- source availability
- source governance
3.18 When federation is attractive
Federation is attractive when:
- data volume is large
- only a small subset is queried
- data must remain at source
- source is already highly available
- duplication is undesirable
- freshness is more important than local independence
- query patterns are controlled
Examples:
- current supplier status
- reference lookup
- low-volume operational data
- data that must remain authoritative in source
3.19 When federation becomes dangerous
Avoid blindly federating:
- high-concurrency dashboards
- large joins
- billion-row fact data
- complex transformations
- unstable source systems
- sources with strict workload limits
- critical reporting with weak network reliability
A query that is fast on a developer machine may become expensive with 500 concurrent users.
3.20 Replication architecture
A persisted integration pattern looks like:
SOURCE
│
│ initial load
▼
DATASPHERE
│
│ delta
▼
LOCAL TABLE
│
▼
MODELING
│
▼
CONSUMPTION
Benefits:
- predictable query performance
- source independence at query time
- local transformation
- local historical storage
- easier analytical optimization
- controlled freshness
Costs:
- storage
- data duplication
- integration monitoring
- reconciliation
- delta management
- operational ownership
3.21 Initial load versus delta
Replication is rarely just:
"Copy the table."
Production replication usually has two phases.
Initial load
Source
│
└──── full dataset ────► Target
Delta
Source Changes
│
├── Insert
├── Update
└── Delete
│
▼
Target
Delta support is one of the most important architecture questions.
3.22 Why delta matters
Suppose a source has 2 billion rows.
If the pipeline reloads everything every hour:
2 billion rows × 24 loads/day
This is usually an unacceptable architecture.
If only 500,000 rows change:
Initial load → 2 billion
Delta loads → 500,000/change window
The architecture becomes much more scalable.
The exact strategy depends on source capabilities.
3.23 Source-side change capture
Delta can be provided through different mechanisms:
- source-native change tracking
- timestamps
- change data capture
- database logs
- triggers
- ODP delta mechanisms
- connector-specific replication
Do not assume every source supports the same delta behavior.
3.24 SAP ODP architecture
For SAP application sources, Operational Data Provisioning (ODP) is a major extraction framework.
Conceptually:
SAP Application
│
▼
ODP Provider
│
▼
Operational Delta Queue
│
▼
ODP Consumer
SAP documentation describes ODP as infrastructure for data extraction and replication from ABAP-based applications, with the Operational Delta Queue supporting delta procedures.
3.25 ODP is not the same as Datasphere
This distinction matters.
ODP is an SAP data provisioning framework.
Datasphere is the data platform receiving, integrating, modeling, governing, and exposing data.
Think:
S/4HANA
│
▼
ODP / Extraction Framework
│
▼
Datasphere Integration
│
▼
Local / Remote Data
│
▼
Semantic Model
Do not describe ODP as "a Datasphere object."
3.26 CDS-based extraction
Modern SAP systems often expose business data through ABAP CDS.
A conceptual extraction path is:
S/4HANA
│
▼
ABAP CDS
│
▼
Extraction Enablement
│
▼
ODP
│
▼
Consumer
This is particularly important when the business object semantics are represented in CDS rather than requiring direct table-level extraction.
3.27 Why CDS matters to Datasphere architects
CDS can provide:
- business semantics
- associations
- reusable definitions
- extraction enablement
- standardized business meaning
For an architect, the question is not:
"Can I extract the table?"
It is:
"What is the supported business-level extraction interface?"
That often leads to a more maintainable integration architecture.
3.28 S/4HANA integration decision
When integrating S/4HANA, consider:
S/4HANA
│
├── CDS / ODP
│
├── Supported ABAP extraction
│
├── Replication pattern
│
└── Live / remote access
Then evaluate:
- business semantics
- delta support
- latency
- source impact
- extraction support
- required history
- transformation needs
Do not start with:
"Which table should I replicate?"
Start with:
"What is the supported source interface for this business requirement?"
3.29 S/4HANA source impact
Every extraction architecture affects the source.
Potential impacts include:
- CPU
- memory
- database I/O
- network
- application workload
- extraction jobs
- change capture
- API limits
A source owner should approve production integration.
3.30 The source-system contract
Every production integration should document:
| Contract item | Example |
|---|---|
| Source owner | S/4 Platform Team |
| Source object | Procurement CDS |
| Extraction mechanism | ODP |
| Refresh | 15 minutes |
| Delta | Supported |
| Expected volume | 50M rows |
| Business SLA | 30 minutes |
| Technical SLA | 15 minutes |
| Source window | Continuous |
| Failure policy | Retry + alert |
This turns an integration into an operational contract.
3.31 SAP HANA integration
SAP HANA sources can be integrated through multiple patterns depending on the HANA edition, connection type, network path, and required feature.
Conceptually:
SAP HANA
│
├── Remote access
│
├── Replication
│
├── Data Flow
│
└── Replication Flow
Do not automatically assume that a HANA connection means "live federation."
The supported feature set must be checked for the specific connection.
3.32 BW integration
BW and BW/4HANA can play several roles:
- historical source
- existing enterprise semantic layer
- transitional platform
- coexistence source
- migration source
- BW bridge integration point
The architecture depends on the target operating model.
Example:
BW/4HANA
│
▼
Datasphere
│
├── Reuse
├── Harmonize
└── Modernize
Phase 17 covers BW/4HANA and BW bridge deeply.
3.33 SAP BW bridge
BW bridge is not simply "BW inside Datasphere."
Think:
SAP Datasphere
│
├── Native Datasphere Modeling
│
└── BW bridge
│
├── BW-compatible modeling
├── SAP extraction patterns
└── Existing BW investment
The integration architecture must decide what remains in BW-compatible patterns and what should become native Datasphere content.
Phase 17 covers this in depth.
3.34 SAP Business Data Cloud data products
A modern SAP architecture may not require direct source-system integration for every dataset.
Governed data products can provide packaged business data.
Conceptually:
SAP Business Data Cloud
│
▼
Data Product
│
▼
Datasphere
│
▼
Enterprise Model
This can reduce custom integration effort when a suitable governed data product already exists.
However, availability, entitlement, semantic fit, freshness, and ownership must be evaluated.
3.35 Non-SAP databases
Typical enterprise sources include:
- Oracle
- Microsoft SQL Server
- PostgreSQL
- MySQL
- Snowflake
- Google BigQuery
- cloud storage
- Hadoop
- other JDBC/ODBC-capable platforms
The architecture should evaluate:
- federation support
- replication support
- delta support
- driver requirements
- network path
- source authentication
- source workload
- data volume
SAP's current connection catalog exposes different capabilities by connection type, so a source name alone is not enough to determine the correct pattern.
3.36 Cloud object storage
Object storage is useful for:
- files
- historical data
- large staging datasets
- data exchange
- archival workloads
- externally generated datasets
Conceptually:
Cloud Object Storage
│
▼
Replication / Ingestion
│
▼
File Space or Standard Space
│
▼
Preparation
File-space architecture is covered more deeply in Phase 02 and later in Phase 11.
3.37 APIs and SaaS
API-driven sources behave differently from databases.
Examples:
- REST APIs
- OData services
- SaaS applications
- partner systems
Architectural concerns include:
- pagination
- rate limits
- authentication
- incremental extraction
- retries
- idempotency
- schema drift
- API versioning
- error handling
A database pipeline and an API pipeline should not be designed identically.
3.38 API integration pattern
API
│
├── Authentication
├── Request
├── Pagination
├── Rate limiting
├── Retry
└── Response validation
│
▼
Raw staging
│
▼
Transform
│
▼
Curated
This is a pipeline architecture, not just a connection.
3.39 File integration
Files remain common in enterprise environments.
Typical sources:
- supplier files
- bank statements
- CSV extracts
- Excel files
- JSON
- XML
- Parquet
- operational exports
The architect should define:
- naming convention
- landing location
- schema contract
- encoding
- delimiter
- duplicate detection
- arrival detection
- late-arriving files
- quarantine
- retention
3.40 File ingestion anti-pattern
Avoid:
File arrives
↓
Developer manually uploads
↓
Dashboard works
This is a proof of concept.
Production should become:
Source
↓
Managed Landing
↓
Validation
↓
Ingestion
↓
Transformation
↓
Quality
↓
Curated Data
↓
Consumption
3.41 Streaming and event data
Not every workload belongs in a traditional batch pipeline.
Streaming is appropriate when events matter:
- IoT
- transactions
- operational telemetry
- clickstream
- machine events
- event-driven processes
The architecture may look like:
Event Producer
│
▼
Streaming Platform
│
▼
Streaming / Integration Layer
│
▼
Analytical Platform
Datasphere should not automatically become the event broker.
Use the right platform for event transport and use Datasphere where governed analytical integration and modeling are required.
3.42 Batch versus streaming
| Requirement | Batch | Streaming |
|---|---|---|
| Daily reporting | Strong fit | Usually unnecessary |
| Hourly analytics | Strong fit | Possible |
| Seconds-level monitoring | Weak | Strong |
| Huge historical loads | Strong | Usually unnecessary |
| Event-driven decisions | Weak | Strong |
| Operational simplicity | Strong | Lower |
| Cost predictability | Strong | More complex |
The latency requirement should drive the decision.
3.43 Real-time is not always better
Real-time introduces:
- higher operational complexity
- more monitoring
- tighter source dependency
- more failure modes
- potentially higher cost
- stronger source impact
If the business only needs:
"Data by 8 AM every morning"
then a real-time pipeline may be unnecessary.
Architects optimize for business value, not technical novelty.
3.44 Latency classification
Use a simple classification:
Class A — Live
Data must reflect source state at query time.
Class B — Near real-time
Seconds to minutes.
Class C — Frequent batch
Minutes to hours.
Class D — Scheduled batch
Hours to daily.
Class E — Historical
Daily, weekly, or on demand.
Example:
| Business requirement | Candidate architecture |
|---|---|
| Current inventory | Federation or frequent replication |
| Executive finance | Replicated curated data |
| Supplier reference | Federation or scheduled replication |
| Monthly close | Persisted analytical model |
| Historical archive | Low-cost persisted storage |
| Event monitoring | Streaming-oriented architecture |
3.45 Latency budget
Do not define latency only as:
"15 minutes."
Break it down:
Source Change
│
▼
Capture Delay
│
▼
Transfer Delay
│
▼
Load Delay
│
▼
Transformation Delay
│
▼
Model Availability
│
▼
Consumer Refresh
Total latency is the sum of the relevant stages.
3.46 Freshness SLA versus query SLA
These are different.
Freshness SLA
How old can the data be?
Example:
Data must be no older than 15 minutes.
Query SLA
How quickly must a dashboard respond?
Example:
Dashboard must respond within 5 seconds.
You can have:
- 15-minute freshness + 5-second queries
- live freshness + 20-second queries
The integration and modeling architecture changes accordingly.
3.47 Data ownership
Integration ownership should be explicit.
Source team
Owns:
- source availability
- source semantics
- source authorization
- extraction enablement
Datasphere integration team
Owns:
- connection
- ingestion
- replication
- monitoring
- pipeline operations
Data modeling team
Owns:
- harmonization
- semantic modeling
- business rules
Consumer team
Owns:
- dashboard/application requirements
This avoids the classic:
"The data is wrong, ask the other team."
3.48 Integration ownership matrix
| Activity | Source Team | Integration Team | Modeling Team |
|---|---|---|---|
| Source availability | A/R | C | I |
| Connection | C | A/R | I |
| Extraction enablement | A/R | C | I |
| Replication | C | A/R | I |
| Transformation | C | R | A/R |
| Semantic model | C | C | A/R |
| Data quality | A/R | R | A/R |
| Consumer reporting | C | I | A/R |
The exact RACI varies by organization.
The important principle is explicit accountability.
3.49 Connectivity architecture
Connectivity can involve several layers:
DATASPHERE
│
Connection Object
│
┌────────────┼────────────┐
│ │ │
Internet Cloud Connector DP Agent
│ │ │
└────────────┼────────────┘
▼
SOURCE SYSTEM
The actual path depends on the connection type and source location.
3.50 Cloud Connector
Cloud Connector is relevant for supported on-premise connectivity patterns.
Conceptually:
Datasphere
│
▼
Cloud Connector
│
▼
On-Premise Network
│
▼
Source
It helps establish controlled connectivity without exposing the source system directly to the public internet.
Do not assume every connection needs Cloud Connector.
Check the specific connection's prerequisites.
3.51 Data Provisioning Agent
The SAP HANA smart data integration Data Provisioning Agent can be required for certain remote-source connectivity and adapter scenarios.
Conceptually:
Datasphere
│
▼
DP Agent
│
▼
Adapter
│
▼
Source
The DP Agent is not equivalent to Cloud Connector.
They solve different connectivity requirements.
3.52 Cloud Connector versus DP Agent
| Concern | Cloud Connector | DP Agent |
|---|---|---|
| Primary purpose | Secure network connectivity | Data provisioning connectivity/adapters |
| Typical role | Reach on-premise systems | Support specific integration adapters |
| Network tunnel | Yes | Not the same role |
| Transformation engine | No | No |
| Replication engine | No | Supports adapter-based provisioning scenarios |
| Universal requirement | No | No |
The exact prerequisites are connection-specific.
For example, SAP documentation lists DP Agent requirements for certain Oracle remote-table scenarios, while replication-flow connectivity can use Cloud Connector.
3.53 Connection security
Production integration should define:
- authentication mechanism
- credential ownership
- certificate management
- encryption
- network path
- secret rotation
- least privilege
- auditability
Never design a production integration around:
"It works with my personal credentials."
Use managed technical identities wherever supported.
3.54 Technical user architecture
A typical pattern:
Datasphere
│
▼
Connection
│
▼
Technical Identity
│
▼
Source Authorization
The source account should have only the permissions required for:
- reading
- extracting
- replication
- metadata discovery
Do not give broad source administration rights to an analytics connection.
3.55 Source authorization versus Datasphere authorization
Two security domains exist:
Datasphere security
Controls what users can do in Datasphere.
Source security
Controls what the Datasphere connection can do at the source.
Therefore:
User
│
▼
Datasphere Role
│
▼
Datasphere Connection
│
▼
Technical Identity
│
▼
Source Authorization
Both sides must be secured.
3.56 Integration naming
Use consistent names.
Example:
CON_S4_PRD_FIN
CON_S4_PRD_PROC
CON_ORACLE_SUPPLIER
CON_SNOWFLAKE_ANALYTICS
CON_S3_LANDING
Avoid:
TEST
NEW
SOURCE1
FINAL_CONNECTION
TEMP2
A connection name should communicate:
- source
- environment
- domain
- purpose
3.57 Connection ownership
For every production connection define:
- owner
- backup owner
- source owner
- credentials owner
- certificate owner
- support group
- environment
- business purpose
- supported features
- dependent objects
This becomes essential during incidents.
3.58 Source-to-target architecture catalog
Create an enterprise catalog:
| Source | Domain | Pattern | Latency | Delta | Target |
|---|---|---|---|---|---|
| S/4HANA | Finance | ODP / Replication | 15 min | Yes | Local |
| S/4HANA | Procurement | ODP / Replication | 15 min | Yes | Local |
| Oracle | Supplier | Replication | Hourly | Connector-specific | Local |
| Snowflake | Analytics | Federation | Live | N/A | Remote |
| S3 | History | File ingestion | Daily | File-based | File/Local |
| SaaS API | CRM | API ingestion | Hourly | API-specific | Local |
This catalog is one of the most valuable architecture artifacts.
3.59 Integration pattern decision tree
Use this decision tree before implementation:
START
│
▼
Does the business require source-time data?
│
├── YES ──► Can the source handle live queries?
│ │
│ ├── YES ──► Consider Federation
│ │
│ └── NO ───► Consider Replication
│
└── NO
│
▼
Does the source support reliable delta extraction?
│
├── YES ──► Replication / Incremental Load
│
└── NO
│
▼
Can the dataset be batch loaded?
│
├── YES ──► Scheduled ingestion
│
└── NO ───► Re-evaluate source integration
This is a starting decision framework, not an automatic rule.
3.60 Integration decision matrix
| Question | Federation | Replication | Data Flow | Replication Flow |
|---|---|---|---|---|
| Need local persistence | No | Yes | Yes | Yes |
| Live source dependency | High | Lower | Lower | Lower |
| Delta support | Source-dependent | Source-dependent | More limited | Stronger where supported |
| Complex transformations | Limited | Limited in flow itself | Strong | Targeted transformation capabilities |
| Large initial load | Source-dependent | Strong | Moderate | Strong |
| Recovery | Source-dependent | Source-dependent | More limited | Stronger |
| Typical role | Live access | Persisted copy | ETL | Modern replication |
Always verify current feature support for the exact connection.
3.61 Do not confuse movement with transformation
This is one of the most important architecture distinctions.
Movement
Source → Target
Transformation
Source
↓
Business logic
↓
Target
Orchestration
Task A
↓
Task B
↓
Task C
Therefore:
Replication Flow = movement / replication
Transformation Flow = transformation
Task Chain = orchestration
Data Flow can combine movement and transformation within its supported capabilities.
3.62 Integration layering
A mature architecture separates:
Layer 1 — Connectivity
Can systems communicate?
Layer 2 — Acquisition
Can data be extracted?
Layer 3 — Persistence
Should data be stored locally?
Layer 4 — Transformation
Where are business rules applied?
Layer 5 — Semantics
What does the data mean?
Layer 6 — Consumption
Who uses it?
Connectivity
↓
Acquisition
↓
Persistence
↓
Transformation
↓
Semantics
↓
Consumption
Do not collapse all six layers into one pipeline.
3.63 Raw, harmonized, and semantic layers
A useful architecture is:
SOURCE
│
▼
RAW / ACQUIRED
│
▼
HARMONIZED
│
▼
BUSINESS SEMANTIC
│
▼
CONSUMPTION
For example:
S/4HANA Procurement
│
▼
RAW PURCHASE ORDER
│
▼
HARMONIZED PROCUREMENT
│
▼
PROCUREMENT SPEND MODEL
│
▼
SAC
The integration layer should not contain every business rule.
3.64 Where should transformation happen?
Possible locations:
- Source system
- Integration flow
- Datasphere preparation layer
- Semantic model
- Consumer
Architectural preference:
Transform as close as practical to the layer that owns the meaning, while preserving reusable raw data where justified.
Do not push business semantics into a connection merely because the connector supports a transformation.
3.65 Pushdown thinking
When processing data, ask:
Where is the cheapest place to execute this operation?
Potential locations:
Source DB
│
▼
Integration Engine
│
▼
Datasphere
│
▼
Consumer
Good architecture minimizes unnecessary data movement.
But source pushdown is not always desirable.
A source system may be mission-critical and should not be burdened by analytical joins.
3.66 Source protection principle
A useful rule:
Never optimize Datasphere performance by blindly pushing analytical workload onto a transactional source.
For S/4HANA:
- protect OLTP workloads
- use supported extraction interfaces
- respect source SLAs
- coordinate with source teams
- avoid uncontrolled federated queries
3.67 Query-time versus load-time cost
Federation
Cost occurs mainly at query time.
Dashboard Query
↓
Remote Source
↓
Result
Replication
Cost occurs mainly at ingestion time.
Source Change
↓
Replication
↓
Local Storage
This is a powerful architectural trade-off.
3.68 Cost model
A simple conceptual model:
Federation Cost
=
Network
+
Source Compute
+
Query Concurrency
+
Operational Risk
Replication Cost
=
Transfer
+
Storage
+
Load Compute
+
Operational Management
The cheaper option depends on workload.
3.69 Performance model
For federation:
Performance ≈
Source Performance
+
Network
+
Pushdown
+
Query Complexity
For local persistence:
Performance ≈
Local Storage
+
Local Compute
+
Model Design
This is why high-concurrency analytics often benefit from persistence.
3.70 Historical requirements
Ask:
"Do we need to reconstruct what the business looked like six months ago?"
If yes, live federation may be insufficient.
Historical analytics often requires:
- persisted data
- snapshots
- effective dates
- change history
- retention strategy
Example:
Current Supplier Master
≠
Supplier Master as of 31 March
The integration architecture must preserve the required history.
3.71 Delete handling
Many developers think only about:
- inserts
- updates
Production integration must also consider:
- deletes
- soft deletes
- source deactivation
- late-arriving deletes
- referential cleanup
A delta strategy that cannot correctly process deletes can create silent data corruption.
3.72 Idempotency
A production pipeline should be safe to retry.
Example:
Batch 101
↓
Partial failure
↓
Retry
If retry duplicates the data, the architecture is unsafe.
A better design ensures:
Retry
↓
Same business state
Idempotency mechanisms depend on the integration pattern.
3.73 Reconciliation
Replication should be reconcilable.
Useful checks:
- source row count
- target row count
- delta count
- checksum where appropriate
- amount totals
- key counts
- rejected records
- last successful timestamp
Example:
Source
Rows = 10,000,000
Amount = 4.82B
Target
Rows = 10,000,000
Amount = 4.82B
Status = PASS
Do not rely only on "job succeeded."
3.74 Integration observability
Monitor:
- connection health
- load duration
- rows processed
- rows rejected
- delta lag
- task failures
- source latency
- target latency
- storage growth
- retry count
The business metric should include freshness.
Example:
Procurement data is 47 minutes old.
That is more useful than:
Replication flow status = green.
3.75 Failure architecture
Design failure before production.
SOURCE
│
▼
INTEGRATION
│
├── SUCCESS ─────► TARGET
│
└── FAILURE
│
├── Retry
├── Alert
├── Quarantine
└── Escalate
Ask:
- Can the task restart?
- From where?
- Will it duplicate data?
- What is the business impact?
- Who is alerted?
- How is the incident reconciled?
3.76 Recovery point
For every pipeline define:
"What is the smallest unit from which we can safely recover?"
Possibilities:
- object
- batch
- partition
- file
- timestamp
- source checkpoint
Replication Flow's recovery behavior can differ from Data Flow, which is one reason current architecture should consider the newer flow types.
3.77 Schema evolution
Sources change.
Examples:
- new column
- renamed column
- removed column
- changed data type
- changed precision
- changed semantics
A production integration should define:
Schema Change
│
├── Compatible → Accept
│
├── Review → Test
│
└── Breaking → Block / Remediate
Never assume source schemas are immutable.
3.78 Contract testing
For critical integrations, test:
- required columns
- data types
- key uniqueness
- nullability
- expected ranges
- record counts
- delta behavior
- delete behavior
This turns data integration into an engineering discipline.
3.79 Data quality boundary
A useful architecture:
RAW
│
▼
VALIDATION
│
├── PASS ───► HARMONIZED
│
└── FAIL
│
▼
QUARANTINE
Do not allow malformed source data to silently contaminate the curated layer.
3.80 Late-arriving data
Suppose:
Business Date = Monday
Arrival Date = Wednesday
The pipeline must determine:
- whether to backfill
- whether to reopen aggregates
- whether consumers see corrected numbers
- whether historical partitions are updated
This is especially important in finance and procurement.
3.81 Integration with SAP Analytics Cloud
SAC is generally a consumer of modeled data.
Do not make the SAC dashboard responsible for integration logic.
Prefer:
Source
↓
Datasphere Integration
↓
Datasphere Modeling
↓
Analytic Model
↓
SAC
This centralizes data engineering and governance.
3.82 Integration with AFO
Analysis for Office can consume analytical models and data exposed through the Datasphere architecture.
The same principle applies:
Keep acquisition and business transformation upstream of the reporting client.
AFO should not become an ETL engine.
3.83 Integration architecture for procurement
Consider:
- S/4HANA purchase orders
- supplier master in Oracle
- market pricing in S3
- historical procurement in BW/4HANA
- SAC procurement dashboard
Possible architecture:
S/4HANA
│
▼
ODP / Supported Extraction
│
▼
PROCUREMENT RAW
│
├──────────────┐
│ │
▼ ▼
Oracle S3 Pricing
Supplier Historical
│ │
└──────┬───────┘
▼
HARMONIZED
PROCUREMENT
│
▼
SPEND SEMANTIC
│
▼
SAC
This is the kind of architecture you should be able to defend.
3.84 Integration architecture for finance
Consider:
- S/4HANA finance
- BW historical balances
- external FX rates
- external planning data
Architecture:
S/4HANA Finance
│
▼
ODP / CDS
│
▼
Finance Acquisition
│
├── Historical BW
│
├── FX Source
│
└── Planning Source
│
▼
Harmonization
│
▼
Finance Model
│
▼
SAC
The integration architecture should preserve source semantics and clearly define ownership.
3.85 Integration architecture for operational analytics
Requirement:
Data must be available within a few minutes.
Potential design:
Source
│
▼
Delta-Capable Integration
│
▼
Persisted Local Data
│
▼
Lightweight Transformation
│
▼
Analytic Model
│
▼
SAC
Avoid unnecessary transformations in the critical latency path.
3.86 Integration architecture for historical analytics
Requirement:
Five years of history with low query latency.
Potential design:
Historical Sources
│
▼
Batch / Replication
│
▼
Persisted Historical Layer
│
▼
Curated Model
│
▼
Analytic Model
Federating five years of remote source data for every dashboard query is generally a poor default.
3.87 Integration architecture for large files
Requirement:
5 TB monthly supplier files.
Potential direction:
Supplier
│
▼
Object Storage
│
▼
File Space
│
▼
Transformation
│
▼
Curated Standard Space
│
▼
Analytical Consumption
The object-store pattern can reduce pressure on conventional database storage for staging workloads.
3.88 Multi-source harmonization
Suppose:
S/4HANA → Customer
Salesforce → Customer
Oracle → Customer
Do not simply union them.
First define:
- business key
- source priority
- survivorship
- canonical attributes
- duplicate strategy
- source lineage
Then:
Multiple Sources
↓
Identity Resolution
↓
Canonical Customer
↓
Semantic Model
Integration is therefore partly a business architecture problem.
3.89 Master data integration
Master data often has:
- multiple sources
- different codes
- different descriptions
- different ownership
- different update frequency
Example:
S/4
MATNR = 100001
│
├── description
├── plant
└── material group
Supplier
SKU = ABC-100001
│
└── supplier code
The integration layer must establish the relationship.
Do not assume source keys are globally consistent.
3.90 Semantic integration
Integration is not complete when rows arrive.
Example:
Source A:
Revenue = 100
Source B:
Sales = 100
Are they the same metric?
Maybe.
Maybe not.
Semantic integration requires:
- definition
- grain
- currency
- unit
- time basis
- aggregation rule
- source authority
This is why integration and semantic modeling must remain connected but distinct.
3.91 Grain is an integration concern
Before joining two sources, document grain.
Example:
Purchase Order Item
PO + Item
versus:
Supplier
Supplier ID
A join can be valid.
But:
PO Item
JOIN
Supplier
may multiply rows if the supplier side is not unique.
Integration architecture must therefore understand data grain before transformation.
3.92 Duplicate amplification
Example:
Orders = 1,000
Suppliers = 100
If supplier has multiple versions:
Orders
×
Supplier Versions
you may accidentally create:
1,000
↓
3,500 rows
This can silently corrupt measures.
Always validate cardinality.
3.93 Integration anti-patterns
Anti-pattern 1 — Replicate everything
Creates storage and operational overhead.
Anti-pattern 2 — Federate everything
Creates runtime source dependency.
Anti-pattern 3 — Real-time everything
Creates unnecessary complexity.
Anti-pattern 4 — Directly query transactional sources
Can harm source performance.
Anti-pattern 5 — No delta strategy
Causes repeated full loads.
Anti-pattern 6 — No reconciliation
Failures become silent data quality issues.
Anti-pattern 7 — No ownership
Incidents become organizational problems.
Anti-pattern 8 — One giant flow
Mixes acquisition, transformation, semantics, and consumption.
Anti-pattern 9 — Consumer-side transformation
Duplicates business logic.
Anti-pattern 10 — Manual production file uploads
Creates operational fragility.
3.94 Architect rule: choose by workload
A useful decision matrix:
| Workload | Preferred direction |
|---|---|
| Small live lookup | Federation |
| High-concurrency analytics | Replication |
| Large incremental source | Replication Flow where supported |
| Complex transformation | Data Flow / Transformation Flow depending on source and target |
| Existing local data transformation | Transformation Flow |
| Multi-step orchestration | Task Chain |
| SAP business extraction | Supported CDS / ODP pattern |
| Large object-store staging | File-space / replication pattern |
| Event-driven requirement | Streaming-oriented architecture |
| Historical analytics | Persisted data |
Always verify current feature support for the exact connection.
3.95 Architect rule: minimize unnecessary copies
A copy should exist for a reason.
Valid reasons include:
- performance
- resilience
- history
- transformation
- governance
- source protection
- workload isolation
- cost optimization
Invalid reason:
"We usually replicate everything."
3.96 Architect rule: minimize source impact
Before production:
- estimate extraction volume
- estimate frequency
- understand source windows
- confirm delta behavior
- validate source capacity
- test concurrency
- agree on monitoring
- agree on failure handling
The source system is part of the architecture.
3.97 Architect rule: design failure first
Ask:
"What happens at 02:00 when the source is unavailable?"
Possible response:
Source unavailable
↓
Retry
↓
Last successful copy remains
↓
Alert
↓
Source restored
↓
Delta catch-up
↓
Reconciliation
If you cannot explain this flow, the integration is not production-ready.
3.98 Architect rule: freshness is a business metric
Do not monitor only:
- task status
- duration
- row count
Also monitor:
Data Freshness = Current Time - Last Business Data Timestamp
Example:
Last Procurement Event = 13:42
Current Time = 14:00
Freshness = 18 minutes
If SLA is 15 minutes:
Status = BREACHED
This is much more useful to business operations.
3.99 Enterprise integration reference architecture
ENTERPRISE SOURCES
│
┌────────────────────┼────────────────────┐
│ │ │
SAP DATABASES CLOUD
│ │ │
┌──────┼──────┐ ┌─────┼─────┐ ┌─────┼─────┐
│ │ │ │ │ │ │ │ │
S4 ECC BW Oracle SQL Snowflake S3 APIs
│ │ │ │ │ │ │ │
└──────┼──────┘ └─────┼───────┘ └─────┼───┘
│ │ │
└────────────────────┼────────────────────┘
▼
CONNECTION LAYER
│
┌──────────────────┼──────────────────┐
│ │ │
FEDERATION REPLICATION INGESTION
│ │ │
│ ▼ │
│ RAW / LOCAL │
│ │ │
└──────────────────┼──────────────────┘
▼
TRANSFORMATION
│
▼
HARMONIZATION
│
▼
SEMANTIC MODELING
│
▼
ANALYTIC
│
▼
SAC / AFO / BI / APIs
This is a reference architecture, not a mandatory SAP template.
3.100 Integration architecture checklist
Before approving an integration, verify:
Source
- Source owner identified
- Source availability SLA documented
- Source interface documented
- Business semantics understood
Connectivity
- Connection type validated
- Network path validated
- Authentication defined
- Certificates managed
- Cloud Connector requirement checked
- DP Agent requirement checked
Data movement
- Federation versus replication decision documented
- Initial load defined
- Delta mechanism defined
- Delete handling defined
- Historical requirement defined
Transformation
- Transformation location documented
- Grain documented
- Join cardinality validated
- Data quality rules documented
Operations
- Monitoring defined
- Freshness SLA defined
- Retry strategy defined
- Reconciliation defined
- Alerting defined
- Support owner defined
Security
- Technical identity defined
- Least privilege applied
- Source authorization reviewed
- Datasphere authorization reviewed
3.101 Architect exercise
Scenario
A company has:
- S/4HANA Finance
- S/4HANA Procurement
- Oracle Supplier Management
- Salesforce CRM
- Snowflake analytics
- S3 historical files
- BW/4HANA historical finance
- SAC as the main analytics client
Requirements:
- finance dashboards within 15 minutes
- procurement dashboards within 10 minutes
- supplier lookup live
- customer analytics hourly
- five years of historical finance
- source systems must not be overloaded
- regional data restrictions
- 500 analytical users
Design the integration architecture
You must decide:
- Which sources are federated?
- Which are replicated?
- Which use ODP/CDS?
- Which use Replication Flow?
- Where is transformation performed?
- Where is history stored?
- Which data goes through a file space?
- How are source systems protected?
- How is delta handled?
- How is freshness monitored?
- What happens when S/4HANA is unavailable?
- Which objects are shared across spaces?
3.102 Exercise answer framework
Do not jump directly to tools.
Use:
BUSINESS REQUIREMENT
↓
LATENCY
↓
VOLUME
↓
SOURCE CAPABILITY
↓
DELTA
↓
SOURCE IMPACT
↓
PERSISTENCE
↓
TRANSFORMATION
↓
SECURITY
↓
OPERATIONS
↓
COST
↓
FINAL PATTERN
This is the architecture reasoning you should practice.
3.103 Hands-on lab — integration catalog
Create a spreadsheet or Markdown table with:
| Source | Object | Owner | Pattern | Latency | Delta | Target | SLA |
|---|
Populate at least:
- one S/4HANA source
- one Oracle source
- one cloud source
- one file source
- one federated source
Then justify every pattern.
3.104 Hands-on lab — federation decision
Choose a remote source.
Document:
- Data volume
- Query frequency
- Number of consumers
- Source SLA
- Network latency
- Pushdown potential
- Failure impact
- Historical requirement
Then answer:
Should this data remain federated?
If yes, defend it.
If no, design replication.
3.105 Hands-on lab — replication decision
Choose a high-volume source.
Document:
- initial volume
- daily change volume
- delta mechanism
- load frequency
- target table
- retention
- reconciliation
- failure recovery
Then estimate:
Initial Load Volume
+
Daily Delta Volume
+
Retention
=
Storage Requirement
This is the beginning of capacity-aware integration design.
3.106 Hands-on lab — SAP source architecture
For an S/4HANA business object:
- Identify the business object.
- Identify the supported CDS/extraction interface.
- Determine whether ODP is involved.
- Determine delta support.
- Identify required connection.
- Define target pattern.
- Define freshness SLA.
- Define source impact.
- Define reconciliation.
- Document the final architecture.
Do not begin by selecting a raw database table.
3.107 Hands-on lab — failure simulation
Design:
Replication
│
▼
Failure
│
├── Detect
├── Retry
├── Alert
├── Recover
└── Reconcile
Answer:
- What checkpoint exists?
- Can the flow restart safely?
- Can records duplicate?
- How is freshness calculated?
- Who receives the alert?
- How does the business know the data is stale?
3.108 Troubleshooting framework
When a pipeline does not deliver data, troubleshoot in layers.
1. Network
↓
2. Connection
↓
3. Authentication
↓
4. Source authorization
↓
5. Source object
↓
6. Extraction
↓
7. Delta
↓
8. Flow execution
↓
9. Target persistence
↓
10. Modeling
↓
11. Consumption
Do not immediately rebuild the flow.
Find the failing layer.
3.109 Troubleshooting — connection fails
Check:
- connection type
- endpoint
- network path
- Cloud Connector
- certificates
- credentials
- source availability
- required driver/adapter
- DP Agent
- connection-specific prerequisites
The correct diagnosis depends on the connection type.
3.110 Troubleshooting — remote table is empty
Check:
- source object exists
- source authorization
- metadata import
- filter
- remote-table status
- source connectivity
- query behavior
- source-side restrictions
Do not assume "empty" means "no source data."
3.111 Troubleshooting — replication is behind
Check:
- source change volume
- initial load still running
- delta backlog
- source throttling
- network throughput
- target write speed
- transformation bottleneck
- failed objects
- retry loops
- resource contention
Measure:
Current Source Position
-
Target Position
=
Replication Lag
3.112 Troubleshooting — full load is too slow
Check:
- partitioning
- source extraction
- network
- target resources
- object count
- parallelism
- unnecessary columns
- unnecessary rows
- source indexes
- target persistence design
Do not immediately increase capacity.
First determine where the bottleneck is.
3.113 Troubleshooting — delta is incorrect
Investigate:
- source change tracking
- deleted rows
- timestamps
- keys
- duplicate changes
- late-arriving records
- timezone handling
- restart behavior
- reconciliation
Delta bugs are often more dangerous than complete failures because they can produce plausible but incorrect data.
3.114 Troubleshooting — source is overloaded
Immediately evaluate:
- federation queries
- extraction frequency
- query concurrency
- large full loads
- unnecessary columns
- unnecessary rows
- missing filters
- poorly designed source interface
Possible remediation:
Federation
↓
Persisted Replication
↓
Local Analytics
Protect the source before optimizing the dashboard.
3.115 Troubleshooting — data is fresh but wrong
Freshness is not correctness.
Check:
- grain
- joins
- duplicate records
- filters
- currency
- units
- date logic
- delta deletes
- source semantics
- transformation rules
A dashboard can be refreshed every minute and still be wrong.
3.116 Interview preparation
What is the difference between federation and replication?
Federation accesses source data remotely, while replication creates a local copy in Datasphere. Federation reduces duplication but increases runtime source dependency; replication increases storage and integration cost but generally provides more predictable local analytical performance.
What is a connection?
A connection is the configured technical integration endpoint that enables Datasphere to communicate with a source or target. It does not by itself determine whether the architecture will federate, replicate, or transform data.
What is a remote table?
A remote table represents source data exposed through a connection. Depending on the connection's supported features, it can be used for live remote access or replication.
When would you replicate instead of federate?
When query performance, concurrency, source protection, historical persistence, or source independence justify maintaining a local copy.
When would you federate?
When live access, low duplication, selective querying, or source ownership make remote access appropriate and the source can safely support the workload.
Why is delta important?
Because repeatedly moving the complete dataset does not scale for large sources. Delta processing moves only changes where the source and integration pattern support it.
What is ODP?
Operational Data Provisioning is an SAP framework for data extraction and replication, including delta mechanisms through the Operational Delta Queue.
What is the role of CDS?
CDS can provide business-oriented semantic and extraction interfaces in ABAP-based SAP systems. It can be preferable to treating raw database tables as the primary integration contract.
Cloud Connector versus DP Agent?
They solve different connectivity requirements. Cloud Connector provides controlled network connectivity for supported scenarios, while DP Agent supports specific data-provisioning adapter scenarios.
Is real-time always better?
No. Real-time adds complexity and cost. The latency requirement should be driven by business value.
What is the biggest integration architecture mistake?
Choosing a technical tool before understanding latency, volume, delta capability, source impact, history, security, and operational requirements.
3.117 Architect checkpoint
You should now be able to look at any source and answer:
WHO OWNS IT?
↓
WHAT IS THE BUSINESS INTERFACE?
↓
HOW FRESH MUST IT BE?
↓
HOW MUCH DATA EXISTS?
↓
HOW MUCH CHANGES?
↓
CAN WE EXTRACT DELTAS?
↓
SHOULD WE FEDERATE OR REPLICATE?
↓
WHERE SHOULD TRANSFORMATION OCCUR?
↓
HOW DO WE PROTECT THE SOURCE?
↓
HOW DO WE RECOVER?
↓
HOW DO WE PROVE CORRECTNESS?
↓
WHAT DOES IT COST?
If you can answer these questions, you are thinking like an integration architect.
3.118 The integration maturity model
Level 1 — Connectivity
"We can connect to the source."
Level 2 — Movement
"We can load the data."
Level 3 — Reliability
"We can recover and monitor the load."
Level 4 — Optimization
"We select federation, replication, and transformation based on workload."
Level 5 — Architecture
"We optimize source impact, cost, freshness, security, and resilience."
Level 6 — Enterprise
"We standardize integration patterns across domains and platforms."
The goal of this masterclass is Level 6.
3.119 The architect's integration scorecard
For every integration, score:
| Dimension | Question |
|---|---|
| Freshness | Does the pattern meet the SLA? |
| Performance | Can consumers meet query SLAs? |
| Cost | Is duplication justified? |
| Resilience | Can the platform operate during source outages? |
| Scalability | Can volume and concurrency grow? |
| Security | Are both sides least-privileged? |
| Governance | Is ownership clear? |
| Operability | Can support teams diagnose failures? |
| Maintainability | Can the integration survive source changes? |
| Reusability | Can other domains reuse the output? |
A technically working pipeline can still score poorly as an enterprise architecture.
3.120 Final mental model
Remember this sequence:
SOURCE
│
▼
CONNECTIVITY
│
▼
ACQUISITION
│
├───────────────┐
│ │
FEDERATION REPLICATION
│ │
│ ▼
│ PERSISTENCE
│ │
└───────┬───────┘
▼
TRANSFORMATION
│
▼
HARMONIZATION
│
▼
SEMANTICS
│
▼
CONSUMPTION
And remember the most important principle:
Integration architecture is the art of deciding where data should live, where computation should happen, how fresh it must be, and how much source dependency the business can tolerate.
3.121 What comes next
Phase 04 — Data Acquisition
Phase 04 moves from architecture into the acquisition layer itself.
We will go deeply into:
- creating and managing connections
- connection categories
- source discovery
- remote tables
- local tables
- importing source objects
- source metadata
- SAP source acquisition
- non-SAP acquisition
- files
- APIs
- connection prerequisites
- Cloud Connector
- Data Provisioning Agent
- credentials and certificates
- source-side authorization
- source object discovery
- acquisition troubleshooting
- initial-load design
- source filters
- acquisition standards
- production acquisition checklist
Then later:
Phase 11 becomes the deep engineering phase for:
- Data Flow
- Replication Flow
- Transformation Flow
- Task Chain
- delta processing
- restart/recovery
- orchestration
- performance
- production operations