Skip to main content

18.1 Integration Mental Model

1. Start with the workload​

Do not start with the connector catalog.

Start with the workload:

Business requirement
|
Freshness
Volume
Latency
Availability
Security
|
Integration pattern
|
Connection / adapter
|
Runtime design

2. Four primary patterns​

Federation​

Query the source when the consumer needs data.

Best when:

  • data is relatively stable,
  • source query performance is acceptable,
  • low-copy architecture matters,
  • freshness must be immediate.

Risk:

  • consumer workload becomes source workload.

Replication​

Copy source data into Datasphere.

Best when:

  • repeated analytics queries are expected,
  • source protection matters,
  • predictable performance is required,
  • history or local transformation is valuable.

Risk:

  • storage, freshness management and duplicate data.

ETL or transformation​

Extract data, transform it and persist the result.

Best when:

  • business logic is complex,
  • multiple sources must be harmonized,
  • transformation should be separated from source systems.

Push​

An external application or ETL platform writes into an Open SQL schema.

Best when:

  • the external platform owns extraction,
  • a custom integration is required,
  • application-driven ingestion is appropriate.

3. Decision matrix​

RequirementFederationReplicationETLPush
Live accessStrongDepends on freshnessDependsDepends
Source protectionWeakerStrongerStrongerStronger
Complex transformationLimitedModerateStrongExternal
Predictable query performanceDepends on sourceStrongStrongStrong
Minimal persistenceStrongWeakerWeakerDepends

4. Anti-pattern​

Source
|
"Connect everything"
|
Remote tables everywhere
|
Complex views
|
Source overloaded

A remote table is not automatically the best architecture.

5. Architect exercise​

For these workloads choose the primary pattern:

  • operational dashboard requiring current inventory,
  • monthly finance reporting,
  • 5 TB historical analytics,
  • customer-service application lookup,
  • cross-cloud enterprise data product.

For each decision state why the rejected alternatives are inferior.