Skip to main content

18.9 APIs, Partners & Application Data

1. API data is different​

APIs often expose:

  • paginated records,
  • rate limits,
  • nested JSON,
  • asynchronous jobs,
  • mutable entities,
  • event streams.

Treating an API like a relational table creates operational problems.

API
 |
extractor
 |
landing / staging
 |
normalize
 |
validate
 |
curated table
 |
semantic model

3. Rate limits​

The ingestion architecture must account for:

  • page size,
  • retry-after behavior,
  • exponential backoff,
  • concurrency limits,
  • token expiration,
  • partial responses.

4. Idempotency​

Every ingestion batch should have a deterministic identity.

If the same page is retried, the target should not silently create duplicate business records.

5. Schema evolution​

APIs can add fields without warning.

Define:

  • contract tests,
  • version handling,
  • unknown-field policy,
  • nullability policy,
  • backward compatibility.

6. Partner connections​

SAP Datasphere also supports partner-oriented connection patterns where partner platforms can push data through supported mechanisms such as Open SQL.

Choose partner integration when the partner platform owns extraction and provides useful capabilities beyond what a native connector provides.

Exercise​

A SaaS CRM API allows 100 requests per minute and contains 300 million customer interactions.

Design an incremental ingestion architecture with restart and deduplication.