Skip to main content

Phase 12 — Persistence & Performance Engineering

SAP Datasphere persistence and performance architecture

12.1 What this phase solves​

A Datasphere model can be logically correct and still be operationally poor.

A query can be slow because of remote federation, excessive data movement, repeated calculation, an inefficient join, a poor persistence boundary, unsuitable partitioning, memory pressure, or a transformation workload that is simply too large.

This phase teaches the performance engineering discipline required to answer:

  • Should this data remain virtual?
  • Should it be replicated?
  • Should a view be persisted?
  • Where should persistence occur?
  • Should a local table be partitioned?
  • Which columns should be filtered or projected?
  • Is the bottleneck source, network, compute, storage, or query logic?
  • When does persistence improve performance and when does it make freshness worse?
  • How do we prove an optimization actually worked?

The central principle is:

Performance is an architecture property, not a last-minute tuning activity.

12.2 Current SAP performance principles​

Current SAP Datasphere guidance emphasizes reducing unnecessary data volume, filtering and projecting data appropriately, persisting data where necessary, and using analysis tools to identify expensive model components. Remote tables can be optimized by restricting loaded columns and filters. Views can be persisted when repeated execution of complex logic would otherwise affect performance. Local tables can be partitioned for large volumes.

Official references:

12.3 The performance mental model​

Use this pipeline:

SOURCE | v FEDERATION / REPLICATION | v PERSISTENCE | v MODEL | v QUERY | v CONSUMPTION

At every boundary ask:

  1. How much data moves?
  2. Where does it execute?
  3. How often does it execute?
  4. How much data is read?
  5. How much data is written?
  6. Can the result be reused?
  7. What freshness is required?
  8. What resource does it consume?
  9. What happens under concurrency?
  10. What happens when volume grows?

12.4 Performance is multidimensional​

Do not reduce performance to query duration.

Measure:

DimensionExample
LatencyDashboard response time
ThroughputRows processed per minute
FreshnessAge of latest data
ConcurrencyNumber of simultaneous users
Resource usageMemory / compute consumption
StabilityFailure frequency
ScalabilityBehavior as data grows
CostStorage and processing consumption

A design that runs in 3 seconds for one user but collapses with 100 users is not a performant enterprise design.

12.5 The persistence spectrum​

Think in terms of a spectrum:

REMOTE / FEDERATED | v REPLICATED REMOTE TABLE | v LOCAL TABLE | v PERSISTED VIEW | v PRECOMPUTED CURATED DATA

These are not interchangeable.

They trade:

  • freshness,
  • compute,
  • storage,
  • source-system load,
  • reuse,
  • recovery,
  • complexity.

12.6 Federation​

Federation means data remains at or is accessed through the remote source rather than being fully materialized locally.

Advantages:

  • low local storage footprint;
  • potentially fresher source data;
  • fewer copies;
  • useful for selected scenarios.

Risks:

  • network dependency;
  • source-system latency;
  • remote-system workload;
  • unpredictable performance;
  • repeated remote execution;
  • reduced isolation from source outages.

Use federation when the workload and SLA tolerate those characteristics.

12.7 Replicated remote tables​

Many Datasphere connections allow remote tables to be replicated so data is stored in Datasphere rather than fetched from the remote system for every use.

Current SAP documentation notes that replication can improve performance, while unnecessary columns and filters can reduce data footprint. In-memory storage can also be used for applicable replicated tables.

Architecture:

Remote Source | | replication v Datasphere Storage | v Local Consumption

The key decision is whether the benefit of local access justifies storage and replication maintenance.

12.8 Remote-table volume reduction​

If a remote table contains 500 columns but the workload requires 40:

500 source columns | v projection / restriction | v 40 required columns

Likewise, if only one fiscal year is needed for a workload, a valid central filter can reduce the loaded data.

Current SAP guidance explicitly recommends removing unnecessary columns and setting filters to reduce remote-table data volume and memory peaks.

Do not remove a column without checking dependent objects and source-contract requirements.

12.9 In-memory versus disk-oriented storage​

Storage choice affects the performance profile.

In-memory access can provide very fast access for suitable workloads but consumes memory resources.

Disk-oriented storage can support larger footprints with a different performance profile.

Do not use the simplistic rule:

"In-memory is always faster."

Instead ask:

  • What is the working set?
  • How frequently is it queried?
  • What is the table size?
  • What is the concurrency?
  • Is the workload scan-heavy?
  • What is the freshness requirement?
  • What does the current Datasphere storage configuration support?

12.10 View execution versus view persistence​

A non-persisted view is evaluated when it is accessed.

Conceptually:

User query | v View logic | +--> joins +--> filters +--> calculations +--> consumed views +--> remote sources | v result

If a complex view is accessed repeatedly, the same expensive logic can be executed repeatedly.

A persisted view changes the pattern:

View definition | v Persistence run | v Stored result | v Consumer

Current SAP documentation states that persisting a view can improve performance and that persisted views can be refreshed on a schedule.

12.11 When to persist a view​

Good candidates often have:

  • expensive joins;
  • expensive calculations;
  • large input datasets;
  • repeated consumption;
  • stable or controlled refresh requirements;
  • multiple downstream consumers;
  • acceptable refresh latency.

Example:

20 dashboards | v expensive semantic view | v repeated computation

Potential redesign:

expensive semantic view | v persisted result / |
v v v BI1 BI2 BI3

But persistence is not free.

12.12 Persistence trade-off​

Persistence introduces:

  • storage;
  • refresh processing;
  • freshness latency;
  • scheduling;
  • failure handling;
  • lifecycle management.

Therefore:

Performance gain versus Freshness + storage + processing cost

The correct question is not:

Should I persist?

It is:

Which computation should happen once and be reused rather than recomputed repeatedly?

12.13 Persisting the wrong layer​

A common mistake is persisting every intermediate view.

Example:

Source | View A -> persisted | View B -> persisted | View C -> persisted | View D -> persisted

This can create:

  • unnecessary storage;
  • extra refresh jobs;
  • stale intermediate layers;
  • operational complexity;
  • more failure points.

Prefer persistence boundaries that create clear reuse or operational value.

12.14 Persisted view freshness​

Suppose:

Source changes at 10:00 View persisted at 09:00

The persisted result may remain stale until its next update.

Therefore every persisted view needs a freshness contract:

AttributeExample
Refresh cadenceEvery 30 minutes
Maximum staleness45 minutes
OwnerAnalytics platform
Failure alertRequired
RecoveryRe-run persistence
Consumer expectationIntraday

A performance improvement that violates the business freshness SLA is not an improvement.

12.15 Persistence run modes​

Current Datasphere documentation describes configurable persistence run modes, including a performance-oriented mode that processes SQL directly in memory and can increase memory consumption for complex or high-volume models.

Architectural decision:

Performance-oriented mode | +--> faster execution | +--> higher memory pressure

Choose based on workload evidence rather than assuming the fastest mode is always safest.

12.16 Memory engineering​

Memory pressure can come from:

  • large intermediate results;
  • wide rows;
  • high-cardinality joins;
  • large aggregations;
  • repeated calculations;
  • persisted-view execution;
  • concurrent workloads;
  • remote-table loads.

A useful diagnostic sequence:

Memory issue | +--> Which object? | +--> Which operation? | +--> How many rows? | +--> How many columns? | +--> What join cardinality? | +--> What concurrency? | +--> Can the workload be reduced? | +--> Can it be persisted or partitioned?

Do not solve memory problems by simply increasing resources without understanding the workload.

12.17 Projection pruning​

Read only what is needed.

Bad:

SELECT * FROM very_large_dataset

Better:

required business columns only

At each layer ask:

  • Is this column required downstream?
  • Is it required for a join?
  • Is it required for a filter?
  • Is it required for semantics?
  • Is it required for auditing?

Current SAP guidance explicitly recommends minimizing data processed at each step.

12.18 Filter pushdown​

A filter can dramatically reduce data volume.

Conceptually:

Source: 1 billion rows | v Filter | v 10 million rows | v Join

versus:

Source: 1 billion rows | v Join | v Filter

If the filter can safely be applied earlier, the second design may perform much more work.

However, do not move filters blindly. Verify semantic equivalence, especially with outer joins and null-producing logic.

12.19 Join performance​

Always establish grain.

Example:

Sales Item one row per Order + Item

Customer one row per Customer

This is usually a controlled relationship.

But:

Sales Item many rows per Customer

Customer Address History many rows per Customer

can create:

N x M row multiplication

Do not hide the problem with DISTINCT.

12.20 Aggregation placement​

Consider:

1 billion sales rows | +--> Customer join | +--> aggregate

versus:

1 billion sales rows | v aggregate | v smaller set | v customer join

The second can be much cheaper when the aggregation is semantically valid.

The rule is:

Reduce data before expensive operations when the business grain permits it.

12.21 Partitioning local tables​

Current Datasphere documentation supports partitioning local tables to break large datasets into smaller partitions.

Partitioning can help:

  • manage large volumes;
  • reduce the amount of data relevant to an operation;
  • improve parallel processing opportunities;
  • support maintainability of large tables.

It is not a universal performance switch.

12.22 Partition types​

Current Datasphere documentation includes partitioning approaches such as:

  • range;
  • hash;
  • dynamic range, where applicable.

Conceptually:

Range​

2024 | 2025 | 2026 | 2027

Useful when workloads naturally filter on the range key.

Hash​

Key -> hash -> partition

Useful for distributing rows across partitions where a range predicate is not the primary access pattern.

Dynamic range​

Partitions can evolve according to configured intervals for suitable incremental values.

Always verify current limitations and eligible data types before implementation.

12.23 Choosing a partition key​

A good candidate often has:

  • high relevance to query filters;
  • predictable distribution;
  • stable semantics;
  • acceptable cardinality;
  • manageable growth.

For time-series fact data, a date or timestamp can be attractive.

Bad choices can create:

  • skew;
  • tiny partitions;
  • huge partitions;
  • ineffective pruning.

Partitioning should follow workload patterns, not simply table size.

12.24 Partition skew​

Example:

Partition A = 800 GB Partition B = 20 GB Partition C = 15 GB Partition D = 10 GB

This can create imbalance.

Hash partitioning can distribute data differently, but it may not help time-range pruning.

Architectural choice depends on the workload.

12.25 Partition pruning​

If a query asks:

Fiscal Year = 2026

and the table is partitioned by a compatible year/range strategy, only relevant partitions may need to participate.

Conceptually:

2024 2025 2026 2027 | | | | skip skip READ skip

This is valuable because the query can avoid unnecessary data.

But pruning depends on the query predicate and partition design.

12.26 Partition monitoring​

Current Datasphere monitoring can expose partition-level information such as record count, growth, memory size where applicable, disk size where applicable, and range boundaries.

Use this information to identify:

  • skew;
  • abnormal growth;
  • unexpected partition size;
  • storage anomalies;
  • future capacity problems.

12.27 The View Analyzer​

View Analyzer is one of the most important performance-analysis tools in Datasphere.

It can analyze a graphical or SQL view and the entities it consumes, including consumed views, local tables, and remote sources.

Use it to answer:

  • Which component contributes most to the workload?
  • Where is remote data involved?
  • Which view is expensive?
  • Which layer should be persisted?
  • Where might memory problems originate?

Current SAP documentation specifically positions View Analyzer as a tool for optimizing data models and deciding which views to persist.

12.28 Performance investigation workflow​

Use:

Symptom | v Baseline | v View Analyzer / runtime evidence | v Identify expensive component | v Hypothesis | v Change one variable | v Measure | v Compare | v Keep / revert

Do not change five things at once.

12.29 Baseline before optimization​

Capture:

  • query duration;
  • rows returned;
  • source volume;
  • persistence state;
  • runtime;
  • memory symptoms;
  • concurrency;
  • freshness;
  • timestamp;
  • exact model version.

Without a baseline, you cannot prove improvement.

12.30 Performance experiment design​

Example:

VersionChangeDurationRowsResource symptomResult
ABaseline
BProjection
CFilter
DPersistence
EPartitioning

Change one major variable at a time.

Otherwise:

faster

does not tell you:

why faster

12.31 Source versus Datasphere bottleneck​

A query may look like a Datasphere problem but actually be source-bound.

Trace:

Consumer | v Datasphere model | v Remote table | v Source system

If the remote source is slow, optimizing downstream joins may provide little benefit.

This is why federation versus replication is a performance architecture decision.

12.32 Remote source protection​

Do not allow analytical concurrency to become uncontrolled source-system concurrency.

Potential controls include:

  • replication;
  • restricted remote-table filters;
  • reduced column footprint;
  • persistence;
  • workload scheduling;
  • semantic reuse.

The principle is:

Protect operational source systems from analytical workloads when the business architecture allows it.

12.33 Freshness versus performance​

A simple model:

Federation

  • freshness
  • isolation
  • predictable latency

Replication

  • local performance
  • isolation
  • replication latency
  • storage

Persistence

  • reusable computation
  • predictable consumption
  • refresh latency
  • storage / processing

There is no universally best option.

12.34 Performance and concurrency​

A query that performs well alone may behave differently under concurrency.

Test:

1 user 10 users 50 users 100 users

where relevant to the workload.

Measure:

  • median latency;
  • tail latency;
  • resource pressure;
  • failures;
  • source impact.

Enterprise architecture must optimize the workload, not only the demo.

12.35 Performance and semantic correctness​

Never optimize by changing business meaning.

Example:

original: left join Customer

optimized: inner join Customer

The second may be faster in some cases but can remove unmatched business records.

Optimization must preserve:

  • row semantics;
  • null behavior;
  • aggregation semantics;
  • currency semantics;
  • hierarchy semantics;
  • security semantics.

12.36 Performance and security​

Security filters can affect execution plans and data volume.

Do not remove security logic for performance.

Instead ask:

  • Is authorization applied at the right layer?
  • Is sensitive data unnecessarily replicated?
  • Is the security model compatible with persistence?
  • Is the persisted result still protected correctly?

Performance and security are jointly engineered.

12.37 Persistence with Data Access Controls​

Persisted data must continue to satisfy the intended authorization behavior.

Before persisting a secured model, validate:

  • which layer enforces access;
  • whether persistence changes execution behavior;
  • whether consumers see the same authorized result;
  • whether data is physically materialized in a way that changes risk.

Use current SAP documentation for the exact Data Access Control and persisted-view combination in your tenant release.

12.38 Performance anti-pattern — giant semantic view​

15 sources | 20 joins | 30 calculations | 10 aggregations | one giant view

This can be difficult to optimize and can repeatedly recompute expensive logic.

Better:

source-aligned | reusable conformed | reusable semantic | analytic

Persist only where evidence justifies it.

12.39 Performance anti-pattern — persistence everywhere​

View A -> persisted View B -> persisted View C -> persisted View D -> persisted View E -> persisted

This creates an ETL-like chain without necessarily improving the architecture.

Persist boundaries that provide:

  • reuse;
  • isolation;
  • performance;
  • recovery;
  • stable contracts.

12.40 Performance anti-pattern — SELECT all columns​

Wide intermediate datasets increase data movement and processing.

Prefer explicit column selection.

12.41 Performance anti-pattern — DISTINCT as a repair​

If duplicates originate from an incorrect join, DISTINCT only hides the symptom and can add expensive processing.

Fix the grain and relationship.

12.42 Performance anti-pattern — partition by habit​

Partitioning every large table by date is not automatically correct.

If the workload rarely filters by date, the partition may provide little value.

Choose based on access patterns.

12.43 Performance anti-pattern — optimize without baseline​

A change from 60 seconds to 40 seconds sounds good.

But if the workload changed from 100 million to 10 million rows, the comparison is meaningless.

Keep the workload constant.

12.44 Performance scorecard​

Score each model:

DimensionScore 1–5
Data volume control
Projection
Filtering
Join quality
Aggregation placement
Federation strategy
Persistence strategy
Partitioning
Concurrency
Freshness
Observability
Cost

Anything below 4 requires investigation.

12.45 Architect decision tree​

Query slow | +--> Remote source? | | | +--> Yes -> federation / replication decision | +--> Excessive rows? | | | +--> filter / projection | +--> Expensive join? | | | +--> grain / cardinality / aggregation | +--> Repeated expensive computation? | | | +--> persistence candidate | +--> Huge local table? | | | +--> partitioning / storage review | +--> Concurrency issue? | | | +--> workload isolation / model redesign | +--> Memory pressure? | +--> reduce volume / complexity / persistence strategy

12.46 Enterprise reference architecture​

SAP / NON-SAP SOURCES | v ACQUISITION federation / replication | v SOURCE-ALIGNED PERSISTENCE | v PREPARATION transformation flows / views | v CONFORMED DATA | v SEMANTIC MODELS | v ANALYTIC MODELS | v CONSUMPTION

Performance controls surround the architecture:

volume control filter pushdown projection persistence partitioning monitoring workload isolation reconciliation

12.47 Hands-on Lab 1 — Federation baseline​

Use a supported remote table.

Record:

  • query duration;
  • source row volume;
  • selected columns;
  • concurrency;
  • freshness.

Do not optimize yet.

12.48 Hands-on Lab 2 — Remote-table reduction​

Start with a wide remote table.

  1. remove genuinely unnecessary columns;
  2. add a safe filter;
  3. redeploy;
  4. repeat the same workload;
  5. compare volume and performance;
  6. validate dependent objects.

Document the trade-off.

12.49 Hands-on Lab 3 — Replication comparison​

Compare:

federation

versus:

replicated remote table

Measure:

  • first access;
  • repeated access;
  • refresh time;
  • source impact;
  • local storage;
  • freshness.

12.50 Hands-on Lab 4 — Persist a complex view​

Create a deliberately expensive view.

Measure baseline performance.

Persist the view.

Refresh it.

Measure again.

Then evaluate:

  • latency;
  • refresh cost;
  • freshness;
  • storage;
  • operational complexity.

12.51 Hands-on Lab 5 — Partition experiment​

Create a large local table suitable for partitioning.

Compare:

no partition

versus:

range partition

and, where appropriate:

hash partition

Use the same workload and record results.

12.52 Hands-on Lab 6 — View Analyzer investigation​

Select a representative complex view.

Use View Analyzer to inspect:

  • consumed entities;
  • local tables;
  • remote sources;
  • expensive components.

Produce an optimization proposal before changing anything.

12.53 Hands-on Lab 7 — Join explosion​

Create:

Sales | +--> Customer

Then deliberately introduce a one-to-many history table.

Observe row multiplication.

Fix the model by defining the correct business grain.

12.54 Hands-on Lab 8 — Persistence placement​

Create three candidate persistence points.

For each, score:

  • reuse;
  • computation cost;
  • freshness;
  • storage;
  • recovery;
  • downstream impact.

Choose one and defend the decision.

12.55 Hands-on Lab 9 — Concurrency experiment​

Run a representative workload under increasing concurrency.

Record:

ConcurrencyMedianSlowestResource symptomFailure
1
10
50
100

Use only levels appropriate to your environment.

12.56 Hands-on Lab 10 — Performance review​

Take one production-like model and produce a one-page performance review:

  • baseline;
  • bottleneck;
  • evidence;
  • change;
  • result;
  • freshness impact;
  • cost impact;
  • rollback;
  • recommendation.

12.57 Troubleshooting — slow dashboard​

Trace:

Dashboard | v Analytic Model | v Semantic View | v Consumed Views | v Local / Remote Tables

At each layer ask:

  • How much data?
  • How many joins?
  • Remote or local?
  • Persisted or virtual?
  • Filtered?
  • Partitioned?
  • Repeated computation?
  • Concurrent?

12.58 Troubleshooting — persisted view is stale​

Check:

  1. persistence schedule;
  2. last successful run;
  3. source freshness;
  4. failed persistence task;
  5. dependency readiness;
  6. consumer expectation;
  7. time-zone assumptions.

Do not immediately rebuild the model.

12.59 Troubleshooting — memory failure​

Use:

Memory failure | v Identify object | v Identify operation | v Reduce columns | v Reduce rows | v Review joins | v Review aggregation | v Review persistence run mode | v Review concurrency

Then re-test.

12.60 Troubleshooting — partition is ineffective​

Check:

  • query predicate;
  • partition key;
  • partition distribution;
  • partition size;
  • data growth;
  • workload access pattern.

If the workload does not benefit from the partition key, redesign rather than adding more partitions.

12.61 Troubleshooting — replication is fast but dashboard is slow​

This is a crucial distinction.

Acquisition fast | v Dashboard slow

The bottleneck is downstream.

Investigate:

  • semantic model;
  • joins;
  • calculated logic;
  • persistence;
  • analytic model;
  • consumption query;
  • concurrency.

Do not tune replication when replication is not the bottleneck.

12.62 Troubleshooting — dashboard is fast but pipeline is slow​

Reverse problem:

Pipeline slow | v Dashboard fast

Investigate:

  • source extraction;
  • transformation;
  • target write;
  • task scheduling;
  • partitioning;
  • delta strategy;
  • unnecessary data movement.

Performance is end-to-end.

12.63 Cost engineering​

Performance and cost are related but not identical.

A faster design can cost more.

For each optimization ask:

Faster? | +--> At what storage cost? | +--> At what compute cost? | +--> At what freshness cost? | +--> At what operational cost?

Architectural optimization is multi-objective optimization.

12.64 Capacity planning​

Estimate:

  • current data volume;
  • daily growth;
  • peak load;
  • retention;
  • concurrency;
  • refresh frequency;
  • transformation volume;
  • storage footprint.

Example:

Current = 5 TB Growth = 100 GB/day Retention = 5 years

A simple linear estimate is:

100 GB x 365 x 5 = 182,500 GB

Before using this number for budgeting, account for compression, actual retention policy, replicas, intermediate data, partitioning, and workload behavior.

12.65 Performance regression management​

Performance must be tested after model changes.

Track:

ReleaseQueryBaselineNewDeltaDecision
R1Sales
R2Sales
R3Sales

A model that becomes 10 percent slower every release eventually becomes a production incident.

12.66 Performance ownership​

Define:

  • platform owner;
  • model owner;
  • data owner;
  • source owner;
  • consumer owner.

A performance problem crossing system boundaries needs coordinated ownership.

12.67 Production readiness checklist​

Data access​

  • Federation versus replication explicitly decided
  • Remote-table volume minimized where appropriate
  • Unnecessary columns removed
  • Filters reviewed
  • Source-system impact assessed

Modeling​

  • Grain documented
  • Join cardinality documented
  • Aggregation semantics validated
  • Projection reviewed
  • Expensive calculations identified

Persistence​

  • Persistence points justified
  • Freshness SLA defined
  • Refresh schedule defined
  • Failure handling defined
  • Storage impact assessed

Partitioning​

  • Large tables reviewed
  • Partition key justified
  • Distribution assessed
  • Query pruning tested
  • Growth monitored

Performance​

  • Baseline captured
  • View Analyzer used where relevant
  • Bottleneck identified
  • Optimization measured
  • Regression test defined
  • Concurrency considered

Operations​

  • Monitoring configured
  • Alerts defined
  • Ownership defined
  • Recovery documented
  • Cost reviewed

12.68 Interview preparation​

When should you persist a view?​

When repeated computation is expensive and persistence provides a measurable performance or reuse benefit while the resulting freshness and operational cost remain acceptable.

Is persistence always faster?​

No. It trades repeated computation for refresh work, storage, and freshness latency.

When should a remote table be replicated?​

When local access, source protection, performance, or workload isolation justifies the replication and its maintenance cost.

Why partition a table?​

To manage large datasets and potentially reduce the amount of data participating in workloads when the partition design matches access patterns.

Does partitioning guarantee faster queries?​

No. Poor partition keys or workloads that do not align with the partition can provide little benefit.

What is View Analyzer?​

A Datasphere analysis tool that examines the entities composing a view and provides information useful for performance optimization and persistence decisions.

Why is SELECT all columns bad?​

It increases data volume and can make downstream processing more expensive.

Why not use DISTINCT to solve duplicates?​

Because it hides the underlying grain or join problem and can add expensive processing.

What is the biggest performance mistake?​

Optimizing without identifying the bottleneck and without a baseline.

12.69 Architect exercise — Enterprise Procurement​

Design a procurement platform with:

  • 500 million purchase-order items;
  • hourly S/4HANA updates;
  • 15-minute analytical freshness;
  • 50 concurrent analytical users;
  • five years of history;
  • supplier and material dimensions;
  • SAC consumption.

Decide:

  1. federation or replication;
  2. landing strategy;
  3. persistence boundaries;
  4. partition strategy;
  5. transformation runtime;
  6. concurrency strategy;
  7. freshness architecture;
  8. monitoring;
  9. recovery;
  10. cost controls.

Defend every choice.

12.70 Architect exercise — Finance close​

Design for:

  • daily financial close;
  • late postings;
  • audit requirements;
  • restricted financial access;
  • historical reporting;
  • predictable dashboard response.

Your design must explicitly balance:

correctness freshness performance auditability security cost

12.71 Architect exercise — 100x growth​

Today:

2 TB 20 million rows/day

Future:

200 TB 2 billion rows/day

Do not simply scale every setting.

Revisit:

  • acquisition;
  • replication;
  • storage;
  • partitioning;
  • transformation;
  • persistence;
  • retention;
  • concurrency;
  • semantic model;
  • consumption;
  • cost.

The architecture must evolve with the workload.

12.72 Architect decision matrix​

QuestionDecision
Source location
Federation acceptable?
Replication required?
Local persistence required?
View persistence candidate?
Freshness SLA
Data volume
Growth rate
Partition strategy
Expected concurrency
Bottleneck evidence
Runtime choice
Storage strategy
Monitoring
Recovery
Cost envelope
Owner

If the answer is based only on habit, it is not an architecture decision.

12.73 Architect takeaway​

The performance hierarchy is:

REDUCE DATA | v REDUCE WORK | v EXECUTE AT THE RIGHT LOCATION | v REUSE EXPENSIVE RESULTS | v PARTITION WHEN THE WORKLOAD BENEFITS | v MEASURE | v REGRESS | v GOVERN

The best performance optimization is often the computation you never perform.

The second-best is the data you never move.

The third is the expensive result you compute once and safely reuse.

That is the foundation for enterprise-scale Datasphere performance engineering.

Official SAP documentation anchors​