Skip to main content

16. Performance Engineering & Cost Optimization

Learning objective: design Databricks workloads that meet business performance targets at the lowest sustainable total cost — by controlling data layout, query plans, compute selection, concurrency, storage maintenance, serverless usage, workload shape and FinOps governance.

Phase 15 taught you how to operate a Databricks platform.

This phase teaches you how to engineer its efficiency.

The enterprise architect's question is not:

"How do I make this query faster?"

It is:

"What business outcome must this workload deliver, what latency and throughput does it require, what architecture can meet that SLO, and what is the lowest sustainable cost of doing so?"

Performance and cost are therefore two sides of the same engineering problem.

A workload that finishes in 30 seconds but costs 20× more than necessary is not well engineered.

A workload that is extremely cheap but misses a regulatory deadline is also not well engineered.

The target is fit-for-purpose performance, reliability and unit economics.

Official starting points:

1. Performance engineering is an architecture discipline​

Performance problems rarely belong to one layer.

A slow dashboard can be caused by:

  • excessive data scanned;
  • poor table layout;
  • a suboptimal join strategy;
  • skew;
  • too many small files;
  • a queueing bottleneck;
  • an undersized warehouse;
  • expensive transformations;
  • stale statistics;
  • unnecessary recomputation;
  • a downstream dependency;
  • network or storage behavior.

Therefore investigate from the outside in:

BUSINESS SLO
    |
    v
SERVICE LATENCY
    |
    v
QUERY / JOB LATENCY
    |
    v
SPARK PLAN
    |
    v
DATA ACCESS
    |
    v
FILE / TABLE LAYOUT
    |
    v
COMPUTE + CONCURRENCY
    |
    v
CLOUD / STORAGE / NETWORK

Do not start by changing cluster size.

First identify where time and money are actually being spent.

2. The performance-cost triangle​

Every workload has three competing dimensions:

DimensionQuestion
PerformanceHow quickly and predictably must it finish?
ReliabilityWhat happens when load, data volume or dependencies change?
CostHow much resource consumption is acceptable?

A fourth dimension matters at enterprise scale:

DimensionQuestion
OperabilityCan the team understand, govern and continuously tune it?

A good architecture balances all four.

                 PERFORMANCE
                     /\
                    /  \
                   /    \
                  /      \
                 /        \
                /__________\
             COST          RELIABILITY
                    |
                    v
               OPERABILITY

Optimizing one dimension in isolation often creates a problem somewhere else.

3. Define the workload contract first​

Before tuning, document:

  • workload owner;
  • workload type;
  • data volume;
  • expected growth;
  • execution frequency;
  • concurrency;
  • freshness requirement;
  • latency target;
  • throughput target;
  • recovery requirement;
  • security classification;
  • monthly cost target.

Example:

Finance reporting workload

  • 8 TB source data;
  • 2 TB incremental change per month;
  • daily processing;
  • 200 concurrent BI consumers;
  • data ready by 06:00;
  • p95 dashboard query under 8 seconds;
  • month-end load may be 4× normal;
  • production cost target defined per successful reporting day.

This converts "slow" into measurable engineering requirements.

4. The optimization hierarchy​

Use this order unless evidence tells you otherwise:

1. REMOVE UNNECESSARY WORK
        |
2. REDUCE DATA READ
        |
3. IMPROVE DATA LAYOUT
        |
4. IMPROVE QUERY PLAN
        |
5. IMPROVE COMPUTE EFFICIENCY
        |
6. IMPROVE CONCURRENCY / SCALING
        |
7. TUNE CODE
        |
8. TUNE COST CONTROLS

The biggest optimization is often the work you never execute.

Examples:

  • don't scan historical partitions unnecessarily;
  • don't rebuild unchanged data;
  • don't select unused columns;
  • don't materialize intermediate data that is never consumed;
  • don't run maintenance jobs already handled by managed optimization;
  • don't keep oversized compute running when demand is intermittent.

5. Databricks' current optimization direction​

Current Databricks guidance recommends several high-impact defaults for many workloads:

  • liquid clustering for table layout;
  • Unity Catalog managed tables where appropriate;
  • Photon-enabled compute;
  • deletion vectors for supported update/delete/merge patterns;
  • current Databricks Runtime versions;
  • automatic optimizations such as dynamic file pruning, adaptive query execution and predictive I/O;
  • predictive optimization for eligible Unity Catalog managed tables.

These capabilities do not make every workload automatically optimal. Workload shape, table design, query patterns and configuration still matter.

Official reference: https://docs.databricks.com/aws/en/optimizations

6. Read less data​

The first performance question is:

How many bytes did the workload actually need to read?

A query that reads 5 TB to return 50 MB is a data-access problem before it is a compute problem.

Reduce data scanned with​

  • selective predicates;
  • column pruning;
  • data skipping;
  • liquid clustering;
  • dynamic file pruning;
  • appropriate table design;
  • incremental processing;
  • avoiding unnecessary full-table scans.

Example:

SELECT
    customer_id,
    revenue
FROM gold.sales
WHERE fiscal_year = 2026
  AND region = 'EMEA';

This is generally preferable to:

SELECT *
FROM gold.sales;

especially when downstream logic only needs two columns.

7. Column pruning​

Columnar storage is powerful because engines can avoid reading unused columns.

Bad pattern:

SELECT *
FROM silver.customer_transactions;

Better:

SELECT
    customer_id,
    transaction_date,
    amount,
    currency
FROM silver.customer_transactions;

The principle applies to:

  • SQL;
  • DataFrame transformations;
  • joins;
  • aggregations;
  • views;
  • downstream materializations.

Do not carry dozens of unused columns through the entire DAG.

8. Predicate pushdown and data skipping​

A selective predicate allows the engine to avoid unnecessary work.

The goal is:

QUERY
  |
  v
PREDICATE
  |
  v
SKIP IRRELEVANT FILES
  |
  v
READ ONLY CANDIDATE DATA
  |
  v
PROCESS

Data skipping is especially valuable when table layout correlates with common filter patterns.

However, do not invent a clustering strategy from a single query.

Use workload evidence.

9. Liquid clustering​

Liquid clustering is the current Databricks approach for flexible table layout and replaces the need to design traditional static partitioning and Z-ordering for many use cases.

It organizes data around clustering columns so that commonly filtered data can be located efficiently.

Example:

CREATE TABLE sales (
    sale_id BIGINT,
    customer_id BIGINT,
    sale_date DATE,
    region STRING,
    amount DECIMAL(18,2)
)
CLUSTER BY (sale_date, region);

The correct clustering columns depend on:

  • filter patterns;
  • join patterns;
  • data distribution;
  • table size;
  • write patterns;
  • workload evolution.

Do not automatically cluster on every high-cardinality column.

Official reference: https://docs.databricks.com/aws/en/tables/clustering

10. Liquid clustering is not free​

Clustering requires work.

The architect must consider:

  • data rewrite activity;
  • maintenance frequency;
  • workload write rate;
  • table size;
  • incremental clustering behavior;
  • predictive optimization interaction;
  • expected query benefit.

If predictive optimization manages the table, avoid creating redundant scheduled OPTIMIZE jobs without a clear reason.

Current Databricks guidance recommends disabling scheduled OPTIMIZE jobs for tables where predictive optimization is being used.

11. Partitioning: use the modern decision tree​

Do not use partitioning because:

"Large tables should always be partitioned."

That is an oversimplification.

Use the table's workload and current Databricks guidance.

For modern Delta workloads, liquid clustering is often the preferred layout mechanism.

Traditional partitioning may still be appropriate for specific scenarios, but it should be justified by:

  • predictable partition pruning;
  • suitable partition cardinality;
  • stable access patterns;
  • operational requirements.

Avoid millions of tiny partitions.

12. Small files are a system problem​

A table with millions of tiny files creates:

  • metadata overhead;
  • scheduling overhead;
  • more file-open operations;
  • inefficient scans;
  • longer planning time;
  • expensive maintenance.

The problem is not simply "file count".

The architect must understand:

WRITE PATTERN
    +
FILE SIZE
    +
TABLE LAYOUT
    +
READ PATTERN
    =
FILE EFFICIENCY

Streaming, frequent micro-batches and high-frequency incremental writes can create small-file pressure.

Use current Delta and Databricks maintenance capabilities rather than blindly creating manual compaction jobs.

13. OPTIMIZE: know what problem it solves​

OPTIMIZE reorganizes files to improve read efficiency.

It is not a universal "make everything faster" button.

Before using it, ask:

  1. Is the table actually suffering from file-layout problems?
  2. Is the table eligible for predictive optimization?
  3. How frequently is data changing?
  4. What is the maintenance cost?
  5. What workload benefits?
  6. Is the table read-heavy enough to justify the rewrite work?

For liquid-clustered tables, OPTIMIZE is incremental and works on data that needs clustering.

Official reference: https://docs.databricks.com/aws/en/tables/clustering

14. Predictive optimization​

Predictive optimization can automatically perform maintenance such as:

  • OPTIMIZE;
  • VACUUM;
  • ANALYZE/statistics maintenance;

for eligible Unity Catalog managed tables.

This changes the architecture from:

ENGINEER
  |
  +--> schedule OPTIMIZE
  +--> schedule VACUUM
  +--> schedule ANALYZE

toward:

MANAGED TABLE
      |
      v
PREDICTIVE OPTIMIZATION
      |
      +--> layout maintenance
      +--> statistics maintenance
      +--> file lifecycle maintenance

This can reduce operational toil, but it is still a workload cost that should be understood and governed.

Official reference: https://docs.databricks.com/aws/en/optimizations

15. Query plan literacy​

An architect should be able to read a Spark SQL plan.

Look for:

  • scans;
  • filters;
  • projections;
  • joins;
  • aggregations;
  • exchanges;
  • sorts;
  • shuffles;
  • broadcast operations;
  • skew indicators;
  • unexpectedly large intermediate datasets.

A simplified mental model:

SCAN
 |
FILTER
 |
JOIN
 |
SHUFFLE
 |
AGGREGATE
 |
WRITE

The expensive operator is not necessarily the one with the largest wall-clock duration. Understand the dependencies and the amount of data flowing between stages.

16. Shuffle is expensive​

Shuffle moves data across executors.

Common causes:

  • joins;
  • groupBy;
  • distinct;
  • repartition;
  • window functions;
  • aggregations;
  • sorts.

Conceptually:

EXECUTOR 1 ----\
EXECUTOR 2 -----+----> SHUFFLE ----> NEW PARTITIONS
EXECUTOR 3 ----/

Shuffle can introduce:

  • network traffic;
  • serialization;
  • disk spill;
  • task imbalance;
  • additional stages.

Do not try to eliminate all shuffle. Some correct algorithms require it.

The goal is to make required shuffle efficient and proportional to the workload.

17. Broadcast joins​

When one side of a join is sufficiently small, broadcasting can avoid a large shuffle of the other side.

Conceptually:

SMALL DIMENSION
       |
       +----> Executor 1
       +----> Executor 2
       +----> Executor 3
       +----> Executor 4

LARGE FACT
       |
       +----> processed locally with broadcast dimension

But never use a broadcast hint blindly.

A table that is small today may become large tomorrow.

Use:

  • statistics;
  • runtime evidence;
  • workload growth forecasts;
  • Spark UI/query plans.

18. Adaptive Query Execution​

Adaptive Query Execution (AQE) allows Spark to adjust aspects of the execution plan using runtime information.

It can help with problems such as:

  • inaccurate size estimates;
  • inefficient join strategies;
  • excessive shuffle partitions;
  • skewed workloads;
  • small output partitions.

AQE is valuable because real-world data is often different from estimates.

The architect should still understand the original and final plans.

Automation is not a replacement for plan literacy.

19. Data skew​

Skew occurs when a small number of keys contain a disproportionate amount of data.

Example:

customer_id = 1001  ->  4.2 TB
customer_id = 1002  ->  20 MB
customer_id = 1003  ->  18 MB
...

One task may receive vastly more work than others.

Symptoms:

  • most tasks finish quickly;
  • a few tasks remain running;
  • long tail execution;
  • executor spill;
  • poor parallelism.

Investigate:

  • key distribution;
  • join keys;
  • aggregation keys;
  • data quality;
  • business meaning of hot keys.

Do not solve skew by simply adding more workers if the bottleneck is a single hot partition.

20. Photon​

Photon is Databricks' native vectorized execution engine for supported workloads.

Its purpose is not simply "use more CPU".

It is designed to accelerate supported SQL and DataFrame workloads through optimized execution.

Evaluate Photon using:

  • representative workload benchmarks;
  • query latency;
  • throughput;
  • DBU consumption;
  • total cost per business outcome.

The correct comparison is not:

Photon DBU price vs non-Photon DBU price.

It is:

Total cost and performance required to deliver the same business result.

21. Benchmark correctly​

A benchmark must be representative.

Control:

  • data volume;
  • data distribution;
  • cache state;
  • query parameters;
  • concurrency;
  • compute configuration;
  • runtime version;
  • table layout;
  • workload frequency.

Measure:

  • p50 latency;
  • p95 latency;
  • p99 latency;
  • throughput;
  • DBUs;
  • wall-clock time;
  • cost;
  • failure/retry behavior.

A single warm-cache query is not a production benchmark.

22. Performance per dollar​

A useful metric is:

PERFORMANCE EFFICIENCY
=
BUSINESS WORK COMPLETED
-----------------------
TOTAL COST

Examples:

  • rows processed per dollar;
  • successful pipeline runs per dollar;
  • reports delivered per dollar;
  • TB transformed per dollar;
  • queries meeting SLO per dollar.

This is more useful than simply asking:

"How many DBUs did we consume?"

23. Compute selection​

Compute selection should follow workload characteristics.

WorkloadKey considerations
Batch ETLruntime, scaling, startup, throughput
Streamingsustained processing rate, state, latency
Interactive SQLconcurrency, latency, queueing
BI dashboardsconcurrency, cache behavior, predictable latency
ML trainingaccelerator requirements, data throughput
Developmentisolation, cost controls, auto-termination
Serverlesssimplicity, startup, managed operations, usage economics

Do not use one compute configuration for every workload.

24. Right-size before scaling​

If a workload is slow, distinguish:

TOO LITTLE COMPUTE
        vs
TOO MUCH WORK
        vs
BAD PARALLELISM
        vs
BAD DATA LAYOUT
        vs
QUEUEING
        vs
SKEW

Adding workers to a bad plan can make an expensive bad plan execute faster.

That may improve latency while worsening unit economics.

25. Scale out versus scale up​

A workload can often improve by:

  • adding workers;
  • using larger workers;
  • changing concurrency;
  • changing execution engine;
  • reducing data scanned;
  • changing algorithm.

Choose based on the bottleneck.

If the workload is embarrassingly parallel, scale-out may help.

If the workload is memory-bound or dominated by a small number of tasks, scaling out may not solve it.

26. Autoscaling is a control system​

Autoscaling is not simply:

"Turn on autoscaling."

Think of it as:

DEMAND
  |
  v
WORKLOAD QUEUE / UTILIZATION
  |
  v
SCALING SIGNAL
  |
  v
RESOURCE CHANGE
  |
  v
NEW CAPACITY
  |
  v
OBSERVED PERFORMANCE
  |
  +----> feedback loop

Important questions:

  • How quickly does capacity respond?
  • What is the minimum capacity?
  • What is the maximum?
  • What happens during a burst?
  • Is the workload sensitive to startup time?
  • Does scale-out actually reduce the bottleneck?
  • What is the cost impact?

27. Serverless changes the operating model​

Serverless reduces infrastructure-management responsibility for supported workloads.

The architect still owns:

  • workload efficiency;
  • concurrency;
  • query design;
  • cost attribution;
  • budgets;
  • tags/policies;
  • data access;
  • business SLOs.

Serverless does not mean costless.

Databricks recommends benchmarking representative workloads and analyzing system.billing.usage for serverless cost. Serverless usage can appear in billing records with workload metadata, and usage data can be delayed.

Official reference: https://docs.databricks.com/aws/en/compute/serverless

28. Serverless cost attribution​

For serverless workloads, use available billing metadata such as:

  • usage_metadata;
  • identity_metadata;
  • custom_tags;
  • billing_origin_product;
  • product_features.

A useful enterprise question is:

Which business domain consumed the DBUs?

not merely:

Which workspace consumed the DBUs?

Use policies and consistent tagging to make this attribution possible.

Official reference: https://docs.databricks.com/aws/en/admin/system-tables/serverless-billing

29. The billing system table​

The primary analytical source for actual billable usage is:

system.billing.usage

Important concepts include:

  • usage date/time;
  • usage quantity;
  • usage unit;
  • SKU;
  • workspace metadata;
  • identity metadata;
  • workload metadata;
  • custom tags;
  • billing origin product;
  • product features.

Current Databricks documentation notes that corrections can appear as retraction/restatement records. Aggregate usage carefully rather than assuming every record is an independent final charge.

Official reference: https://docs.databricks.com/aws/en/admin/system-tables/billing

30. DBU is not total cloud cost​

This distinction is critical.

A Databricks workload can incur:

  • Databricks DBUs;
  • cloud compute/storage charges where applicable;
  • object storage;
  • network transfer;
  • external services;
  • marketplace or third-party services.

Therefore:

TOTAL COST
=
DATABRICKS USAGE
+
CLOUD INFRASTRUCTURE
+
NETWORK
+
STORAGE
+
EXTERNAL SERVICES

The exact components depend on architecture and deployment model.

Never present DBU consumption as total TCO without qualification.

31. Billing origin product​

Current billing records expose billing_origin_product, allowing cost analysis by product category.

Examples include:

  • JOBS;
  • DLT;
  • SQL;
  • ALL_PURPOSE;
  • MODEL_SERVING;
  • INTERACTIVE;
  • DEFAULT_STORAGE;
  • VECTOR_SEARCH;
  • LAKEHOUSE_MONITORING;
  • PREDICTIVE_OPTIMIZATION.

The exact available values evolve as the platform evolves.

Do not hard-code an old list into an enterprise FinOps model without checking the current reference.

32. A practical cost query​

A simple starting point:

SELECT
    usage_date,
    billing_origin_product,
    SUM(usage_quantity) AS usage_quantity
FROM system.billing.usage
WHERE usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
    usage_date,
    billing_origin_product
ORDER BY
    usage_date DESC,
    usage_quantity DESC;

This is a usage view, not a complete invoice calculation.

For financial reporting, enrich usage with the applicable pricing and contractual model.

33. Cost allocation model​

A mature enterprise should allocate cost through a hierarchy:

ACCOUNT
  |
  v
WORKSPACE
  |
  v
ENVIRONMENT
  |
  v
DOMAIN / COST CENTER
  |
  v
PRODUCT
  |
  v
WORKLOAD
  |
  v
JOB / QUERY / MODEL

Not every workload can be attributed perfectly.

The architect should explicitly document:

  • attribution precision;
  • unallocated cost;
  • shared platform cost;
  • overhead;
  • allocation rules.

False precision is worse than transparent approximation.

34. Tagging strategy​

A minimum enterprise tagging model might include:

TagExample
environmentprod
domainfinance
cost_centerFIN-001
applicationenterprise-reporting
ownerfinance-data-eng
criticalitytier1
data_classificationconfidential

Keep tag names stable.

A tagging standard is an architecture control, not a cosmetic naming convention.

35. Budgets and guardrails​

Databricks budgets can monitor account spending and support usage controls for supported products.

Important distinction:

  • budget monitoring tells you how spending compares with a threshold;
  • system.billing.usage remains the source of truth for actual billable usage;
  • budget estimates and billing records can update at different rates.

Do not design a financial control that assumes every budget number is an instantaneous invoice value.

Official reference: https://docs.databricks.com/aws/en/admin/account-settings/budgets

36. Cost anomaly detection​

A strong FinOps system looks for:

  • sudden DBU increase;
  • unexpected serverless growth;
  • idle compute;
  • retry storms;
  • runaway queries;
  • excessive all-purpose usage;
  • unexpected model serving traffic;
  • unusual storage growth;
  • expensive maintenance;
  • workload growth without business-volume growth.

The alert should identify:

  1. what changed;
  2. by how much;
  3. which workload caused it;
  4. who owns it;
  5. what action is expected.

37. Cost of retries​

Retries are often hidden cost.

Example:

NORMAL RUN = 100 DBU
3 RETRIES = 300 DBU
TOTAL = 400 DBU

If the root cause is a deterministic data or code defect, increasing retry count can multiply cost without improving reliability.

Design:

  • bounded retries;
  • exponential backoff where appropriate;
  • idempotent processing;
  • failure classification;
  • alerting on repeated retries.

38. Cost of over-partitioning​

Over-partitioning can increase:

  • file counts;
  • metadata;
  • small files;
  • maintenance;
  • planning overhead;
  • write amplification.

The cost may appear in multiple workloads:

BAD TABLE DESIGN
     |
     +--> WRITE COST
     +--> MAINTENANCE COST
     +--> QUERY COST
     +--> STORAGE COST
     +--> OPERATIONS COST

This is why table layout is both a performance and FinOps decision.

39. Cost of unnecessary data copies​

Every unnecessary copy can create:

  • storage cost;
  • write cost;
  • read cost;
  • network cost;
  • maintenance cost;
  • governance complexity.

Before creating a duplicate dataset, ask:

  • Is the copy required for performance?
  • Is it required for isolation?
  • Is it required for regulatory retention?
  • Can a governed view or shared table satisfy the use case?
  • Can the workload use the existing managed table efficiently?

40. Storage lifecycle engineering​

Storage cost should be managed through lifecycle design.

Consider:

  • retention;
  • historical versions;
  • temporary data;
  • raw ingestion;
  • intermediate datasets;
  • abandoned tables;
  • old checkpoints;
  • logs;
  • model artifacts.

Never apply destructive retention rules merely to reduce cost.

Retention must respect:

  • legal requirements;
  • business retention;
  • recovery requirements;
  • audit obligations;
  • Delta table semantics.

41. VACUUM and retention​

VACUUM removes old data files that are no longer needed under the configured retention policy.

The architect must understand the relationship between:

  • time travel;
  • concurrent readers;
  • retention;
  • recovery;
  • legal requirements.

A lower retention period can reduce storage but can also reduce historical recovery capability.

Cost optimization must not silently weaken recoverability.

42. SQL warehouse efficiency​

For SQL workloads monitor:

  • query latency;
  • queue time;
  • concurrency;
  • warehouse utilization;
  • bytes scanned;
  • expensive queries;
  • workload mix;
  • dashboard behavior.

A common mistake is increasing warehouse size whenever queries are slow.

First determine:

QUERY PROBLEM
      OR
WAREHOUSE CAPACITY PROBLEM
      OR
CONCURRENCY PROBLEM
      OR
DATA LAYOUT PROBLEM

43. Materialization is a trade-off​

Materializing a result can improve latency.

But it creates:

  • storage;
  • refresh cost;
  • additional pipeline complexity;
  • freshness considerations.

Use materialization when the repeated computation cost exceeds the maintenance cost and the business requires predictable performance.

The question is:

RECOMPUTE COST
      vs
MATERIALIZE + REFRESH COST

44. Caching is not an architecture substitute​

Caching can improve repeated access.

But:

  • cache behavior can change;
  • data freshness matters;
  • cache misses can be expensive;
  • cache-heavy benchmarks can hide cold-start behavior.

Use caching to accelerate a good architecture, not to conceal a poor one.

45. End-to-end optimization example​

Suppose a finance dashboard scans 4 TB and takes 40 seconds.

Do not immediately double the warehouse.

Investigate:

  1. Does the query read unnecessary columns?
  2. Can predicates prune data?
  3. Is table layout aligned with filters?
  4. Is liquid clustering appropriate?
  5. Are joins causing large shuffles?
  6. Are statistics available?
  7. Is AQE helping?
  8. Is Photon appropriate?
  9. Is warehouse queueing involved?
  10. Is concurrency the real problem?

A successful optimization might look like:

4 TB SCANNED
   |
   +--> COLUMN PRUNING
   |
   +--> BETTER TABLE LAYOUT
   |
   +--> FILTER PRUNING
   |
   +--> BETTER JOIN PLAN
   |
   v
300 GB SCANNED
   |
   v
LOWER LATENCY + LOWER COST

The best optimization reduced work rather than simply adding compute.

46. Performance regression engineering​

Performance must be tested continuously.

Create a baseline:

MetricBaselineThreshold
p50 runtime90 sec+20%
p95 runtime150 sec+20%
data scanned800 GB+25%
DBU/run35+20%
output rows2.1 Bexpected range

A release that passes functional tests but increases DBU/run by 80% should not automatically be considered successful.

Performance is part of production quality.

47. Workload classes​

Define workload classes so optimization decisions are consistent.

Tier A — business critical​

  • strict SLO;
  • high reliability;
  • controlled change;
  • aggressive performance monitoring.

Tier B — important​

  • standard SLO;
  • normal operational controls.

Tier C — opportunistic​

  • flexible schedule;
  • cost-sensitive;
  • can tolerate queueing.

Tier D — development​

  • strict cost controls;
  • limited capacity;
  • automatic shutdown where applicable.

The same performance target should not be applied to every workload.

48. FinOps operating model​

A mature operating model has three layers:

Engineering​

Owns:

  • workload efficiency;
  • table design;
  • query optimization;
  • compute configuration.

Platform​

Owns:

  • policies;
  • tagging;
  • budgets;
  • shared architecture;
  • observability.

Finance / FinOps​

Owns:

  • cost reporting;
  • forecasting;
  • allocation;
  • unit economics;
  • financial governance.

Together:

ENGINEERING
     |
     v
PLATFORM
     |
     v
FINOPS
     |
     v
BUSINESS DECISION

49. Unit economics​

A platform should eventually answer:

  • cost per successful pipeline;
  • cost per TB processed;
  • cost per report delivered;
  • cost per million events;
  • cost per dashboard refresh;
  • cost per model prediction.

Example:

COST PER SUCCESSFUL DAILY DELIVERY
=
MONTHLY DATA PLATFORM COST
---------------------------
SUCCESSFUL BUSINESS DELIVERIES

This turns infrastructure usage into a business metric.

50. What not to optimize​

Avoid optimizing a metric that has no business value.

Examples:

  • reducing runtime from 20 seconds to 15 seconds when the SLO is 60 seconds and the change increases cost 3×;
  • reducing DBUs by 30% while causing missed business deadlines;
  • minimizing storage while violating retention requirements;
  • reducing cluster size while increasing retries;
  • reducing query latency for an unused dashboard.

Optimization must be tied to a business outcome.

51. Anti-patterns​

"Bigger cluster fixes everything"​

Usually false.

"More partitions means more parallelism"​

Not necessarily.

"OPTIMIZE every table every night"​

Can waste compute and create unnecessary rewrites.

"Always broadcast the dimension"​

Can fail as data grows.

"Partition by date because everyone does it"​

Not a universal rule.

"DBU equals cloud cost"​

Incorrect.

"Serverless means free"​

Incorrect.

"Cost dashboard equals FinOps"​

A dashboard without ownership and action is reporting, not FinOps.

"Fastest query is always best"​

Not if it is disproportionately expensive.

"Retry until it works"​

Can create runaway cost and duplicate processing.

52. Performance investigation playbook​

When a workload becomes slow:

Step 1 — establish the regression​

Compare:

  • current vs baseline;
  • p50/p95;
  • input volume;
  • data distribution;
  • compute;
  • runtime.

Step 2 — inspect data scanned​

Ask:

  • Did volume increase?
  • Did pruning disappear?
  • Did table layout change?

Step 3 — inspect the plan​

Look for:

  • shuffle;
  • skew;
  • join changes;
  • unexpected scans;
  • spills.

Step 4 — inspect compute​

Check:

  • utilization;
  • queueing;
  • scaling;
  • memory;
  • concurrency.

Step 5 — inspect code​

Only now tune transformations and algorithms.

Step 6 — measure cost​

Verify that the optimization improved:

  • latency;
  • throughput;
  • cost;
  • reliability.

53. FinOps investigation playbook​

When cost increases:

  1. Identify the date and magnitude of change.
  2. Break usage down by product.
  3. Break usage down by workspace.
  4. Break usage down by tags/domain.
  5. Identify top workloads.
  6. Compare workload volume.
  7. Check retries and failures.
  8. Check compute changes.
  9. Check table maintenance.
  10. Check serverless activity.
  11. Check model-serving or AI workload growth where relevant.
  12. Identify the owner.
  13. Apply a corrective action.
  14. Measure the result.

Never stop at:

"DBUs increased."

Explain why.

54. Enterprise performance architecture​

                    BUSINESS SLOs
                         |
             +-----------+-----------+
             |                       |
        PERFORMANCE               COST
             |                       |
             v                       v
      WORKLOAD PROFILE         FINOPS MODEL
             |                       |
             +-----------+-----------+
                         |
                         v
                 DATA ARCHITECTURE
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
          LAYOUT       QUERY       COMPUTE
             |           |           |
             +-----------+-----------+
                         |
                         v
                  OBSERVABILITY
                         |
             +-----------+-----------+
             |                       |
             v                       v
        PERFORMANCE               COST
        DASHBOARD                DASHBOARD
             |                       |
             +-----------+-----------+
                         |
                         v
                  OPTIMIZATION LOOP
                         |
                         v
                  MEASURE AGAIN

This loop is continuous.

55. Hands-on lab — query performance engineering​

Choose a deliberately slow SQL workload.

Capture:

  • execution time;
  • data scanned;
  • shuffle;
  • query plan;
  • DBUs/cost proxy;
  • concurrency.

Then optimize in this order:

  1. column pruning;
  2. predicate pruning;
  3. table layout;
  4. join strategy;
  5. statistics;
  6. AQE/runtime capabilities;
  7. compute;
  8. concurrency.

Record before/after results.

Deliverable: a performance engineering report showing evidence for every change.

56. Hands-on lab — Delta table layout​

Create a large synthetic table with:

  • date;
  • region;
  • customer;
  • transaction type;
  • amount.

Create several representative query patterns.

Measure:

  • bytes scanned;
  • runtime;
  • maintenance effort.

Compare a poorly organized layout with a modern liquid-clustered design where appropriate.

Document:

  • clustering columns selected;
  • why they were selected;
  • workload evidence;
  • maintenance trade-offs.

57. Hands-on lab — skew investigation​

Create a skewed dataset where one key owns a large percentage of rows.

Build a join or aggregation.

Observe:

  • task duration distribution;
  • long-tail tasks;
  • shuffle;
  • spill.

Then test a mitigation.

Do not simply increase worker count.

Explain why the mitigation works.

58. Hands-on lab — Photon benchmark​

Run the same representative workload under supported configurations.

Record:

  • runtime;
  • p95;
  • DBUs;
  • cost;
  • throughput.

Calculate:

=
TOTAL COST
------------
SUCCESSFUL RUNS

Choose the architecture based on the complete result, not on raw execution time alone.

59. Hands-on lab — FinOps dashboard​

Use system.billing.usage to create:

  • daily usage trend;
  • usage by billing origin product;
  • usage by workspace;
  • usage by cost center;
  • top workloads;
  • serverless usage;
  • month-to-date usage;
  • month-end projection.

Clearly label:

  • DBU usage;
  • estimated cost;
  • actual billable usage;
  • allocation assumptions.

Do not present an estimate as an invoice.

60. Hands-on lab — cost anomaly​

Simulate:

A production job normally consumes 40 DBU/run. A code change increases it to 180 DBU/run.

Build an investigation:

  1. detect anomaly;
  2. identify job;
  3. compare runtime;
  4. compare input volume;
  5. inspect query plan;
  6. identify root cause;
  7. calculate monthly impact;
  8. decide rollback vs optimization;
  9. verify after remediation.

61. Enterprise architect exercise​

Design the performance and FinOps operating model for:

  • 6 workspaces;
  • 12 data domains;
  • 150 Lakeflow Jobs;
  • 40 pipelines;
  • 25 streaming applications;
  • 20 SQL warehouses;
  • 300 BI consumers;
  • 24×7 critical workloads.

Requirements:

  • Tier-1 finance delivery by 06:00;
  • p95 critical dashboard latency under 8 seconds;
  • critical streaming freshness under two minutes;
  • measurable cost allocation;
  • performance regression detection;
  • workload-level ownership;
  • monthly capacity forecast;
  • budget thresholds;
  • cost anomaly detection;
  • controlled optimization changes.

Produce:

  1. workload classification;
  2. performance SLOs;
  3. benchmark methodology;
  4. table-layout standards;
  5. compute standards;
  6. query optimization standards;
  7. serverless policy;
  8. tagging strategy;
  9. billing model;
  10. cost allocation hierarchy;
  11. budget model;
  12. anomaly model;
  13. performance regression gates;
  14. FinOps review cadence;
  15. exception process.

Defend every design decision in terms of performance, reliability, cost and operability.

62. Troubleshooting matrix​

SymptomFirst investigation
Query suddenly slowerCompare plan, data scanned, layout and input volume
CPU highDetermine whether compute is actually the bottleneck
One task runs much longerInvestigate skew
Huge shuffleInspect joins, aggregations and partitioning
Dashboard queuesInspect concurrency and warehouse capacity
DBUs suddenly spikeBreak usage down by product/workload/tag
Serverless cost unclearInspect system.billing.usage metadata
Table maintenance expensiveCheck predictive optimization and redundant OPTIMIZE jobs
Small-file explosionInspect write pattern and table maintenance
Scaling does not improve runtimeIdentify non-parallel bottleneck or skew
Query reads too much dataImprove pruning and table layout
Cost falls but SLA failsRebalance architecture around the business SLO
Runtime improves but cost explodesMeasure cost per successful outcome
Storage keeps growingReview retention, versions and unnecessary copies

63. Production checklist​

Performance​

  • Business SLO is defined.
  • Baseline p50/p95/p99 is known.
  • Data scanned is measured.
  • Query plans are reviewed for critical workloads.
  • Skew is monitored.
  • Shuffle is understood.
  • Table layout matches access patterns.
  • Compute is right-sized.
  • Concurrency is understood.
  • Performance regression is tested.

Data layout​

  • Liquid clustering is evaluated where appropriate.
  • Traditional partitioning is justified when used.
  • Small-file risk is monitored.
  • Predictive optimization is considered for eligible managed tables.
  • Redundant maintenance jobs are avoided.
  • Retention supports business and recovery requirements.

Cost​

  • DBU usage is monitored.
  • DBU is not confused with total cloud TCO.
  • Cost is attributable to owners/domains.
  • Serverless usage is visible.
  • Budgets are defined.
  • Cost anomalies are detected.
  • Retry costs are measured.
  • Shared platform costs are allocated transparently.
  • Unit economics are reviewed.

Governance​

  • Tags are standardized.
  • Workload owners exist.
  • Optimization changes are reviewed.
  • Financial assumptions are documented.
  • Estimates are clearly distinguished from actual billable usage.

64. Interview questions​

Fundamentals​

  1. What is performance engineering in Databricks?
  2. Why should you reduce data scanned before increasing compute?
  3. What is the relationship between performance and cost?
  4. What is data skipping?
  5. What is liquid clustering?
  6. Why can small files hurt performance?
  7. What is shuffle?
  8. What is data skew?
  9. What does AQE solve?
  10. What is Photon?

Architecture​

  1. How do you choose compute for batch vs SQL vs streaming?
  2. When would you scale up instead of out?
  3. How would you design performance SLOs?
  4. How do you benchmark a workload correctly?
  5. How do you design a performance regression gate?
  6. When should a result be materialized?

FinOps​

  1. What is system.billing.usage?
  2. Why is DBU not the same as total cloud cost?
  3. How do you allocate serverless cost?
  4. How would you design tagging?
  5. How do budgets differ from actual billing data?
  6. How do you investigate a 3× cost increase?
  7. What is cost per successful business outcome?
  8. How do retries affect FinOps?

Architect scenarios​

  1. A query is 40 seconds and the business wants 8 seconds. What do you investigate?
  2. A job is slow after adding workers. Why?
  3. A table is expensive to maintain. How do you investigate?
  4. DBUs doubled but data volume did not. What do you check?
  5. How would you optimize a 10 TB finance workload?
  6. How would you balance a 06:00 business deadline against a cost ceiling?

65. The enterprise optimization mental model​

For every important workload, ask:

WHAT BUSINESS OUTCOME?
        |
        v
WHAT SLO?
        |
        v
HOW MUCH DATA?
        |
        v
HOW MUCH WORK IS ACTUALLY NECESSARY?
        |
        v
HOW IS THE DATA LAID OUT?
        |
        v
WHAT DOES THE QUERY PLAN DO?
        |
        v
WHAT COMPUTE IS REQUIRED?
        |
        v
WHAT DOES IT COST?
        |
        v
CAN WE IMPROVE PERFORMANCE WITHOUT
UNNECESSARY COST?
        |
        v
CAN WE REDUCE COST WITHOUT
BREAKING THE SLO?
        |
        v
MEASURE AGAIN

This is the architect's optimization loop.

66. Final enterprise takeaway​

Performance engineering and cost optimization are not separate activities.

They are one continuous engineering discipline.

A mature Databricks architect can explain:

  • what the workload needs;
  • why it is slow;
  • where resources are consumed;
  • which architectural change will help;
  • how much the change costs;
  • whether the business SLO improves;
  • how the result will be monitored;
  • when the optimization should be revisited.

Architect principle: The best Databricks workload is not the one that uses the most compute or the one with the lowest bill. It is the one that reliably meets the business SLO with the least unnecessary work and the lowest sustainable total cost.

That is enterprise performance engineering and FinOps.

Official references​