Skip to main content

07. Databricks Compute

Compute is the processing capacity used to execute notebooks, jobs, SQL queries, streaming workloads, and other data applications.

Data is durable; compute is workload-specific.

Common compute categories​

Interactive compute​

Useful for development, exploration, debugging, and interactive analysis.

Job compute​

Used for automated production workloads.

SQL warehouses​

Optimized for SQL analytics and BI workloads.

Serverless​

Serverless options abstract infrastructure management for supported workloads.

Photon​

Photon is Databricks' native vectorized execution engine designed to accelerate supported SQL and DataFrame workloads.

Compute selection​

Identify workload
      │
 ┌────┼────────────┐
 ▼    ▼            ▼
SQL  Batch      Interactive
 │     │            │
SQL  Job         Dev/Explore
WH   Compute      Compute

Evaluate input size, transformation complexity, shuffle volume, concurrency, memory, CPU, I/O, SLA, Photon support, and cost.

Cost optimization​

Good cost optimization starts with workload design:

  • Incremental processing.
  • Efficient filtering.
  • Appropriate file layout.
  • Reduced shuffle.
  • Ephemeral job compute.
  • Autoscaling where appropriate.
  • Monitoring workload utilization.
  • Avoiding unnecessarily large interactive compute.

Architect question​

Why separate storage and compute?

Because durable data can remain available independently while processing resources scale according to workload requirements.

Next​

Continue with the next topic in 01. Databricks Foundations.