Skip to main content

02.4 Compute Architecture

Phase 2 — Platform Architecture

Compute is where the Lakehouse executes. The workspace is the operational boundary, Unity Catalog is the governance layer, cloud storage is the durable data foundation, and compute is the execution layer connecting them.

Databricks compute architecture

1. The mental model​

WORKLOAD │ ▼ COMPUTE MODEL │ ├── Serverless ├── Classic compute └── SQL warehouse │ ▼ EXECUTION ├── Driver / Workers ├── Runtime └── Photon │ ▼ SECURITY + NETWORK │ ▼ UNITY CATALOG → S3 / Delta / Services

Every compute decision should answer: what workload, who manages infrastructure, what isolation is required, what performance is required, and what does it cost?

2. Current compute families​

FamilyPrimary use
Serverless computeManaged notebooks, jobs and pipelines
Classic computeCustomer-configured all-purpose, jobs and pipeline compute
SQL warehousesSQL analytics and BI

Databricks currently recommends serverless for most supported workloads. Classic remains important for specialized requirements. SQL warehouses are optimized for SQL analytics.

Databricks Compute
Compute selection recommendations

3. Serverless compute​

Serverless means Databricks manages the underlying compute infrastructure for the supported workload.

User / Job → Databricks → SERVERLESS COMPUTE → Governed data

Benefits:

  • Less infrastructure administration
  • Automatic provisioning and scaling
  • Faster startup for many workloads
  • Fewer cluster-sizing decisions
  • Easier standardization

Evaluate classic compute when a required feature is unsupported, or when the workload requires specialized runtime, GPU, RDD/R support, custom infrastructure or other capabilities unavailable in serverless.

Do not interpret serverless as no compute. It changes the infrastructure-management model.

4. Classic compute​

Classic compute is customer-configured compute. On AWS, classic resources run in the customer's cloud account.

CLASSIC ├── All-purpose → interactive development ├── Jobs → automated workloads └── Pipeline → pipeline execution

The trade-off is: more control ↔ more infrastructure responsibility.

Classic compute overview

5. All-purpose vs Jobs compute​

All-purposeJobs
Primary purposeInteractiveAutomated
LifecycleUser-controlled / auto-terminationRun-aligned
Typical userEngineer / analystScheduler / service principal
Production defaultAvoid for jobsPreferred when classic is required

All-purpose compute is excellent for notebooks, exploration, debugging and iterative development. Databricks recommends serverless jobs where supported, otherwise classic jobs compute, instead of using shared all-purpose compute for normal production automation.

Configure compute for jobs

6. SQL warehouses​

SQL warehouses are specialized compute for SQL analytics.

BI / SQL / Dashboard │ ▼ SQL Warehouse │ ▼ Gold / Lakehouse

Use them for Databricks SQL, BI, reporting, SQL analytics and SQL-heavy transformations. Current options include serverless, Pro and classic SQL warehouses.

Do not force every SQL workload onto a general Spark cluster. SQL warehouses provide a separate scaling, concurrency and query-execution model.

7. Workload selection matrix​

WorkloadRecommended starting point
Interactive notebookServerless compute
Python ETL jobServerless jobs
SQL BIServerless SQL warehouse
SQL job taskServerless SQL warehouse
Custom Spark / JARClassic jobs
Spark SubmitClassic jobs
GPU MLDedicated/classic where required
R / RDD workloadDedicated/classic
Specialized networkingClassic or appropriate Pro configuration

Always validate current serverless support before production deployment because the feature set changes over time.

8. Driver and workers​

Classic Spark compute follows the familiar distributed model:

DRIVER │ ├── Spark session ├── execution planning ├── task coordination └── application state │ ┌──────┼──────┐ ▼ ▼ ▼ WORKER WORKER WORKER executor executor executor │ │ │ └──────┼──────┘ ▼ Cloud storage

The driver coordinates the application. Executors perform distributed tasks against partitions.

Architectural warning: increasing worker count does not automatically fix driver bottlenecks, skew, excessive shuffle, poor partitioning, small files or inefficient joins.

9. Access modes​

Access modeMeaningTypical use
StandardMulti-user compute with workload isolationMost workloads
DedicatedAssigned to one user or groupSpecialized workloads

Databricks recommends Standard unless required functionality is not supported. Dedicated is appropriate for specific GPU, ML, RDD, R or other compatibility requirements.

Standard compute
Dedicated compute

10. Access mode is not IAM​

These are different control layers:

Identity ↓ Workspace permissions ↓ Access mode / workload isolation ↓ Unity Catalog privileges ↓ Cloud IAM / storage authorization ↓ Network connectivity ↓ S3 / database / API

LayerQuestion
IdentityWho are you?
WorkspaceCan you use this resource?
Access modeHow is execution isolated?
Unity CatalogWhich governed assets can you access?
Cloud IAMWhat cloud resources can execution access?
NetworkCan the resource communicate?

Enterprise security requires these layers to align rather than relying on one control.

11. Photon​

Photon is Databricks' high-performance execution engine for supported workloads.

Spark / SQL → Query plan → Photon or Spark → Result

Photon is especially valuable for SQL, large scans, joins, aggregations and DataFrame transformations. Unsupported operations can fall back to the Spark runtime.

Current documentation identifies limitations involving UDFs, RDD APIs, Dataset APIs and stateful streaming.

Photon accelerates supported execution; it does not replace good Spark design.

What is Photon?

12. Autoscaling​

Autoscaling adjusts worker capacity according to workload demand.

Low load → 2 workers → Peak → 8–12 workers → Low load → 4 workers

Use autoscaling for variable workloads, but do not expect it to solve data skew, bad joins, excessive shuffle, driver pressure or poor file layout.

Think in terms of capacity efficiency, not maximum workers.

13. Instance sizing​

Choose the instance based on the limiting resource:

BottleneckPrioritize
CPU-boundCPU / parallelism
Memory-boundRAM / executor memory
Shuffle-heavyMemory / local disk / network
GPUGPU type / GPU memory
I/O-heavyStorage / network throughput

Start with what is limiting the workload?, not what is the biggest machine?

14. Compute policies​

Policies are enterprise guardrails for compute creation.

Developer → Compute Policy → Approved Compute Policy can enforce: runtime, access mode, instance family, worker limits, auto-termination, tags and other controls.

Useful policy families include development, shared analytics, production jobs, ML/GPU and controlled exceptions.

Policies improve security, cost governance, standardization and operational consistency.

Default policies and policy families

15. Networking​

Classic compute can be a major network-architecture decision.

CONTROL PLANE │ management ▼ CUSTOMER VPC /
DRIVER WORKERS \ / private endpoints │ S3 / SAP / DB / APIs

Enterprise requirements may include private subnets, VPC endpoints, controlled egress, firewall inspection, private connectivity and on-premises access.

Therefore compute architecture and network architecture must be designed together.

16. Compute-to-storage​

An S3-backed Lakehouse typically follows:

Executor → governed authorization → S3 → Delta files + transaction log

Compute is ephemeral. Data is durable.

Compute is disposable. Data is durable.

17. Cost architecture​

Do not model cost as only VM price × hours.

Consider:

  • DBU consumption

  • Cloud instance cost

  • Idle time

  • Autoscaling

  • Runtime efficiency

  • Concurrency

  • Storage

  • Network / egress

  • SQL warehouse sizing

    TOTAL WORKLOAD COST = compute + cloud infrastructure + storage + network + operational overhead

The cheapest hourly compute is not necessarily the cheapest workload. Optimize cost per successful unit of work.

18. Enterprise compute by environment​

DEV ├── Serverless notebooks └── Small standard classic exceptions

TEST ├── Serverless jobs └── Controlled classic jobs

PROD ├── Serverless jobs ├── Serverless SQL └── Policy-controlled classic exceptions

Make the recommended architecture the easiest architecture for developers to use.

19. Shared vs dedicated​

Shared compute improves utilization and collaboration. Dedicated compute is justified when workload requirements need it.

Do not create dedicated compute for everyone by default. Do not create one giant shared cluster for everything either.

The right answer is workload isolation based on real requirements.

20. BI, engineering and ML patterns​

BI​

BI tools → SQL Warehouse → Gold / serving data

Data engineering​

Source → Auto Loader → Bronze → Silver → Gold → BI / ML

Use serverless jobs where supported and classic jobs where custom configuration is required.

Machine learning​

Notebook → CPU/GPU compute → Training → Model artifacts

ML may require dedicated compute, GPUs, specialized runtimes or dependency isolation.

21. Anti-patterns​

One giant cluster for everything​

Creates contention, weak isolation and poor cost attribution.

All-purpose for every production job​

Creates idle cost and the wrong lifecycle model.

Maximum workers by default​

Does not fix skew, bad joins or poor partitioning.

Dedicated for everyone​

Creates low utilization and unnecessary administration.

Ignoring access mode​

Creates security and compatibility surprises.

Treating Photon as a magic switch​

Execution acceleration cannot compensate for bad data layout or query design.

22. Hands-on architecture exercise​

Scenario​

Daily sales pipeline, 10 TB/day, Bronze → Silver → Gold, Power BI dashboards and occasional data-science notebooks.

Design​

  1. Serverless jobs for the pipeline if supported. 2. SQL warehouse for BI. 3. Serverless notebooks for supported data science. 4. Dedicated/classic only for explicit exceptions. 5. Standard access mode by default. 6. Compute policies for cost and security. 7. Unity Catalog for data governance.

Target​

Engineering → Serverless Jobs BI → SQL Warehouse ML → Serverless / Dedicated as required All → Unity Catalog → Delta → S3

23. Troubleshooting decision tree​

WORKLOAD SLOW? │ ▼ CPU saturated? │ │ YES NO │ │ CPU issue Memory/spill high? │ YES → memory / local I/O │ NO → Shuffle high? │ YES → joins / partitioning / skew │ NO → query plan / files / network / external system

Inspect Spark UI, query profile, execution plans, stage duration, task skew, shuffle read/write, spill, input/output size and executor utilization.

24. Architect-level scenario​

A company has 500 data engineers, 100 analysts, 20 ML engineers and 50 production pipelines, but uses one large shared all-purpose cluster.

Problems: analysts compete with ETL, jobs compete with notebooks, ML needs GPUs, cost attribution is poor and security boundaries are unclear.

Better architecture:

Engineering → Serverless Jobs BI → SQL Warehouses ML → Dedicated GPU where required All → Unity Catalog → S3

The architecture separates workloads according to their execution characteristics instead of forcing everything through one cluster.

25. Architect decision framework​

Evaluate in this order:

  1. Workload type — notebook, job, SQL, pipeline, ML or streaming.
  2. Serverless support — prefer serverless when supported and suitable.
  3. Specialized requirements — GPU, RDD, R, JAR/Spark Submit, custom networking or runtime.
  4. Security/isolation — shared vs dedicated, access mode, identity and UC privileges.
  5. Performance — CPU, memory, shuffle, I/O, concurrency and data size.
  6. Cost — DBU, VM cost, idle time, scaling, storage and egress.

26. Five-layer compute architecture​

  1. WORKLOAD Notebook / Job / SQL / ML / Pipeline ↓
  2. COMPUTE MODEL Serverless / Classic / SQL Warehouse ↓
  3. EXECUTION Driver / Workers / Photon / Runtime ↓
  4. GOVERNANCE + SECURITY Identity / UC / Access Mode / Policies ↓
  5. DATA + NETWORK S3 / Delta / DB / API / VPC / Endpoints

An enterprise compute architecture is correct only when all five layers work together.

27. Architect's final mental model​

WORKLOAD │ ▼ SERVERLESS / CLASSIC / SQL WAREHOUSE │ ▼ SPARK / PHOTON / RUNTIME │ ├── Security └── Network │ ▼ Unity Catalog │ ▼ Cloud Storage

The architect's job is not to pick the biggest cluster. It is to create the right execution model for each workload, with the smallest operational and financial footprint that still satisfies performance, security, networking and reliability requirements.

What you should be able to explain​

  • Serverless vs classic compute
  • All-purpose vs jobs compute
  • SQL warehouses
  • Standard vs dedicated access mode
  • Driver vs workers
  • Photon
  • Autoscaling
  • Instance sizing
  • Compute policies
  • Networking
  • Cost architecture
  • Production compute strategy
  • Troubleshooting and workload selection

Official references​

Next​

02.5 Security Model — identity, workspace permissions, Unity Catalog privileges, access modes, cloud IAM, network controls, secrets, service principals, auditability and defense-in-depth.