02. Control Plane vs Compute Plane
This is one of the most important architecture topics in Databricks.
A beginner usually sees:
Workspace → Notebook → Cluster → Data
An architect needs to see something deeper:
Management plane → execution plane → storage/services → network/security boundaries
Databricks currently describes its platform using a control plane and a compute plane. The control plane contains Databricks-managed backend services and the web application. The compute plane is where workloads execute. For serverless compute, the compute plane is Databricks-managed; for classic compute, the resources run in the customer's cloud account.
Terminology note: You will often hear older architecture discussions call the execution side the data plane. In current Databricks documentation, the preferred term is compute plane.
1. The Big Picture
Think of the platform as two major responsibilities.
Control plane
The control plane answers:
- Who is the user?
- Which workspace are they using?
- What job should run?
- What permissions apply?
- What configuration should be applied?
- What metadata and platform services are involved?
- What should the UI/API expose?
Compute plane
The compute plane answers:
- Where does Spark execute?
- Where do the driver and workers run?
- Where is SQL executed?
- Where are transformations performed?
- Where is data processed?
The simplest mental model is:
CONTROL PLANE
"Manage the workload"
│
│ instructions
▼
COMPUTE PLANE
"Execute the workload"
│
│ reads / writes
▼
CLOUD DATA + SERVICES
This distinction is foundational for understanding networking, security, cost, workspace architecture and troubleshooting.
2. What Lives in the Control Plane?
The control plane is the Databricks-managed portion of the platform.
Databricks documents the control plane as including backend services managed by Databricks and the web application.
Conceptually, think about:
Web application
The browser-based Databricks interface.
You use it to:
- Open notebooks
- Create jobs
- Browse data
- Manage workspace objects
- Monitor workloads
- Configure resources
- Use SQL
- Access administration functions
Workspace services
These support workspace-level operations such as:
- Notebook management
- Job definitions
- Workspace objects
- APIs
- Workflow configuration
- Operational metadata
Identity and access
The platform needs to know:
- Which identity is calling?
- Which groups does the user belong to?
- What permissions are granted?
- What workspace can the user access?
Orchestration and management
The control plane can coordinate execution without itself being the place where your Spark transformations execute.
This distinction is critical.
3. What Does NOT Mean "Control Plane"?
Do not think:
"The control plane processes my 10 TB dataset."
That is the wrong mental model.
For example:
User clicks "Run"
│
▼
Control Plane
│
│ starts / coordinates execution
▼
Compute Plane
│
▼
Spark executors process data
The control plane is primarily concerned with management and orchestration, while the compute plane performs the actual workload execution.
4. What Is the Compute Plane?
The compute plane is the execution environment.
This is where your workload actually runs.
For Spark workloads, conceptually:
COMPUTE PLANE
Driver
│
┌────────┼────────┐
▼ ▼ ▼
Worker Worker Worker
Executor Executor Executor
│ │ │
└────────┼────────┘
│
▼
Cloud Storage
The exact implementation depends on whether you use classic or serverless compute.
5. Classic Compute Plane
With classic Databricks compute, the compute resources run in the customer's cloud account.
On AWS, Databricks documents the classic compute plane as resources deployed in the customer's AWS account and associated virtual network.
Conceptually:
DATABRICKS ACCOUNT
┌─────────────────────┐
│ CONTROL PLANE │
│ │
│ UI / APIs / Mgmt │
└──────────┬──────────┘
│
│ management connection
▼
┌─────────────────────┐
│ CUSTOMER AWS ACCOUNT│
│ │
│ VPC / Network │
│ │ │
│ ▼ │
│ Driver + Workers │
│ │
└─────────────────────┘
This gives the organization significant control over:
- Network topology
- Subnets
- Security groups
- Private connectivity
- Cloud-native controls
- Instance configuration
- Egress architecture
But it also means the organization has more infrastructure responsibility.
6. Serverless Compute Plane
Serverless changes the infrastructure responsibility.
Databricks currently documents serverless compute as running in a serverless compute plane within the Databricks account.
Conceptually:
DATABRICKS ACCOUNT
┌────────────────────────┐
│ CONTROL PLANE │
│ │
│ UI / APIs / Services │
└───────────┬────────────┘
│
▼
┌────────────────────────┐
│ SERVERLESS COMPUTE │
│ PLANE │
│ │
│ Managed execution │
│ Managed infrastructure │
└───────────┬────────────┘
│
▼
Customer data
/ services
The major difference is who manages the compute infrastructure.
| Area | Classic | Serverless |
|---|---|---|
| Compute infrastructure | Customer cloud account | Databricks-managed |
| Cluster configuration | Customer responsibility | Much less infrastructure management |
| Scaling | Customer/configuration dependent | Automatically managed |
| Networking | Customer controls more directly | Databricks-managed architecture + configured access |
| Infrastructure operations | More responsibility | Less responsibility |
| Best fit | Custom infrastructure/network needs | Most modern workloads where supported |
Databricks currently recommends serverless for many interactive and automated workloads because it reduces infrastructure management and provides automatic scaling.
7. Control Plane Does Not Mean "Public Internet"
This is an important security clarification.
A simplified diagram is often drawn as:
User
│
▼
Internet
│
▼
Control Plane
But enterprise deployments can introduce private connectivity and controlled network paths.
Databricks provides architectures for:
- Private connectivity
- Egress control
- Network isolation
- Customer VPC/VNet integration
- Private endpoints
- Firewall inspection
For example, Databricks documents AWS architectures where users can connect through VPN or inbound PrivateLink, classic compute can connect privately to the control plane, and compute can access AWS services through VPC endpoints.
So the real architecture depends on the organization's security requirements.
8. The Three Important Network Conversations
When designing Databricks networking, don't ask only:
"Can Databricks access my S3 bucket?"
Break connectivity into separate paths.
Path A — User → Databricks
Developer
│
▼
Workspace
Questions:
- Is public access allowed?
- Do we require VPN?
- Do we need PrivateLink/private connectivity?
- How is identity authenticated?
Path B — Control Plane ↔ Compute Plane
Control Plane
│
▼
Compute Plane
Questions:
- How does orchestration happen?
- Is the connection public or private?
- What firewall rules apply?
- What network endpoints are required?
Path C — Compute → Data
Compute
│
├── S3 / ADLS / GCS
├── SAP
├── Databases
├── APIs
└── Kafka
This is usually the path that directly affects data access and egress architecture.
Databricks' current network reference architecture explicitly separates inbound user connectivity, control-plane/classic-compute connectivity, and compute-to-resource connectivity.
9. Where Is the Data?
The compute plane processes the data, but the durable data does not have to live inside the compute resources.
For a Lakehouse:
COMPUTE
│
│ read/write
▼
┌───────────┐
│ DELTA │
│ TABLE │
└─────┬─────┘
│
▼
OBJECT STORAGE
S3 / ADLS / GCS
This is why terminating a cluster does not mean deleting the Lakehouse data.
Compute is temporary execution capacity.
Storage is durable data.
10. Example: Running a PySpark Notebook
Suppose you execute:
df = spark.read.table("sales.orders")
result = (
df.groupBy("customer_id")
.sum("net_amount")
)
result.write.mode("overwrite").saveAsTable("sales.customer_summary")
What happens architecturally?
Step 1 — User opens notebook
The browser communicates with the Databricks workspace/control services.
Step 2 — User runs the cell
The request is handled by the platform.
Step 3 — Workload executes
The actual Spark computation happens on the selected compute plane.
Step 4 — Spark reads data
The compute resources access the underlying table/storage.
Step 5 — Spark transforms data
Driver and executors perform the distributed computation.
Step 6 — Results are written
The workload writes the resulting data/table state to the governed storage layer.
Conceptually:
Browser
│
▼
Control Plane
│
│ execute
▼
Compute Plane
│
├── Driver
├── Executor 1
├── Executor 2
└── Executor N
│
▼
Delta / Cloud Storage
11. Why This Separation Matters for Security
Suppose your organization has a strict requirement:
Developers must not have broad access to the production S3 bucket.
The architecture should not rely only on:
"Don't give the developer the S3 credentials."
Instead, design the access path:
Developer
│
▼
Databricks identity
│
▼
Workspace / Unity Catalog
│
▼
Authorized compute
│
▼
Governed storage
The user should receive access to data assets, not necessarily raw cloud infrastructure credentials.
This is one reason Unity Catalog and compute identity/network design must be considered together.
12. Why This Separation Matters for Cost
Consider two workloads.
Workload A — BI
08:00 → 18:00
High usage
Workload B — Nightly ETL
01:00 → 03:00
High usage
With elastic compute, the execution capacity can be aligned to workload demand rather than keeping a permanently sized environment running.
The architecture becomes:
Durable Data
│
┌──────────┴──────────┐
▼ ▼
BI compute ETL compute
daytime nighttime
The important principle is:
Share data; optimize execution independently.
13. Classic vs Serverless — Architect's Decision
Do not ask:
"Which one is better?"
Ask:
"Which workload and organizational constraints make one appropriate?"
Choose serverless when you want
- Less infrastructure management
- Faster provisioning
- Automatic scaling
- Standardized platform behavior
- Modern managed execution
- Reduced cluster administration
Databricks currently recommends serverless for most jobs and many interactive workloads.
Choose classic when you need
- Custom infrastructure configuration
- Specific network topology
- Specialized runtime requirements
- Infrastructure-level control
- Capabilities not available in serverless
Databricks documents classic compute as customer-managed infrastructure in the cloud account, while serverless is Databricks-managed.
14. Control Plane vs Compute Plane vs Storage
These three should never be mentally merged.
| Layer | Think of it as | Primary responsibility |
|---|---|---|
| Control plane | Management | UI, APIs, orchestration, platform services |
| Compute plane | Execution | Spark/SQL workload processing |
| Storage | Persistence | Durable data and table files |
The architecture becomes:
MANAGEMENT
Control Plane
│
▼
EXECUTION
Compute Plane
│
▼
PERSISTENCE
Cloud Storage
This is the architecture you should keep in your head.
15. Where Does Unity Catalog Fit?
Unity Catalog cuts across the architecture as the governance layer.
Think:
UNITY CATALOG
─────────────────────
Identity / Policies
Metadata / Lineage
Data organization
Access governance
─────────────────────
│
┌────────────┼────────────┐
▼ ▼ ▼
Control Compute Storage
plane plane
It is therefore better to think of Unity Catalog as a governance/control layer, not simply another storage location.
Databricks currently describes account, workspace, data governance, and compute plane as distinct administrative layers.
16. Enterprise AWS Example
Suppose we deploy Databricks on AWS.
Control side
Databricks Account
│
├── Identity
├── Workspaces
├── APIs
└── Platform services
Compute side
For classic compute:
Customer AWS Account
│
└── VPC
│
├── Driver
├── Workers
└── Network controls
Data side
Amazon S3
│
├── Bronze
├── Silver
├── Gold
└── Delta transaction logs
Governance
Unity Catalog
│
├── Catalogs
├── Schemas
├── Tables
├── Volumes
├── External locations
└── Permissions
This is a much more useful architecture model than simply saying:
"Databricks is a cloud platform."
17. Troubleshooting Using the Architecture
This model is extremely useful when something fails.
Scenario 1 — Notebook UI does not open
Investigate:
User
↓
Control Plane
Likely areas:
- Authentication
- Workspace access
- Network access
- Identity configuration
Scenario 2 — Cluster cannot start
Investigate:
Control Plane
↓
Compute Plane
Likely areas:
- Compute configuration
- Cloud capacity
- Network configuration
- Permissions
- Policies
Scenario 3 — Job starts but cannot read S3
Investigate:
Compute
↓
Storage
Likely areas:
- Cloud IAM
- Storage credentials
- Unity Catalog permissions
- Network connectivity
- External location configuration
Scenario 4 — Spark job is slow
Investigate:
Compute
↓
Spark execution
↓
Storage access
Likely areas:
- Partitioning
- Shuffle
- File layout
- Data skipping
- Compute sizing
- Query plan
This is why architecture knowledge directly improves troubleshooting.
18. Common Misconceptions
"The control plane runs Spark."
Not as the primary execution environment.
"Serverless means there is no compute."
There is compute. You simply don't manage the underlying infrastructure in the same way.
"Classic compute is inside Databricks."
Classic compute resources are deployed in the customer's cloud account.
"The compute cluster is the database."
No. Compute is an execution resource.
"Stopping compute deletes the data."
No. Durable Lakehouse data remains in storage.
"Control plane and compute plane must always be in the same customer VPC."
Not as a universal rule. The architecture depends on classic/serverless deployment and networking configuration.
19. Hands-On Architecture Exercise
Scenario
You are designing Databricks for an enterprise with:
- AWS
- S3
- SAP S/4HANA
- PostgreSQL
- Kafka
- 500 developers
- Production and non-production workspaces
- Strict network isolation
- Private connectivity requirements
- BI and ML workloads
Design these boundaries
A. Control plane
What services should be considered Databricks-managed?
B. Compute plane
Which workloads should use:
- Serverless?
- Classic compute?
C. Storage
Where should Bronze/Silver/Gold data reside?
D. Governance
Where should access control be enforced?
E. Networking
Draw three paths:
- User → Workspace
- Control → Compute
- Compute → Data
Expected high-level answer
USERS
│
Private access
│
▼
CONTROL PLANE
│
┌─────────┴─────────┐
▼ ▼
SERVERLESS CLASSIC
COMPUTE COMPUTE
│ │
└─────────┬─────────┘
▼
S3 / SAP
PostgreSQL
Kafka
UNITY CATALOG
governs access
The exact network implementation should then be selected based on the enterprise's private connectivity, egress and compliance requirements.
20. Interview Questions
Q1. What is the Databricks control plane?
The Databricks-managed portion of the platform containing backend management services and the web application.
Q2. What is the compute plane?
The environment where Databricks workloads execute. It can be classic compute in the customer's cloud account or serverless compute managed by Databricks.
Q3. Is the control plane where Spark jobs execute?
No. Spark workloads execute on compute resources in the compute plane.
Q4. What is the difference between classic and serverless compute?
Classic compute resources are deployed in the customer's cloud environment and require more infrastructure management. Serverless compute is Databricks-managed and provides a more managed execution model.
Q5. Why does an architect care about the distinction?
Because it affects:
- Security
- Networking
- IAM
- Data access
- Cost
- Infrastructure ownership
- Troubleshooting
- Compliance
- Operational responsibility
Q6. What are the three important network paths?
- User/application → Databricks
- Control plane ↔ classic compute plane
- Compute plane → data/external resources
Databricks explicitly uses these connectivity categories in its network architecture guidance.
21. Architect-Level Mental Model
Memorize this:
USERS
│
▼
┌─────────────────┐
│ CONTROL PLANE │
│ │
│ UI / API │
│ Management │
│ Orchestration │
└────────┬────────┘
│
management path
│
┌────────────┴────────────┐
▼ ▼
┌───────────────┐ ┌────────────────┐
│ CLASSIC │ │ SERVERLESS │
│ COMPUTE PLANE │ │ COMPUTE PLANE │
│ │ │ │
│ Customer │ │ Databricks │
│ cloud account │ │ managed │
└───────┬───────┘ └───────┬────────┘
│ │
└──────────┬─────────────┘
▼
DATA / SERVICES
S3 • SAP • DB • Kafka
UNITY CATALOG
governs data + AI assets
Once this becomes second nature, topics such as Workspace Architecture, Networking, Security, Compute, Unity Catalog and Enterprise Architecture become much easier to understand.
22. What You Should Be Able to Explain After This Topic
You should now be able to answer:
- What is the control plane?
- What is the compute plane?
- Why is compute separate from storage?
- Where does classic compute run?
- Where does serverless compute run?
- What does the user actually interact with?
- Where does Spark execute?
- Where does data persist?
- What are the three major network paths?
- Why does this matter for enterprise security?
- Why does this matter for cost?
- How does Unity Catalog fit into the architecture?
- When would you choose classic vs serverless?
If you can draw the architecture from memory, you have the correct foundation for the next architecture topics.
Official References
- Databricks — High-level architecture
- Databricks — Network reference architectures
- Databricks — Compute
- Databricks — Classic compute
- Databricks — Compute selection recommendations
- Databricks — Account and identity architecture