01. What is Databricks?
Databricks is a cloud-based Data + AI Platform built around the Lakehouse architecture. It brings data engineering, SQL analytics, streaming, machine learning, AI applications, and governance onto a common data foundation.
A beginner mistake is to define Databricks as “a place where Spark jobs run.” Spark is important, but that definition is far too narrow.
Architect's mental model: Databricks is the platform; Apache Spark is a major processing engine; Delta Lake provides reliable lakehouse tables; Unity Catalog provides governance; and cloud object storage remains the durable storage foundation.
1. Why Databricks Exists
Traditional enterprise data landscapes often grow as a collection of specialized platforms:
Operational Systems
↓
ETL Tools
↓
Data Lake ──────→ Data Warehouse
↓ ↓
Data Science BI / Reporting
↓
ML Platform
Each additional platform can introduce data duplication, separate security models, multiple pipelines, additional data movement, operational complexity, and difficult lineage.
The Lakehouse approach tries to reduce this fragmentation by keeping data on a common storage foundation while allowing different engines and workloads to operate on it.
Databricks describes the Lakehouse as a system combining benefits of data lakes and data warehouses. Its current architecture uses Apache Spark for scalable processing, Delta Lake for reliable tables, and Unity Catalog for unified governance. citeturn353639search9turn353639search2
2. Databricks Is a Platform, Not Just Spark
Think about Databricks as a stack:
| Layer | Technology / Capability | Purpose |
|---|---|---|
| Storage | Cloud object storage | Durable storage |
| Table layer | Delta Lake | Reliable tables and transactions |
| Compute | Apache Spark / Photon / SQL engines | Process and query data |
| Analytics | Databricks SQL / AI-BI capabilities | BI and analytical workloads |
| Engineering | Auto Loader / Lakeflow / Jobs | Ingestion and orchestration |
| Governance | Unity Catalog | Security, lineage, discovery and audit |
| ML / AI | MLflow / Mosaic AI capabilities | ML and AI lifecycle |
| Development | Notebooks / Git / APIs / Bundles | Build and deploy solutions |
The exact services available depend on cloud, workspace configuration, and enabled capabilities. Databricks' current platform documentation positions the product as a unified foundation for ETL, data warehousing/BI, ML/AI and governed data workloads. citeturn353639search2turn353639search1
3. The Lakehouse Idea
A data lake is generally optimized for inexpensive, flexible storage of large volumes of raw and varied data.
A data warehouse is optimized for governed, structured analytical workloads.
A lakehouse combines important characteristics of both:
| Capability | Data Lake | Data Warehouse | Lakehouse |
|---|---|---|---|
| Low-cost object storage | Strong | Usually indirect | Strong |
| Raw / semi-structured data | Strong | Limited | Strong |
| ACID table transactions | Limited in traditional file layouts | Strong | Strong with Delta |
| SQL analytics | Possible | Strong | Strong |
| Data science / ML | Strong | Usually separate | Integrated |
| Streaming | Strong | Varies | Strong |
| Governance | Often surrounding tools | Strong | Unity Catalog |
| Open storage foundation | Strong | Often proprietary | Open cloud storage |
The key idea is not that a Lakehouse magically replaces every system. Its value is reducing unnecessary movement and duplication while providing reliable tables, scalable compute, and governance.
4. The Four Technologies You Must Understand First
4.1 Apache Spark — Processing
Spark provides distributed processing.
A large workload can be divided into partitions and processed in parallel:
Large Dataset
↓
Partitions
┌───┼───┬───┐
↓ ↓ ↓ ↓
E1 E2 E3 E4
└───┼───┴───┘
↓
Result
You will study the Spark execution model later: driver, executors, jobs, stages, tasks, partitions, shuffle, DAGs and execution plans.
4.2 Delta Lake — Reliable Tables
Delta Lake adds a transaction and metadata layer to data stored in cloud object storage.
It provides capabilities such as:
- ACID transactions
- Schema enforcement
- Schema evolution
- UPDATE and DELETE
- MERGE / upsert patterns
- Time travel
- Reliable concurrent writes
Databricks currently documents Delta Lake as the recommended table/data format for the Lakehouse, and tables created through normal Databricks table operations use Delta by default unless another format is selected. citeturn353639search2turn353639search8
4.3 Unity Catalog — Governance
Unity Catalog is the governance layer for data and AI assets.
It provides capabilities including:
- Access control
- Data discovery
- Lineage
- Auditing
- Central metadata management
The common namespace is:
catalog.schema.object
For example:
prod_finance.sales.orders
Unity Catalog is documented as the unified governance layer for data and AI in Databricks. citeturn353639search4turn353639search6
4.4 Cloud Object Storage — Durable Storage
Databricks does not mean that all data lives inside a compute cluster.
The Lakehouse uses cloud storage such as:
- Amazon S3
- Azure Data Lake Storage
- Google Cloud Storage
Compute is separated from durable data storage. This separation is one of the architectural foundations of the platform. citeturn353639search2
5. End-to-End Enterprise Flow
Consider an enterprise receiving sales data from SAP, CRM systems, APIs and streaming platforms.
SAP / S4HANA CRM APIs Kafka
│ │ │ │
└───────────────┴───────────┴───────────┘
│
▼
Ingestion / Landing
│
▼
┌─────────────────┐
│ BRONZE │
│ Raw / Auditable │
└────────┬────────┘
│
▼
┌─────────────────┐
│ SILVER │
│ Cleansed / │
│ Conformed │
└────────┬────────┘
│
▼
┌─────────────────┐
│ GOLD │
│ Business-ready │
└────────┬────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
BI / SQL ML GenAI
The Bronze → Silver → Gold pattern is commonly called the medallion architecture. Databricks' current architecture guidance describes Bronze as raw data, Silver as refined data, and Gold as business-ready data products. citeturn282533search2
Important: Medallion architecture is a logical design pattern, not a requirement that every organization must create exactly three physical databases.
6. What Happens When You Run a Databricks Workload?
At a simplified level:
You
│
│ Notebook / SQL / Job / API
▼
Databricks Workspace
│
▼
Compute
│
├── Spark / SQL execution
├── Reads Delta tables
└── Writes results
│
▼
Cloud Object Storage
Unity Catalog
│
└── Governs access and metadata
A workspace is the environment where users work with notebooks, queries, jobs and compute. At enterprise level, the Databricks account manages identities, workspaces, Unity Catalog metastores and usage. citeturn340752view0
Do not confuse:
| Term | Think of it as |
|---|---|
| Account | Enterprise-level administration |
| Workspace | Working environment |
| Compute | Resources executing workloads |
| Storage | Durable data |
| Unity Catalog | Governance and metadata |
These concepts will be expanded in the next architecture modules.
7. Typical Databricks Personas
Data Engineer
Builds batch pipelines, streaming pipelines, CDC processing, data quality checks and incremental transformations.
Analytics Engineer / SQL Developer
Builds SQL transformations, views, analytical datasets and BI-ready models.
Data Scientist / ML Engineer
Works with feature datasets, experiments, model training and ML lifecycle.
AI Engineer
Builds RAG applications, vector-search solutions, agentic applications and model-powered applications.
Data Architect
Designs lakehouse architecture, data domains, governance, security, storage, compute, DR/HA, cost and performance patterns.
This is why Databricks should be learned as a platform architecture, not simply as a PySpark library.
8. Where Databricks Fits in Enterprise Architecture
A useful enterprise view is:
Business Consumers
BI • Apps • AI • Analysts
▲
│
┌──────────┴──────────┐
│ Databricks Platform│
│ SQL • Spark • ML/AI │
│ Jobs • Streaming │
└──────────▲──────────┘
│
┌──────────┴──────────┐
│ Delta Lake │
│ Unity Catalog │
└──────────▲──────────┘
│
┌──────────┴──────────┐
│ Cloud Object Storage│
│ S3 / ADLS / GCS │
└─────────────────────┘
The key architectural separation is:
Storage is durable. Compute is elastic. Governance is centralized. Workloads are decoupled from the physical storage layer.
9. Databricks vs “A Spark Cluster”
This distinction matters in interviews and architecture discussions.
| Spark cluster | Databricks platform |
|---|---|
| Processing engine | Complete data + AI platform |
| Focused on execution | Execution + storage + governance + orchestration + analytics + AI |
| You manage more surrounding components | Many capabilities are integrated |
| Spark is the center | Spark is one major platform component |
| Limited platform-level governance | Unity Catalog provides centralized governance |
A Databricks solution can therefore include Spark without every user directly writing PySpark.
For example, a business analyst may work primarily with SQL, while a data engineer builds Spark pipelines and an ML engineer works with MLflow and model-serving capabilities.
10. Common Misconceptions
Reality: Spark is a core processing engine, but Databricks is a broader Data + AI Platform.
Reality: Lakehouse data is stored on cloud object storage; compute processes that data.
Reality: Delta Lake is an open-source storage/table layer. Databricks is the broader platform around it.
Reality: Unity Catalog provides governance capabilities across data and AI assets, including permissions, lineage, discovery and auditing.
Reality: Medallion architecture is a logical pattern. Physical implementation depends on the organization's architecture.
11. Why This Matters for an SAP / Enterprise Data Engineer
If your background is in enterprise data warehousing, the easiest way to understand Databricks is to map concepts rather than memorize product names.
| Traditional enterprise concept | Databricks mental model |
|---|---|
| Data warehouse | Lakehouse |
| ETL platform | Spark / Lakeflow / orchestration |
| Persistent warehouse tables | Delta tables |
| Governed data layer | Curated Delta + SQL models |
| Central catalog | Unity Catalog |
| Batch processing | Spark / Jobs / pipelines |
| Streaming | Structured Streaming / Lakeflow |
| BI consumption | Databricks SQL / BI tools |
| ML platform | MLflow + Databricks ML/AI capabilities |
The mapping is conceptual, not one-to-one. A mature Databricks architecture should not simply reproduce an old warehouse architecture with different product names.
12. Production Architecture Principles Introduced in This Course
Principle 1 — Separate storage and compute
Do not design the platform around a single permanently running processing cluster.
Principle 2 — Govern the data foundation
Security should not be added after pipelines are built. Governance belongs in the architecture from the beginning.
Principle 3 — Process incrementally
For large enterprise datasets, repeatedly processing the entire history is usually inefficient.
Principle 4 — Make pipelines observable
A production pipeline needs metrics, logs, data-quality checks, failure handling, alerting and lineage.
Principle 5 — Optimize based on evidence
Do not tune Spark blindly. Use query plans, Spark UI, workload metrics and data-layout information.
13. Hands-On Exercise
Design a simple architecture for this scenario:
A company receives daily sales data from SAP, customer data from Salesforce, and clickstream events from Kafka. The business wants dashboards, machine-learning features, and a GenAI assistant.
Design:
- Source systems
- Ingestion mechanism
- Bronze layer
- Silver layer
- Gold layer
- Governance boundary
- BI consumption
- ML consumption
- GenAI consumption
Do not worry about implementation yet. The objective is to learn to think in terms of platform architecture and data flow.
14. Interview-Grade Questions
Q1. What is Databricks?
Answer: Databricks is a cloud-based Data + AI Platform built around the Lakehouse architecture. It provides capabilities for data engineering, SQL analytics, streaming, machine learning, AI, orchestration and governance on a common data foundation.
Q2. Is Databricks the same as Apache Spark?
Answer: No. Apache Spark is a distributed processing engine used extensively by Databricks. Databricks provides the broader platform around processing, storage, governance, orchestration, analytics and AI.
Q3. What should you understand first?
Answer: Apache Spark for distributed processing, Delta Lake for reliable lakehouse tables, Unity Catalog for governance, and cloud object storage as the durable storage foundation.
Q4. Why is Lakehouse architecture important?
Answer: It combines scalable cloud storage with reliable table semantics, flexible processing and governed analytical workloads, reducing unnecessary data movement and duplication between separate lake, warehouse and ML platforms.
Q5. Where does the data actually live?
Answer: Lakehouse data is primarily stored in cloud object storage such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage. Databricks compute processes that data.
15. Summary
You should now be able to explain Databricks without relying on product buzzwords.
Remember this mental model:
DATABRICKS
│
┌─────────────┼─────────────┐
▼ ▼ ▼
ENGINEERING ANALYTICS AI / ML
│ │ │
└─────────────┼─────────────┘
▼
LAKEHOUSE
│
┌──────────┴──────────┐
▼ ▼
Delta Lake Cloud Storage
│
▼
Unity Catalog
Governance + Lineage
The next module will explain Lakehouse architecture in depth, including storage/compute separation, Bronze/Silver/Gold, Delta Lake, Unity Catalog, ingestion, processing and consumption patterns.
References
- Databricks — What is a data lakehouse?
- Databricks — The scope of the Data + AI Platform
- Databricks — High-level architecture
- Databricks — Databricks components