02.3 Storage Architecture
Phase 2 — Platform Architecture
Storage is where the logical Databricks architecture meets the physical cloud.
A strong Databricks architect must be able to answer:
- Where does the data physically live?
- Who controls the storage?
- How does Unity Catalog reach it?
- Which identity reaches S3?
- How is access isolated?
- Which storage should be managed by Databricks?
- Which storage must remain customer-managed?
Current Databricks guidance recommends using Unity Catalog to govern data access and using managed tables wherever possible. External locations remain important when data must stay in customer-managed cloud storage.
1. Storage mental model
Do not think of storage as:
Workspace → S3
Think of it as:
UNITY CATALOG
│
governance + metadata
│
┌────────────┴────────────┐
▼ ▼
MANAGED STORAGE EXTERNAL STORAGE
│ │
│ External Location
│ │
▼ Storage Credential
Cloud storage │
▼
AWS IAM
│
▼
S3
Unity Catalog is the governance layer. S3 is the physical storage layer.
2. Storage object patterns
Current Databricks storage architecture distinguishes managed, external, and foreign objects. Managed and external are the key patterns for most data engineering workloads.
| Pattern | Physical data | Governance |
|---|---|---|
| Managed | Databricks-managed cloud storage | Unity Catalog |
| External | Customer-managed cloud storage | Unity Catalog + cloud IAM |
| Foreign | External system | Federated metadata/access |
3. Managed storage
Managed storage is the preferred starting point for most new Databricks-native data.
Unity Catalog
│
▼
Managed Table / Volume
│
▼
Managed Storage Location
│
▼
Cloud object storage
For managed tables and volumes, Databricks manages the storage layout and lifecycle through Unity Catalog.
Why it is attractive
You avoid making every data engineer manage:
- S3 prefixes
- IAM role details
- storage credentials
- external-location registration
- physical layout conventions
If there is no concrete requirement to control the physical storage path, prefer managed storage.
4. Managed storage hierarchy
Managed storage can be associated at:
- metastore
- catalog
- schema
Lower-level locations override higher-level locations when managed objects are created.
Metastore managed location
│
├── Catalog A managed location
│ └── Schema X managed location
│
└── Catalog B managed location
Current Databricks guidance recommends catalog-level managed storage for logical data isolation and warns against relying blindly on default locations when explicit isolation matters.
5. Catalog-level storage
Suppose the enterprise has:
finance_prod
sales_prod
procurement_prod
A useful physical design could be:
S3
├── finance-prod
├── sales-prod
└── procurement-prod
Then:
finance_prod catalog → finance managed location
sales_prod catalog → sales managed location
This creates a meaningful physical boundary without requiring a separate workspace for every domain.
6. Storage root
When a managed location is specified for a catalog or schema, Unity Catalog tracks the supplied path as the storage root and creates generated subdirectories underneath it.
Example:
s3://enterprise-finance-prod/
│
└── __unitystorage/
└── generated managed-object paths
Do not build application logic around Unity Catalog's internal managed-storage subdirectories. Treat the managed location as an architectural boundary.
7. External storage
External storage is appropriate when data must remain in customer-controlled cloud storage.
Typical cases:
- existing enterprise S3 lake
- data produced by another platform
- shared files consumed by multiple systems
- regulatory storage requirements
- migration estates
- large unstructured datasets
Architecture:
Unity Catalog
│
▼
External Table / Volume
│
▼
External Location
│
▼
Storage Credential
│
▼
AWS IAM Role
│
▼
S3
Databricks documents the storage-credential → external-location → cloud-storage pattern as the standard mechanism for governing external cloud storage.
8. Storage credential
A storage credential represents the authentication mechanism used by Unity Catalog to reach cloud storage.
For AWS:
Storage Credential
│
▼
AWS IAM Role
│
▼
S3 permissions
The role should have only the permissions required for the target buckets or prefixes. Databricks recommends separate roles by environment or data domain when segregation is required.
9. External location
An external location combines:
Cloud storage path
+
Storage credential
=
External location
Example:
External Location: finance_prod_data
URL: s3://enterprise-data-prod/finance/
Credential: finance-prod-role
Unity Catalog permissions determine which users or groups can use the external location.
10. External location vs storage credential
| Storage Credential | External Location | |
|---|---|---|
| Represents | Authentication | Path + authentication |
| AWS concept | IAM role | S3 path governed by UC |
| Purpose | Reach cloud storage | Govern a specific storage path |
| Main privilege | CREATE EXTERNAL LOCATION | Access/use privileges |
Mental model:
Credential = HOW do I authenticate?
External Location = WHERE can that credential be used?
11. AWS IAM architecture
For external S3 access:
User / Workload
│
▼
Unity Catalog
│
│ UC privileges
▼
External Location
│
▼
Storage Credential
│
▼
AWS IAM Role
│
│ IAM policies
▼
S3 Bucket / Prefix
Current Databricks AWS guidance uses cross-account IAM roles and trust policies for Unity Catalog storage access.
Critical distinction
Unity Catalog authorization does not eliminate AWS IAM.
You need both layers designed correctly.
12. AWS request flow
For:
SELECT * FROM finance_prod.sales.orders;
a simplified architecture is:
1. User authenticates
│
2. Databricks evaluates UC permissions
│
3. UC resolves table/storage metadata
│
4. Storage credential identifies IAM role
│
5. AWS trust and IAM authorization are evaluated
│
6. S3 authorizes the object operation
│
7. Data is returned
The exact execution path varies by workload and compute mode; the layered authorization model is the important architecture.
13. Bucket policy vs IAM role
IAM role
Defines what the delegated identity can do.
Bucket policy
Adds a resource-side security boundary.
IAM Role
│
├── allowed actions
└── allowed resources
S3 Bucket Policy
│
├── allowed principals
├── allowed conditions
└── allowed actions
Databricks recommends bucket policies as an additional security layer and CloudTrail for auditing.
14. Storage isolation strategy
Avoid designing:
One giant S3 bucket
├── everything
├── everyone
└── every environment
Instead, align physical boundaries with actual requirements:
S3
├── finance-prod
├── sales-prod
├── procurement-prod
├── finance-dev
└── sales-dev
Do not automatically create a bucket for every catalog. Ask:
What physical boundary is required by security, residency, lifecycle, ownership or platform integration?
15. Bucket vs prefix isolation
Bucket-level
s3://finance-prod/
s3://sales-prod/
Prefix-level
s3://enterprise-data/finance/
s3://enterprise-data/sales/
Choose based on:
- IAM boundaries
- bucket policies
- encryption requirements
- ownership
- lifecycle rules
- cross-account requirements
- residency
- operational scale
Do not turn every logical catalog into a separate bucket without a real architectural reason.
16. Managed vs external decision framework
| Requirement | Preferred pattern |
|---|---|
| New Databricks-owned data | Managed |
| Existing S3 data lake | External |
| Data shared directly with other systems | Often external |
| Databricks-managed lifecycle | Managed |
| Fixed physical path requirement | External |
| Direct file interoperability | External |
| Simplest governance | Managed |
| Legacy lake migration | External initially |
Databricks recommends managed tables wherever possible because they simplify governance and storage management.
17. External tables
An external table has metadata governed by Unity Catalog while its underlying data remains at an external storage location.
UC metadata
│
▼
External Table
│
▼
s3://enterprise-existing-data/orders/
This is particularly useful during migrations:
Legacy S3 Data Lake
│
▼
Unity Catalog external table
│
▼
Databricks workloads
Governance can therefore be introduced without immediately rewriting every physical file.
18. External volumes
Volumes provide governed access to file-oriented data such as:
- PDFs
- images
- documents
- CSV files
- configuration files
- ML artifacts
Unity Catalog Volume
│
▼
Managed or External Storage
│
▼
S3
This becomes especially important for AI/ML platforms where not every asset is naturally represented as a table.
19. Storage + medallion architecture
Storage should support—not dictate—the medallion architecture.
Cloud Storage
│
┌────────┼────────┐
▼ ▼ ▼
Bronze Silver Gold
│ │ │
▼ ▼ ▼
Raw Curated Analytics
Logical Bronze/Silver/Gold boundaries do not necessarily require three separate S3 buckets.
20. Environment isolation
A production design often separates environments:
DEV
└── finance_dev
└── S3 dev path
TEST
└── finance_test
└── S3 test path
PROD
└── finance_prod
└── S3 prod path
Combine physical isolation with:
- separate workspaces
- catalog permissions
- workspace bindings
- separate storage credentials
- separate IAM roles
21. Storage credential strategy
Avoid one enterprise-wide credential:
one-role-for-everything
Prefer meaningful trust boundaries:
uc-finance-dev-role
uc-finance-prod-role
uc-sales-prod-role
uc-procurement-prod-role
Databricks explicitly recommends separate IAM roles by environment or business unit when segregation is required and least-privilege permissions.
A storage credential should represent a meaningful trust boundary.
22. Least privilege
If Finance needs:
s3://enterprise-finance-prod/
do not grant:
s3:*
on *
Instead, restrict actions and resources to the required bucket/prefix.
Databricks recommends limiting S3 permissions to the required buckets or prefixes.
23. Read-only external locations
Some data should never be modified by Databricks workloads.
Corporate reference data
│
▼
S3
│
▼
External Location
│
READ ONLY
│
▼
Databricks
Databricks supports read-only restrictions on external locations and storage credentials.
Useful for:
- source-system extracts
- regulatory datasets
- immutable landing zones
- shared enterprise reference data
24. Storage governance and direct S3 access
One of the most important enterprise rules:
Do not let users bypass Unity Catalog for storage that is supposed to be governed by Unity Catalog.
Databricks specifically recommends limiting direct user access to buckets registered as external locations.
GOOD
User → Unity Catalog → External Location → S3
RISKY
User ───────────────────────────────→ S3
bypass
25. DBFS mounts vs Unity Catalog
Legacy environments may contain DBFS mounts.
Current Databricks guidance recommends migrating cloud-storage mounts to Unity Catalog external locations and avoiding DBFS root for new storage designs.
Legacy:
Workspace → DBFS Mount → S3
Modern:
Workspace → Unity Catalog → External Location → S3
This moves storage governance from workspace-specific configuration into the account-level governance model.
26. Cross-account AWS architecture
Large enterprises may keep data in a dedicated AWS data account:
AWS Databricks Account
│
│ assume role
▼
AWS Data Account
│
▼
S3 Data Lake
The IAM role lives in the data account and trusts the appropriate Databricks identity.
Benefits:
- centralized storage ownership
- separation of compute and data accounts
- centralized bucket security
- additional bucket-policy controls
- cleaner enterprise operating model
27. Encryption
A typical enterprise chain is:
Databricks
│
Unity Catalog
│
IAM
│
S3
│
SSE-S3 / SSE-KMS
If an S3 bucket requires encryption headers, Unity Catalog external-location configuration can support the required encryption algorithm.
Architects should additionally define:
- KMS key ownership
- key policies
- rotation
- audit requirements
- cross-account key access
28. Storage lifecycle
Storage architecture must consider lifecycle.
Bronze
│
├── hot
└── older → cheaper storage class
Silver
└── operational retention
Gold
└── business retention
Be careful with lifecycle policies on active Delta data. Retention must consider Delta files, transaction logs, VACUUM, downstream readers and restore requirements.
Delta storage internals will be covered in Phase 4 — Delta Lake.
29. Disaster recovery
A DR design is not simply:
"Copy the S3 bucket."
It includes:
Data
+
Metadata
+
Governance
+
Identity
+
Networking
+
Compute
+
Deployment automation
Questions include:
- Is S3 replicated?
- Is cross-region storage required?
- How is Unity Catalog configuration redeployed?
- Are IAM roles recreated?
- Are KMS keys available?
- Are workspace bindings restored?
- Are external locations deployed from code?
30. Enterprise reference architecture
A mature AWS design could look like:
DATABRICKS ACCOUNT
│
┌────────┴────────┐
▼ ▼
DEV WORKSPACE PROD WORKSPACE
│ │
└────────┬────────┘
▼
UNITY CATALOG
│
┌────────┴────────┐
▼ ▼
finance_prod sales_prod
│ │
managed / external managed / external
│ │
▼ ▼
S3 finance S3 sales
│ │
└────────┬────────┘
▼
AWS IAM / KMS
│
Data Account
This separates:
- Operational isolation
- Logical governance
- Physical storage
- Cloud authorization
- Encryption
31. Hands-on — external storage pattern
Use a development AWS account. Do not execute these commands against production without validating IAM, bucket policy and data ownership.
Step 1 — Create an S3 path
Example:
s3://my-company-databricks-dev/orders/
Step 2 — Create an IAM role
The role should:
- trust the required Databricks identity
- have only required S3 permissions
- target only the required bucket/prefix
Step 3 — Register the role as a storage credential
Conceptually:
CREATE STORAGE CREDENTIAL orders_dev_credential
WITH (AWS_IAM_ROLE = 'arn:aws:iam::<account-id>:role/<role-name>');
Follow the current Databricks AWS setup flow; automated setup is currently recommended for many deployments.
Step 4 — Create an external location
Conceptually:
CREATE EXTERNAL LOCATION orders_dev_location
URL 's3://my-company-databricks-dev/orders/'
WITH (STORAGE CREDENTIAL orders_dev_credential);
Step 5 — Test
Validate:
- IAM trust
- S3 permissions
- UC privileges
- workspace binding
- path correctness
Step 6 — Create an external table
Conceptually:
CREATE TABLE training.orders_external
USING DELTA
LOCATION 's3://my-company-databricks-dev/orders/';
The actual format must match the files at the location.
32. Hands-on — managed storage
Compare the external pattern with managed storage:
CREATE CATALOG finance_dev
MANAGED LOCATION 's3://my-company-databricks-dev/finance/';
Then:
CREATE SCHEMA finance_dev.gold;
Then:
CREATE TABLE finance_dev.gold.revenue (
fiscal_year INT,
customer_id BIGINT,
revenue DECIMAL(18,2)
);
The physical managed-storage layout is handled by Databricks. Managed storage can be defined at catalog or schema level, with lower-level definitions taking precedence.
33. Managed vs external — architect comparison
| Dimension | Managed | External |
|---|---|---|
| Physical path | UC-managed | Customer-controlled |
| Governance | UC | UC + cloud IAM |
| Complexity | Lower | Higher |
| Existing lake integration | Medium | Excellent |
| Direct file interoperability | Limited by design | Strong |
| Storage lifecycle | Databricks-managed | Customer-managed |
| Recommended default | Yes | When required |
| IAM complexity | Lower | Higher |
| Migration usefulness | Medium | Very high |
34. Common storage architecture mistakes
Mistake 1 — One IAM role for the entire enterprise
Creates an unnecessarily large blast radius.
Mistake 2 — One external location for everything
Weakens isolation and makes governance harder.
Mistake 3 — Direct S3 access for developers
Allows bypass of Unity Catalog controls.
Mistake 4 — Treating S3 folders as the governance model
S3 is the physical layer; Unity Catalog is the logical governance layer.
Mistake 5 — Over-fragmenting buckets
Hundreds of buckets without a real security or operational reason increase complexity.
Mistake 6 — Using default storage blindly
Use explicit catalog/schema managed locations when physical isolation matters.
Mistake 7 — Treating DBFS mounts as the modern architecture
Use Unity Catalog external locations for governed cloud storage.
Mistake 8 — Ignoring KMS and bucket policies
IAM role permissions are not the entire AWS security architecture.
35. Troubleshooting decision tree
External table query fails
│
▼
Does UC permission exist?
│
NO ─┴─ YES
│ │
Grant ▼
Workspace binding valid?
│
NO ─┴─ YES
│ │
Fix binding ▼
External location valid?
│
NO ─┴─ YES
│ │
Fix path ▼
Storage credential valid?
│
NO ─┴─ YES
│ │
Fix IAM ▼
S3 policy valid?
│
NO ─┴─ YES
│ │
Fix IAM ▼
Check
encryption,
network and path
This is the troubleshooting sequence to remember.
36. Architect interview scenarios
Scenario 1 — Cross-account finance lake
Requirement: Finance data stays in a dedicated AWS data account.
Design:
Databricks Account
│
▼
UC Storage Credential
│
▼
Cross-account IAM Role
│
▼
Finance Data Account
│
▼
S3
Add bucket policy, KMS and least privilege.
Scenario 2 — Developers must never access production S3
Use multiple layers:
- separate dev/prod workspaces
- separate catalogs
- workspace-catalog bindings
- separate storage credentials
- separate IAM roles
- separate S3 paths
- least-privilege bucket policies
Scenario 3 — Existing enterprise S3 lake
Use:
Existing S3
│
▼
Storage Credential
│
▼
External Location
│
▼
External Tables / Volumes
│
▼
Unity Catalog governance
Scenario 4 — New Databricks-native domain
Prefer:
Catalog
│
Managed Location
│
Managed Tables
│
S3
This minimizes operational complexity.
37. Architect storage decision tree
Is data already in customer-managed storage?
│
┌─────┴─────┐
YES NO
│ │
External Managed
│
▼
Must physical path remain fixed?
│
┌───┴───┐
YES NO
│ │
External Consider
location migration
Then ask:
Does the data require domain isolation?
│
YES
│
Catalog-level managed/external storage
Then:
Does it require environment isolation?
│
YES
│
Separate storage paths + credentials + IAM
38. Architect mental model
Memorize:
LOGICAL PLANE
│
UNITY CATALOG
│
catalog / schema / object
│
permissions / lineage
│
─────────────────
│
STORAGE PLANE
│
┌────────┴────────┐
▼ ▼
MANAGED EXTERNAL
│ │
│ external location
│ │
│ storage credential
│ │
│ AWS IAM
│ │
└────────┬────────┘
▼
S3
Managed storage hides physical complexity. External storage exposes architectural responsibility.
39. What you should be able to explain now
- Managed vs external storage
- Metastore/catalog/schema managed locations
- Storage roots
- S3 architecture
- Storage credentials
- External locations
- IAM roles
- Cross-account access
- Bucket policies
- KMS/encryption
- Read-only locations
- Storage isolation
- Environment isolation
- Domain isolation
- External tables
- External volumes
- DBFS migration
- Lifecycle considerations
- DR considerations
- Storage troubleshooting
- Enterprise storage architecture
Most importantly:
You should be able to draw the complete path from a Databricks table to the physical S3 object and explain every security boundary along the way.
Official references
- Databricks — Design storage architecture
- Databricks — Design Unity Catalog architecture
- Databricks — Connect to AWS S3
- Databricks — Managed storage locations
- Databricks — External locations
- Databricks — Storage credentials
- Databricks — Unity Catalog best practices
Next
02.4 Compute Architecture — Classic compute, serverless, job compute, all-purpose compute, SQL warehouses, Photon, autoscaling, cluster policies, driver/worker architecture, network placement, cost architecture and enterprise compute strategy.