07.5 Unity Catalog Volumes & File Governance

Architect-level goal: understand how Unity Catalog governs non-tabular data: files, documents, images, libraries, ML artifacts, landing data, and other objects that do not naturally belong in a table.

1. Why volumes exist

Tables govern tabular datasets. Enterprises also have CSV, JSON, PDFs, images, ML artifacts, JARs, Python wheels, configuration files, ingestion files, and other objects that are fundamentally files.

Unity Catalog volumes provide the governed abstraction for this class of data. Databricks recommends volumes for governing access to non-tabular data.

Mental model: Tables govern records. Volumes govern files.

2. Volume object model

Volumes live in the Unity Catalog three-level namespace:

catalog.schema.volume

Files are accessed through:

/Volumes/catalog/schema/volume/path/to/file

The volume is a Unity Catalog securable object. Files beneath it are governed through volume privileges.

3. Managed vs external volumes

Managed volume

Unity Catalog controls the storage location and lifecycle.

Unity Catalog ↓ Managed Volume ↓ UC-managed cloud storage

Use managed volumes when Databricks is the primary consumer, a new storage location is acceptable, and you want the simplest governance model.

External volume

The cloud path already exists and remains externally controlled.

Existing S3 path ↓ External Location ↓ External Volume ↓ Unity Catalog governance

Use external volumes when files already exist, another system produces them, the path must remain stable, or external systems also need access.

4. Create volumes

Managed:

CREATE VOLUME enterprise.procurement.landing_files;

External:

CREATE EXTERNAL VOLUME enterprise.procurement.sap_landing LOCATION 's3://enterprise-data/procurement/sap/';

Managed creation normally requires USE CATALOG, USE SCHEMA, and CREATE VOLUME.

External creation additionally requires CREATE EXTERNAL VOLUME on the governing external location.

5. Volume privileges

Important privileges include:

• CREATE VOLUME
• CREATE EXTERNAL VOLUME
• READ VOLUME
• WRITE VOLUME

To read or list files, the effective access path generally includes:

USE CATALOG + USE SCHEMA + READ VOLUME

To create, update, or delete files:

USE CATALOG + USE SCHEMA + READ VOLUME + WRITE VOLUME

Owners and principals with MANAGE can perform administrative operations such as changing permissions, dropping, and renaming volumes.

6. External location vs volume privileges

This distinction is critical.

External Location ↓ External Volume ↓ /Volumes/.../volume/...

Once an external volume exists, access to files under the volume is governed by the volume's privileges.

Do not solve every user-access requirement by granting broad READ FILES or WRITE FILES on the external location. Databricks recommends using volumes as the user-facing abstraction for path-based non-tabular access.

7. Why volumes improve security

Legacy pattern:

User → S3 URI → cloud credentials → raw files

Governed pattern:

User → Unity Catalog → Volume privileges → /Volumes/... → cloud storage

The second model centralizes authorization and avoids distributing long-lived cloud credentials to notebooks and applications.

8. File operations

Example:

files = dbutils.fs.ls('/Volumes/enterprise/procurement/landing_files/')

Read CSV:

df = ( spark.read .option('header', 'true') .csv('/Volumes/enterprise/procurement/landing_files/orders/') )

Write output:

df.write.mode('overwrite').json( '/Volumes/enterprise/procurement/landing_files/output/' )

Volumes can be used through supported Spark, Python, SQL, REST, and Databricks file-management interfaces.

9. Volumes and Auto Loader

A common ingestion architecture is:

SAP / SFTP / API / partner ↓ S3 landing ↓ External Location ↓ External Volume ↓ Auto Loader ↓ Bronze Delta ↓ Silver ↓ Gold

Here the volume is the governed file boundary and Delta tables become the governed tabular layers.

10. SAP/BW example

Imagine SAP extraction files landing in:

s3://enterprise-lake/sap/procurement/2026/11/07/

Expose the landing area through an external volume:

/Volumes/sap/procurement/raw/

Then:

External Volume ↓ Auto Loader ↓ Bronze Delta ↓ Silver ↓ Gold

This is a natural SAP/BW migration pattern because the landing zone remains file-oriented while downstream analytics become Delta-based.

11. Volumes for ML and AI

Volumes can store:

• ML training files
• model artifacts
• evaluation datasets
• prompt templates
• RAG source documents
• images
• Python wheels
• JARs
• configuration files

Example:

/Volumes/ai/rag/documents/ policies/ manuals/ product_docs/ evaluations/

A RAG pipeline can ingest these files into downstream processing while Unity Catalog governs access to the source files.

12. Path rules and collision avoidance

Avoid designs where:

• a table is defined on files inside a volume
• a volume is defined inside another volume
• an external location is nested inside another external location
• a volume is placed on a directory belonging to a table

Architectural rule:

A storage path should have one clear Unity Catalog object boundary.

For external volumes, prefer a dedicated subdirectory rather than using the root of an external location.

13. External volumes and direct cloud access

External volumes represent cloud paths that remain externally managed.

Therefore mature architectures may require both:

Unity Catalog governance + Cloud-native governance

Examples include S3 bucket policies, IAM, KMS, short-lived credential vending, and external-system identities.

Unity Catalog does not automatically govern a system that bypasses Unity Catalog and accesses S3 directly.

14. Ownership and lifecycle

Assign ownership to a group rather than an individual where practical.

Managed volume:

DROP VOLUME → Unity Catalog manages the underlying lifecycle

External volume:

DROP VOLUME → Unity Catalog metadata is removed; underlying cloud data remains

This distinction is critical during migration, cleanup, and disaster-recovery planning.

15. Governance architecture

Unity Catalog | Catalog / Schema | +--+--+ | | Tables Volumes | | Records Files | | SQL Spark / ML / Jobs

This creates a clean enterprise model:

Tables = records

Volumes = files

16. Environment design

Environment	Catalog	Schema	Volume
DEV	dev	procurement	landing
TEST	test	procurement	landing
PROD	prod	procurement	landing

Separate environments at the catalog and storage boundary where practical.

17. Troubleshooting

Cannot read files

Check:

1. Unity Catalog is enabled.
2. Compute is compatible with volumes.
3. USE CATALOG.
4. USE SCHEMA.
5. READ VOLUME.
6. The /Volumes path is correct.
7. The volume is accessible from the workspace.

Cannot write

Check WRITE VOLUME, READ VOLUME, cloud permissions for an external volume, storage availability, and path correctness.

External volume creation fails

Check CREATE VOLUME on the schema, CREATE EXTERNAL VOLUME on the external location, path overlap, workspace binding, storage credential, and cloud IAM.

External system cannot access files

Direct cloud access requires cloud-native controls or a supported short-lived credential pattern. Unity Catalog privileges alone do not govern an external system that bypasses Unity Catalog.

18. Hands-on lab

Objective

Build a governed SAP-style file landing zone.

Steps

1. Create a catalog and schema.
2. Create a managed volume.
3. Create folders under /Volumes.
4. Upload sample CSV and JSON files.
5. Grant READ VOLUME to a consumer group.
6. Grant WRITE VOLUME to an ingestion group.
7. Read the files using Spark.
8. Load them into a Bronze Delta table.
9. Create an external location over a separate S3 landing prefix.
10. Create an external volume on a subdirectory.
11. Compare managed and external lifecycle behavior.
12. Document the security boundary.

Challenge

Design access so ingestion can write, analysts can read, analysts cannot modify files, developers cannot access production, and raw cloud paths are not distributed to end users.

19. Architect decision matrix

Requirement	Recommendation
Databricks-native file storage	Managed volume
Existing S3/ADLS/GCS files	External volume
SAP landing files	External volume
Auto Loader landing	Volume
PDFs/documents	Volume
RAG source documents	Volume
SQL analytics over rows	Table
ML artifacts	Volume
Shared external-system files	External volume + cloud governance
UC should own lifecycle	Managed volume

20. Interview questions

What is a Unity Catalog volume?

A securable Unity Catalog object that provides governed access to non-tabular files.

Managed vs external volume?

Managed volume storage and lifecycle are controlled by Unity Catalog. External volume registers an existing cloud path while Unity Catalog governs access from Databricks.

Why not simply use an external location?

External locations are cloud-storage governance boundaries. Volumes are the user-facing abstraction recommended for path-based access to non-tabular data.

What privileges are required to read a volume?

Typically USE CATALOG, USE SCHEMA, and READ VOLUME.

Can files in a volume be registered as Unity Catalog tables?

No. Volumes are intended for path-based file access. Use tables for governed tabular datasets.

21. Production checklist

• Tables used for tabular data
• Volumes used for non-tabular data
• Managed vs external decision documented
• No overlapping paths
• Volume ownership assigned to a group
• READ VOLUME limited to consumers
• WRITE VOLUME limited to producers
• Production volumes isolated
• Raw cloud URIs not distributed unnecessarily
• IAM/KMS controls aligned for external volumes
• Ingestion and file lifecycle documented
• Recovery/drop behavior tested

22. Architect mental model

Table = records. Volume = files.

Managed = Unity Catalog owns storage lifecycle. External = the cloud path remains externally managed.

Volume privileges govern the file-access interface.

Official references

• What are Unity Catalog volumes?
• Create and manage volumes
• Privileges for Unity Catalog volumes
• Path rules and access
• Unity Catalog best practices
• Connect to cloud object storage

---

Next: 07.6 Unity Catalog Permissions, RBAC & Privilege Inheritance