07.4 External Locations & Storage Credentials

Architect-level goal: understand how Unity Catalog turns customer-owned cloud storage into a governed access boundary without distributing long-lived cloud credentials to data users.

1. Why these two objects matter

When data lives in customer-managed storage such as Amazon S3, Unity Catalog separates two concerns:

• Storage credential — represents the cloud authentication mechanism, such as an AWS IAM role.
• External location — combines a cloud storage URI with a storage credential and becomes a governed path boundary.

A credential answers how can Databricks authenticate? An external location answers which path is governed, and who can use it?

2. The security chain

Think about the architecture as:

User / Group
    ↓
Unity Catalog privilege
    ↓
External Location
    ↓
Storage Credential
    ↓
Cloud IAM
    ↓
S3

The user does not receive an AWS access key. Unity Catalog governs the cloud path and the principals allowed to use it.

3. Storage credential vs external location

Object	Represents	Primary responsibility
Storage credential	Cloud authentication	IAM role / cloud identity
External location	Cloud path + credential	Path-level governance
External table	Table metadata over files	Table-level governance
External volume	Files/non-tabular data	Volume-level governance

A storage credential can be reused by multiple external locations. Databricks recommends granting only the privileges required to create external locations.

4. AWS architecture

Databricks Account
      ↓
Unity Catalog Metastore
      ├── Storage Credential → AWS IAM Role
      └── External Location → s3://company-prod-data/finance/

The IAM role should be scoped to the smallest useful bucket/prefix set.

Cross-account AWS

A common enterprise pattern keeps Databricks and the data lake in separate AWS accounts. The IAM role in the data account trusts the required Databricks identity; its permissions policy controls the exact S3 resources.

Databricks identity is not a human AWS identity.

5. IAM policy design

A conceptual least-privilege policy might allow GetObject, PutObject, DeleteObject and ListBucket only for the required bucket/prefix. Treat examples as teaching material; production policies must match the exact Databricks setup and cloud-security requirements.

Read-only domains should not receive write/delete permissions.

6. Create a storage credential

Conceptually:

CREATE STORAGE CREDENTIAL finance_s3_cred
WITH (AWS_IAM_ROLE = 'arn:aws:iam::<account-id>:role/<role-name>');

Inspect it with:

SHOW STORAGE CREDENTIALS;
DESCRIBE STORAGE CREDENTIAL finance_s3_cred;

The creator needs the appropriate metastore privilege, such as CREATE STORAGE CREDENTIAL. Validate the current SQL reference before production automation.

7. Credential privileges

A credential is not the same thing as data access. A strong enterprise model is:

Cloud Platform Team
      ↓
Storage Credential
      ↓  CREATE EXTERNAL LOCATION
Data Platform Group
      ↓
External Location
      ├── READ FILES → Consumers
      └── WRITE FILES → Producers

Grant only CREATE EXTERNAL LOCATION on a storage credential when that is all the principal needs.

8. Create an external location

An external location combines a storage URI, a storage credential and Unity Catalog governance.

CREATE EXTERNAL LOCATION finance_prod
URL 's3://company-prod-data/finance/'
WITH (CREDENTIAL finance_s3_cred);

Inspect it with:

DESCRIBE EXTERNAL LOCATION finance_prod;
SHOW GRANTS ON EXTERNAL LOCATION finance_prod;

9. External-location privileges

Examples include READ FILES, WRITE FILES and CREATE EXTERNAL TABLE.

GRANT READ FILES
ON EXTERNAL LOCATION finance_prod
TO `finance-readers`;

GRANT READ FILES, WRITE FILES
ON EXTERNAL LOCATION finance_prod
TO `finance-engineering`;

GRANT CREATE EXTERNAL TABLE
ON EXTERNAL LOCATION finance_prod
TO `finance-engineering`;

Important: current Databricks documentation states that WRITE FILES requires READ FILES on the same external location.

10. Path boundaries

Suppose an S3 hierarchy contains finance, procurement, sales and HR prefixes. A broad location over the entire production prefix can become a very large governance boundary.

Create separate locations when there is a real ownership, security, lifecycle or access boundary. Do not create hundreds of locations merely because directories exist.

Before creating one, ask:

1. Does a parent location already cover this path?
2. Is this a real security boundary?
3. Who owns the path?
4. Which workspaces should see it?
5. Is the location required for lifecycle or interoperability?

11. External tables and volumes

External tables reference data stored at governed external locations. External volumes are useful for governed non-tabular files such as PDFs, images and documents.

Architecturally:

External Location
   ├── External Table A
   ├── External Table B
   └── External Volume C

Use volumes for governed file access instead of exposing raw cloud paths to users.

12. Workspace bindings

Storage credentials and external locations can be bound to specific workspaces. This is valuable for dev/test/prod isolation.

Unity Catalog Metastore
       ├── DEV credential → DEV workspace → dev S3
       └── PROD credential → PROD workspace → prod S3

For storage credentials, the workspace binding is checked when the credential is exercised to create an external location. The external location then operates independently of that credential binding. Do not assume credential binding alone governs all later data access.

13. Environment isolation

Environment	Storage	Credential	Location pattern
DEV	dev prefixes	dev credential	dev locations
TEST	test prefixes	test credential	test locations
PROD	prod prefixes	prod credential	prod locations

Avoid allowing developers to create production external locations from development workspaces.

14. Encryption and KMS

For SSE-KMS designs, consider the full chain:

Unity Catalog → IAM role → S3 bucket policy → KMS key policy → Object

A successful Unity Catalog configuration does not automatically mean the KMS policy permits the operation.

15. File events and Auto Loader

External locations can participate in file-event based ingestion. This can reduce repeated cloud listing and improve ingestion efficiency.

File-event configurations can require additional SQS/SNS permissions. Do not add event-management permissions blindly to a broad data-access role; decide whether Databricks should manage events and whether security teams require separation.

16. Managed storage vs external locations

Requirement	Preferred direction
New Delta data and UC should own lifecycle	Managed table
Existing S3 data	External location + external table
Shared non-tabular files	External location + volume
Exact cloud path must remain stable	External location
Curated lakehouse layers	Prefer managed tables

You do not have to make every downstream Delta layer external.

17. SAP/BW example

Consider SAP procurement data landing in S3:

SAP / CDC / ingestion
        ↓
S3 procurement landing
        ↓
External Location
        ↓
Bronze
        ↓
Silver managed Delta
        ↓
Gold managed Delta
        ↓
BI / SQL / SAC / downstream consumers

The architectural point is important: use external locations at the cloud-storage boundary; downstream curated layers can use managed Unity Catalog tables.

18. Troubleshooting framework

Access denied

Trace the request in this order:

UC privilege
   ↓
Workspace binding
   ↓
Storage credential
   ↓
IAM trust policy
   ↓
IAM permissions policy
   ↓
S3 bucket policy
   ↓
KMS policy

Cannot create an external location

Check CREATE EXTERNAL LOCATION on the metastore and storage credential, workspace binding, credential validity, S3 path and IAM permissions.

Read works but write fails

Check READ FILES, WRITE FILES, S3 PutObject, KMS permissions and bucket encryption requirements.

Works in one workspace but not another

Check workspace bindings first.

19. Hands-on lab

Build this chain:

AWS IAM Role → Storage Credential → External Location → External Table → Governed Query

Steps:

1. Create a non-production S3 prefix.
2. Create an IAM role with minimum required permissions.
3. Create the Unity Catalog storage credential.
4. Inspect the credential.
5. Create the external location.
6. Inspect the location and grants.
7. Grant READ FILES to a test group.
8. Grant CREATE EXTERNAL TABLE only to engineering.
9. Create an external Delta table.
10. Test read/write with separate identities.
11. Bind the location to the intended workspace.
12. Document the IAM → UC → S3 chain.

20. Architect decision matrix

Requirement	Recommended design
New Delta table, UC owns lifecycle	Managed table
Existing S3 data	External location + external table
Shared files	External location + volume
Production isolation	Workspace bindings + privileges
Cross-account S3	IAM role + trust + scoped S3 policy
Read-only domain	Read-only credential/location strategy
Multiple domains	Domain-aligned locations
Enterprise landing zone	External locations
Curated layers	Prefer managed tables

21. Interview questions

What is the difference between a storage credential and an external location?

A storage credential represents cloud authentication. An external location combines a cloud path with that credential and becomes the Unity Catalog governance boundary.

Can multiple external locations use one storage credential?

Yes. One credential can represent a common cloud identity while locations scope it to different paths.

Does access to a storage credential mean a user can read all S3 data?

No. Credential privileges govern use of the credential, especially creation of external locations. Data access is governed through external-location privileges and Unity Catalog objects.

Why use workspace bindings?

To prevent credentials or locations from being exercised from unintended workspaces, especially in dev/test/prod isolation.

How do you debug S3 AccessDenied?

Trace Unity Catalog privilege → workspace binding → storage credential → IAM trust → IAM permissions → bucket policy → KMS policy.

22. Production checklist

• IAM trust policy reviewed
• S3 permissions minimized
• Bucket policy reviewed
• KMS policy reviewed
• Environment prefixes isolated
• Storage credential ownership defined
• External-location ownership defined
• CREATE EXTERNAL LOCATION restricted
• File privileges granted by group
• Workspace bindings reviewed
• Paths documented
• Access tests automated
• External locations inventoried
• Orphaned locations periodically reviewed

23. Architect mental model

Identity → Credential → Path → Privilege

1. Identity — Which cloud identity can access the data?
2. Credential — How does Unity Catalog represent that identity?
3. Path — Which cloud storage prefix is governed?
4. Privilege — Which users/groups can use that governed path?

If you can answer those four questions, you understand the core of Unity Catalog cloud-storage security.

Official references

• Connect to cloud object storage using Unity Catalog
• Manage storage credentials
• Manage external locations
• Connect to an AWS S3 external location
• AWS S3 manual setup
• Design Unity Catalog architecture

---

Next: 07.5 Unity Catalog Volumes & File Governance