Skip to main content

02.5 Security Model

Phase 2 — Platform Architecture

Databricks security is not a single permission screen. It is a defense-in-depth architecture spanning identity, workspace controls, compute isolation, Unity Catalog, cloud IAM, networking, secrets, encryption and auditability.

Databricks security model

1. The security mental model​

Think of every request as moving through multiple control layers:

USER / SERVICE PRINCIPAL │ ▼ AUTHENTICATION SSO / OAuth / MFA │ ▼ ACCOUNT + WORKSPACE identity / entitlements / ACLs │ ▼ COMPUTE ISOLATION access mode / policies │ ▼ UNITY CATALOG privileges / ownership / ABAC │ ▼ CLOUD AUTHORIZATION IAM / storage credentials │ ▼ NETWORK VPC / endpoints / firewall / egress │ ▼ DATA + SERVICES S3 / Delta / databases / APIs

Then add two cross-cutting capabilities:

  • Secrets and encryption protect sensitive material.
  • Audit and monitoring provide evidence of what happened.

Databricks describes Unity Catalog as the unified governance layer for data and AI, while account-level identity and workspace controls establish who can operate the platform.

2. Authentication vs authorization​

These concepts are often confused.

ConceptQuestionExample
AuthenticationWho are you?SSO login
AuthorizationWhat can you do?SELECT on a table
EntitlementWhat can you use?Workspace access
IsolationHow is execution separated?Standard / Dedicated compute
Cloud authorizationWhat cloud resource can execution access?S3 IAM role
AuditingWhat happened?Audit event

An authenticated user can still be unauthorized to read a table.

3. Databricks identities​

Databricks currently has three core identity types:

Users​

Human identities used for interactive work.

Groups​

Groups are the preferred unit for scalable access assignment. Instead of granting access individually to hundreds of users, grant the permission to a group and manage membership in the identity provider.

Service principals​

Non-human identities for automation, production jobs, CI/CD and applications.

Databricks recommends service principals for production workloads so that production execution is not dependent on an individual employee account.

Enterprise pattern​

Corporate IdP │ ▼ Users + Groups + Service Principals │ ▼ Databricks Account │ ├── Workspace A ├── Workspace B └── Workspace C

4. Identity federation​

Identity federation allows account-level identities to be assigned to workspaces instead of maintaining disconnected identity populations per workspace.

Identity Provider │ SSO / SCIM │ ▼ Databricks Account │ ┌────┼────┐ ▼ ▼ ▼ DEV TEST PROD

Databricks enables identity federation by default for new workspaces. Centralizing identity improves lifecycle management and reduces inconsistent permissions.

5. SSO and MFA​

Enterprise authentication should normally use the organization's identity provider.

Typical flow:

Browser │ ▼ Corporate IdP │ SSO + MFA │ ▼ Databricks

Databricks supports SSO integrations including SAML and OIDC. MFA should be enforced at the identity-provider level as part of the enterprise authentication policy.

6. Account admin vs workspace admin​

Databricks has different administrative scopes.

RoleTypical scope
Account adminAccount, identities, workspaces, account-level configuration
Workspace adminWorkspace settings, identities and workspace access controls
Delegated / feature-specific adminNarrow administrative responsibility

Do not give every platform engineer account-admin privileges.

Use least privilege and segregation of duties. Databricks recommends limiting the number of administrators and separating administrative responsibilities.

7. Workspace access​

Workspace access answers:

Can this identity use this workspace?

It does not automatically mean:

Can this identity read every table?

A useful separation is:

Account identity │ ▼ Workspace assignment │ ▼ Workspace entitlement │ ▼ Workspace object permission │ ▼ Unity Catalog data permission

This separation becomes essential in multi-workspace enterprise architectures.

8. Workspace ACLs​

Workspace objects such as notebooks, jobs and dashboards can have their own access controls.

Example:

Finance Job ├── CAN VIEW → Finance Analysts ├── CAN MANAGE RUN → Finance Operations └── CAN MANAGE → Platform Team

Do not confuse workspace object permissions with data permissions. A user may have permission to run a job but still be denied access to a governed table.

9. Unity Catalog authorization​

Unity Catalog governs data and AI assets using complementary access-control mechanisms including privileges, ownership, ABAC, row/column controls and workspace restrictions. Databricks currently recommends ABAC for centralized, scalable policy enforcement based on governed tags.

Typical hierarchy:

METASTORE │ ▼ CATALOG │ ▼ SCHEMA │ ▼ TABLE / VIEW / VOLUME / OTHER ASSET

10. Privileges and ownership​

Unity Catalog permissions answer questions such as:

  • Can the principal use the catalog?
  • Can the principal use the schema?
  • Can the principal SELECT the table?
  • Can the principal MODIFY the table?
  • Can the principal create objects?

Every securable object has an owner. Owners have control over privileges on that object.

Least-privilege example​

finance_analysts │ ├── USE CATALOG → finance ├── USE SCHEMA → reporting └── SELECT → reporting.sales_summary

Do not grant broad workspace or catalog privileges when a narrower table-level privilege satisfies the requirement.

11. Workspace bindings​

Workspace bindings add another boundary:

Where can this Unity Catalog object be accessed?

Example:

production catalog │ ├── PROD workspace → allowed ├── TEST workspace → allowed └── DEV workspace → denied

This is valuable when the same metastore serves multiple environments or business workspaces.

12. Row-level security​

Row-level security answers:

Can this user see this row?

Example:

SALES ┌───────────────┐ │ Region │ Sales│ ├───────────────┤ │ EU │ ... │ │ US │ ... │ │ APAC │ ... │ └───────────────┘

An EU analyst may receive only EU rows.

Unity Catalog supports table-level row filters. For centralized policies across many tables, Databricks recommends ABAC.

13. Column-level security​

Column masking answers:

Can this user see the actual value of this column?

Example:

customer_id | email | ssn 10001 | ... | *--1234

Finance users might see salary; general analysts might see a masked value.

Unity Catalog supports column masks and centralized ABAC policies.

14. ABAC — attribute-based access control​

ABAC scales security beyond manually granting permissions table by table.

DATA tag: sensitivity = confidential tag: region = EU │ ▼ ABAC POLICY │ ▼ USER ATTRIBUTES region = EU clearance = confidential │ ▼ ALLOW / FILTER / MASK

Governed tags become policy inputs. A policy can apply across a catalog, schema or other supported scope instead of being duplicated across every table.

Current ABAC capabilities include row/column policies, with additional policy types and supported object scopes evolving over time. Check current documentation before adopting preview features.

15. Dynamic views vs row filters vs ABAC​

MechanismBest fit
Dynamic viewReusable SQL-based presentation/security logic
Table row filterOne-table row-level rule
Column maskOne-table column protection
ABACCentralized, scalable attribute-driven policy

Do not build hundreds of independent dynamic views when one governed policy can express the enterprise rule cleanly.

16. Service principals​

Production automation should not depend on an employee identity.

Weak model​

Developer account │ ▼ Production Job

Problems:

  • Employee leaves
  • Ownership becomes ambiguous
  • Credentials may be personal
  • Audit trails become harder to interpret

Better model​

Service Principal │ ▼ Production Job │ ▼ Unity Catalog + Cloud IAM

Databricks explicitly recommends service principals for production jobs and production data modification.

17. Secrets and credentials​

Never put passwords, API keys or tokens directly into notebooks or source control.

Use a governed secret-management pattern.

Current Databricks options include Unity Catalog secrets and integration with external secret managers. Unity Catalog secrets are securable objects under the three-level namespace and can be governed with Unity Catalog privileges.

Preferred architecture​

Application / Job │ ▼ Governed secret / service credential │ ▼ External service

Databricks also provides Unity Catalog service credentials for accessing supported external cloud services, with access governed through Unity Catalog privileges and auditability.

18. Cloud IAM​

Unity Catalog authorization and AWS IAM solve different problems.

Databricks identity │ ▼ Unity Catalog │ ▼ Storage credential / service credential │ ▼ AWS IAM │ ▼ S3 / AWS service

Think of it as:

Databricks decides what the principal is allowed to access in the governed platform; AWS IAM controls cloud-resource authorization.

Never assume a Unity Catalog GRANT alone can overcome a missing cloud authorization path.

19. Network security​

Authorization is not connectivity.

USER │ ▼ Databricks │ ▼ Compute │ ├── Network controls │ ├── VPC │ ├── subnets │ ├── security groups │ ├── endpoints │ └── egress controls │ ▼ S3 / DB / API

A request can fail because:

  1. The user lacks a Databricks privilege.
  2. The compute identity lacks cloud authorization.
  3. The network cannot reach the target.

These are different failure domains.

20. Encryption​

Enterprise security normally covers:

  • Data at rest
  • Data in transit
  • Secret values
  • Cloud storage
  • Databricks-managed services
  • Customer-managed keys where required

Encryption should be designed together with key management, rotation, access control and auditability—not treated as a checkbox.

21. Auditability​

Security without evidence is difficult to operate.

Databricks provides audit logging for system activity, and Unity Catalog provides auditing for governed data and AI activity. The system.access.audit system table is an important source for account-level audit analysis.

Questions an enterprise should be able to answer:

  • Who accessed this table?
  • When?
  • From which workspace?
  • What action occurred?
  • Which service principal ran the job?
  • Who changed the permission?
  • What data was affected?

22. Lineage as a security capability​

Lineage is not only a governance convenience.

Sensitive source │ ▼ Transformation │ ▼ Gold table │ ▼ Dashboard / Model

Unity Catalog can capture lineage automatically, including column-level lineage for supported workloads, across workspaces attached to the metastore. This enables impact analysis and tracing sensitive data flows.

23. Defense-in-depth architecture​

An enterprise implementation should look approximately like:

┌─────────────────────────────────────────┐ │ IDENTITY │ │ SSO • MFA • Groups • Service Principals│ ├─────────────────────────────────────────┤ │ PLATFORM │ │ Account • Workspace • Entitlements │ ├─────────────────────────────────────────┤ │ COMPUTE │ │ Access Mode • Policies • Isolation │ ├─────────────────────────────────────────┤ │ DATA GOVERNANCE │ │ Unity Catalog • RBAC • ABAC • RLS/CLS │ ├─────────────────────────────────────────┤ │ CLOUD │ │ IAM • Storage Credentials • KMS │ ├─────────────────────────────────────────┤ │ NETWORK │ │ VPC • Endpoints • Firewall • Egress │ ├─────────────────────────────────────────┤ │ DATA │ │ S3 • Delta • DB • APIs │ └─────────────────────────────────────────┘ │ Audit + Monitoring across layers

24. Enterprise environment pattern​

CORPORATE IDENTITY PROVIDER │ SSO / SCIM / MFA │ ▼ DATABRICKS ACCOUNT │ ┌──────┼──────┐ ▼ ▼ ▼ DEV TEST PROD │ │ │ └──────┼───────┘ ▼ UNITY CATALOG │ ┌──────┼───────┐ ▼ ▼ ▼ RBAC ABAC LINEAGE │ ▼ S3

Production workloads use service principals, policy-controlled compute and controlled cloud/network access.

25. Common security anti-patterns​

1. Individual grants everywhere​

Creates permission sprawl. Prefer groups and centralized policies.

2. Human accounts running production​

Creates lifecycle and audit problems. Prefer service principals.

3. Secrets in notebooks​

Creates credential leakage risk. Use governed secret-management mechanisms.

4. Treating Unity Catalog as AWS IAM​

They are complementary control planes.

5. Giving everyone workspace admin​

Violates least privilege and increases blast radius.

6. One security boundary for every environment​

DEV and PROD should not automatically have identical access paths.

7. No audit strategy​

If you cannot reconstruct who changed or accessed something, incident response becomes difficult.

8. Assuming network = authorization​

A reachable S3 bucket does not mean the Databricks principal is authorized to use it.

26. Hands-on security design​

Scenario​

An enterprise has:

  • 3 workspaces: DEV, TEST, PROD
  • Finance and Sales domains
  • PII in customer data
  • Power BI consumers
  • Automated production pipelines
  • AWS S3 as the storage foundation

Design​

  1. Corporate IdP provides SSO and MFA.
  2. Groups represent Finance, Sales, Engineering and Platform teams.
  3. Service principals execute production jobs.
  4. DEV/TEST/PROD workspace access is assigned explicitly.
  5. Unity Catalog provides catalog/schema/table privileges.
  6. ABAC governs sensitive data at scale.
  7. Row filters restrict regional visibility.
  8. Column masks protect PII.
  9. AWS IAM and storage credentials control S3 access.
  10. Network controls restrict egress and private connectivity.
  11. Audit logs feed security monitoring.
  12. Lineage supports impact analysis.

Result​

Identity → Workspace → Compute → UC → IAM → Network → Data │ │ └──── Audit ───────┘

27. Troubleshooting security failures​

Start from the outside and move inward:

LOGIN FAILS? │ ├── SSO / MFA / identity? │ ▼ WORKSPACE ACCESS FAILS? │ ├── assignment / entitlement / ACL? │ ▼ TABLE ACCESS FAILS? │ ├── UC privilege / ownership / ABAC? │ ▼ STORAGE ACCESS FAILS? │ ├── storage credential / AWS IAM? │ ▼ CONNECTION FAILS? │ └── VPC / endpoint / firewall / egress?

Do not immediately change permissions. First identify which security layer rejected the request.

28. Architect interview scenarios​

Scenario A — Analyst can open the workspace but cannot query a table​

Check workspace access first, then Unity Catalog privileges, catalog/schema usage and any ABAC, row-filter or mask policy.

Scenario B — Job works for developer but fails in production​

Check whether the production job uses a service principal, whether that principal has UC privileges, and whether its cloud identity/network path has the required access.

Scenario C — User can SELECT a table but sees no rows​

Investigate row filters or ABAC rather than only checking table-level SELECT.

Scenario D — Databricks can see S3 but query fails​

Separate network connectivity from AWS authorization and Unity Catalog authorization.

Scenario E — Auditor asks who changed access​

Use audit events and correlate the identity, timestamp, object, action and workspace/account context.

29. Architect's security checklist​

AreaEnterprise default
AuthenticationSSO + MFA
Identity lifecycleFederation + automated provisioning
Human accessGroup-based
Production automationService principals
Data authorizationUnity Catalog
Fine-grained securityABAC / RLS / column masks
ComputeStandard + policy controlled by default
CloudLeast-privilege IAM
SecretsGoverned secret manager / UC mechanisms
NetworkPrivate and restricted where required
EncryptionAt rest + in transit + key governance
AuditCentralized audit monitoring
LineageUnity Catalog lineage

30. Final mental model​

WHO? SSO / MFA / Identity │ ▼ WHERE? Account / Workspace │ ▼ HOW? Compute isolation / Policies │ ▼ WHAT DATA? Unity Catalog / RBAC / ABAC │ ▼ WHICH CLOUD RESOURCE? IAM / Storage Credentials │ ▼ CAN IT CONNECT? Network │ ▼ CAN WE PROVE IT? Audit / Lineage / Monitoring

Security is the combination of identity, authorization, isolation, network controls, data governance and evidence—not a single permission.

Official references​

Next​

03. Apache Spark Fundamentals — Spark architecture, driver/executors, jobs, stages, tasks, transformations, actions, lazy evaluation, DAGs, partitions, shuffle and Spark UI.