07.9 Data Classification, Sensitive Data Discovery & Automated Governance

Enterprise governance cannot scale if humans must manually inspect every column.

Unity Catalog Data Classification automatically scans governed data and identifies sensitive content such as PII, financial information, credentials, and other supported classes. Detected classifications can become metadata that drives governance, discovery, and ABAC policies.

This creates an important progression:

discover → classify → tag → govern → monitor → remediate

Databricks Data Classification became generally available in 2026. The current implementation uses an agentic classification system and supports built-in global and regional classification tags. The Data Classification API and Terraform resource became generally available in October 2026. Custom classifiers are currently Beta, while automatic tag-assignment and view-scanning capabilities have their own preview/beta status. Always verify feature availability for your workspace and region before production use.

Official references:

• Data Classification
• Supported classification tags
• Data Classification release notes
• Data discovery in Unity Catalog
• Custom classifiers
• Data Classification CLI

1. Why classification is an architect problem

Imagine an enterprise lakehouse with:

• 8,000 tables;
• 150,000 columns;
• SAP, Salesforce, APIs and files;
• customer, employee and supplier data;
• multiple countries;
• multiple workspaces;
• several BI and AI consumers.

A manual inventory quickly becomes stale.

The architect therefore needs an automated feedback loop:

Data estate
    ↓
Discovery
    ↓
Sensitive-data detection
    ↓
Classification tags
    ↓
Policy / ABAC
    ↓
Controlled access
    ↓
Monitoring + remediation

Classification is not itself authorization.

A detected PII tag does not automatically mean a user is denied access. The tag becomes a governance signal that can be used by policies such as ABAC.

---

2. Classification vs tagging vs governance

These concepts must remain separate.

Concept	Purpose
Classification	Determine what the data appears to contain
Tagging	Record a reusable metadata attribute
Governance	Decide what controls apply
Authorization	Enforce who can access what
Discovery	Help users find and understand assets
Lineage	Explain where data came from and where it goes
Audit	Record activity and changes

A good enterprise architecture connects them without treating them as the same thing.

---

3. Built-in classification

Current Data Classification supports many built-in classes.

Examples include:

• email addresses;
• phone numbers;
• names;
• dates of birth;
• bank numbers;
• credit cards;
• passports;
• IP addresses;
• credentials/secrets;
• health data;
• compensation;
• location;
• vehicle identification numbers.

Regional classes are also available.

For India, current supported examples include:

• PAN;
• Aadhaar;
• ESIC;
• UAN;
• voter ID.

The exact class catalog evolves over time, so production designs should reference the current supported-class documentation rather than hard-code an old list.

---

4. Classification tags

Detected classes are represented through system classification tags.

A simplified example:

customer_email
      ↓
classifier
      ↓
class.email_address
      ↓
PII metadata
      ↓
ABAC policy
      ↓
mask for non-privileged users

This is powerful because the security policy can respond to what the column contains, rather than relying only on a manually maintained column-name convention.

For example, these columns may all contain email addresses:

• email;
• contact_email;
• customer_contact;
• primary_mail;
• supplier_email_address.

A name-based rule would require manual maintenance.

Classification provides a semantic signal.

---

5. Classification is probabilistic governance input

Never treat automated classification as infallible.

A production design should distinguish:

detection → review → policy activation

For high-impact controls:

Detected
   ↓
Confidence / evidence
   ↓
Human review where required
   ↓
Governed tag
   ↓
Policy

This is especially important for regulated data where a false negative can create material exposure.

---

6. Classification coverage

A mature governance program should measure coverage.

Useful metrics:

• percentage of catalogs enabled;
• percentage of tables scanned;
• percentage of columns classified;
• number of sensitive columns;
• number of unreviewed detections;
• number of protected sensitive columns;
• percentage of sensitive columns with policy coverage;
• percentage of users accessing unmasked sensitive data;
• number of classification changes over time.

Think of governance as an observable system.

---

7. Data Classification and ABAC

This is one of the most important connections in the entire Unity Catalog section.

Data Classification
       ↓
class.* tags
       ↓
ABAC policy
       ↓
Row filter / column mask
       ↓
Consumer

Example:

class.email_address
       ↓
PII masking policy
       ↓
email visible only to approved groups

The classification engine discovers the policy input.

ABAC enforces the policy.

This separation is architecturally clean.

---

8. Classification and data discovery

Classification also improves discovery.

A data consumer can ask:

• Which tables contain PII?
• Which datasets contain financial data?
• Where are credentials stored?
• Which customer datasets contain phone numbers?
• Which assets are sensitive?

This changes the catalog from a passive table registry into an enterprise data-discovery layer.

Databricks also provides broader discovery signals such as tags, certified/deprecated status, domains, semantic metadata and usage information.

---

9. Classification and lineage

Classification becomes substantially more useful when combined with lineage.

Suppose:

SAP Customer
   ↓
Bronze
   ↓
Silver
   ↓
Gold Customer 360
   ↓
BI dashboard

If the Bronze source contains an email address, lineage can help determine which downstream assets may inherit or expose that sensitive attribute.

Architecturally:

classification tells you what is sensitive; lineage tells you where it travels.

This enables impact analysis.

---

10. Sensitive-data propagation

Do not assume that a sensitive source column remains sensitive only at the source.

A transformation can:

• copy it;
• rename it;
• derive another value;
• concatenate it;
• aggregate it;
• hash it;
• mask it;
• expose it through a view.

Therefore governance needs both:

1. classification;
2. lineage-aware analysis.

Current Databricks capabilities include tag propagation in Beta for supported scenarios. Treat propagation behavior as feature-specific and verify the current support matrix before relying on it for a regulatory control.

---

11. Custom classifiers

Built-in classes cannot represent every enterprise identifier.

Examples:

• internal employee ID;
• proprietary product code;
• supplier account number;
• partner identifier;
• confidential project code.

Databricks currently provides Custom Classifiers in Beta for organization-specific sensitive data.

The conceptual workflow is:

Business definition
      ↓
Governed tag
      ↓
Representative samples / description
      ↓
Classifier preview
      ↓
Review
      ↓
Scheduled classification

Do not build a custom classifier merely because a column has an unusual name.

First determine whether an existing classification class already solves the problem.

---

12. Classification of views

Current Data Classification can scan Unity Catalog views in Beta when view scanning is enabled.

This is important because sensitive data often appears in curated views rather than only physical tables.

However, view scanning has cost implications because the classification system must scan additional assets.

Architectural rule:

enable view scanning when the governance benefit justifies the additional classification workload and cost.

---

13. Automatic tag assignment

Data Classification focuses on detecting sensitive data.

A broader automated-tagging strategy can also assign organization-defined governance metadata.

Examples:

sensitivity = restricted
domain = finance
owner = procurement
lifecycle = critical
certification = approved

These are different from sensitive-data detection.

A mature enterprise usually separates:

system classification tags

from

organization governance tags.

This prevents the tag namespace from becoming chaotic.

---

14. Tag taxonomy

A useful taxonomy can have four layers.

Layer 1 — System classification

Examples:

class.email_address
class.credit_card
class.in_pan

Layer 2 — Sensitivity

sensitivity.public
sensitivity.internal
sensitivity.confidential
sensitivity.restricted

Layer 3 — Business ownership

domain.finance
domain.procurement
domain.hr
domain.sales

Layer 4 — Operational governance

criticality.high
lifecycle.active
certification.approved

Do not mix these concepts into one overloaded tag.

---

15. India-focused governance example

For an Indian enterprise, consider a customer table containing:

• PAN;
• Aadhaar;
• email;
• phone;
• address.

Classification may detect regional identifiers such as PAN and Aadhaar.

A governance architecture can then map:

class.in_pan
class.in_aadhaar
class.email_address
      ↓
Restricted classification
      ↓
ABAC policy
      ↓
Mask / restrict
      ↓
Approved consumers

This is a much stronger architecture than relying on analysts to remember which columns are sensitive.

The actual legal interpretation and compliance obligation must still come from the organization's legal/compliance function.

---

16. SAP / BW example

Imagine a BW extraction feeding Databricks:

S/4HANA
   ↓
ODP / SLT / CDC
   ↓
Bronze
   ↓
Silver
   ↓
Gold

Sensitive attributes may include:

• employee information;
• supplier bank data;
• customer contact details;
• tax identifiers;
• payment information.

The architect should establish:

Source classification
       ↓
Lakehouse classification
       ↓
Lineage
       ↓
Policy
       ↓
Consumer control

Do not assume that because SAP authorization protected a field at source, the copied lakehouse representation is automatically governed.

The target platform must enforce its own security boundary.

---

17. Classification results as a governance data product

At enterprise scale, classification results themselves become valuable metadata.

Build a governance model containing:

• asset;
• column;
• detected class;
• detection date;
• confidence/evidence where available;
• review status;
• owner;
• policy status;
• downstream impact;
• exception status.

Then executives can answer:

"How much sensitive data do we have, where is it, and how well is it protected?"

That is an architect-level governance capability.

---

18. Exposure analysis

A particularly useful metric is sensitive-data exposure.

Conceptually:

Sensitive columns
      ×
Consumers
      ×
Access level
      ×
Policy coverage

For example:

Asset	Sensitive class	Unmasked users	Policy	Risk
customer.email	email	17	mask	Medium
supplier.bank	bank	4	mask	High
employee.pan	PAN	0	restricted	Low

This converts classification from a discovery feature into a risk-management process.

---

19. Classification + audit

Combine classification with audit logs.

You can ask:

• Who accessed sensitive data?
• Which sensitive columns were exposed?
• Was the value masked?
• Did access patterns change after a policy deployment?
• Which users accessed unmasked values?

This creates a governance feedback loop:

Classification
      ↓
Policy
      ↓
Access
      ↓
Audit
      ↓
Risk analysis
      ↓
Policy improvement

---

20. Hands-on Lab 1 — Enable classification

Choose a non-production catalog.

Create a small dataset containing:

• email;
• phone;
• name;
• PAN-like test values;
• ordinary identifiers.

Enable Data Classification according to the current Databricks workspace procedure.

Run a classification scan.

Review the detected columns.

Do not use real personal data in a learning environment.

---

21. Hands-on Lab 2 — Build a sensitive-data inventory

Create an inventory with:

Catalog	Schema	Table	Column	Class	Policy
sales	customer	customer	email	email	mask
finance	supplier	bank	account	bank	restrict

Then answer:

1. Which sensitive classes exist?
2. Which have no policy?
3. Which are accessed by unmasked users?
4. Which require manual review?

This becomes your first governance dashboard specification.

---

22. Hands-on Lab 3 — Classification → ABAC

Take a classified email column.

Implement:

classification tag
       ↓
ABAC policy
       ↓
column mask

Test two identities:

• privileged data steward;
• standard analyst.

Verify that the same data asset produces different results according to policy.

This lab connects 07.8 and 07.9.

---

23. Hands-on Lab 4 — Custom classifier

Design a fictional enterprise identifier:

EMP-123456
EMP-482901
EMP-774201

Create a governed tag representing the internal identifier.

Use a custom classifier in a non-production environment if the Beta capability is available.

Evaluate:

• precision;
• false positives;
• false negatives;
• policy impact.

Never promote an experimental classifier directly into a regulatory control without validation.

---

24. Hands-on Lab 5 — Classification + lineage

Take a sensitive source column.

Trace:

Bronze
 ↓
Silver
 ↓
Gold
 ↓
View
 ↓
Dashboard

Document:

• where the sensitive attribute originated;
• where it exists;
• where it is transformed;
• where it is masked;
• where it is exposed.

This is the foundation of sensitive-data impact analysis.

---

25. Troubleshooting

Classification is not detecting a column

Check:

• catalog is enabled;
• table is supported;
• scan has completed;
• classification class is supported;
• regional class matches the workspace region;
• data actually contains detectable values;
• classification configuration is active.

A tag exists but ABAC does not apply

Classification and enforcement are separate.

Check:

1. tag;
2. ABAC policy;
3. policy scope;
4. principal;
5. UDF;
6. object privileges;
7. runtime support.

Classification produces a false positive

Review the detection and determine whether:

• the value format is genuinely ambiguous;
• the class is too broad;
• a custom classifier is needed;
• the detection should be excluded where supported.

Classification is expensive

Review:

• number of catalogs;
• number of tables;
• view scanning;
• scan frequency;
• unnecessary catalog coverage.

Classification should be treated as an operational workload.

---

26. Production operating model

A mature organization should define ownership.

Capability	Owner
Classification taxonomy	Data Governance
Sensitive-data review	Data Steward
ABAC policy	Security / Data Platform
Tag administration	Unity Catalog Admin
Legal interpretation	Compliance / Legal
Lineage	Data Platform
Access monitoring	Security
Exceptions	Data Owner + Security

Do not make the Databricks platform team the sole owner of business classification.

---

27. Architect decision framework

Use built-in classification when

The sensitive class matches Databricks-supported detection.

Use custom classification when

The enterprise has a domain-specific identifier not covered by built-in classes.

Use manual tags when

The attribute is organizational metadata rather than detected sensitive content.

Use ABAC when

The classification should drive reusable access policies.

Use lineage when

You need downstream impact analysis.

Use audit when

You need evidence of access and policy effectiveness.

The strongest design combines all of them.

---

28. Enterprise reference architecture

                 DATA SOURCES
                      |
             Bronze / Silver / Gold
                      |
                      v
              Unity Catalog
                      |
        +-------------+-------------+
        |             |             |
 Classification    Lineage       Metadata
        |             |             |
        +-------------+-------------+
                      |
                Governed Tags
                      |
                      v
                 ABAC Policies
                      |
          +-----------+-----------+
          |                       |
      Row Filter             Column Mask
          |                       |
          +-----------+-----------+
                      |
                  Consumers
              BI / SQL / AI / Apps
                      |
                      v
                    Audit
                      |
                      v
             Governance Metrics

This is the target architecture for an enterprise lakehouse.

---

29. Interview questions

What is Databricks Data Classification?

An automated Unity Catalog capability that identifies and tags supported sensitive data.

Does classification enforce access?

No. Classification creates governance metadata. Controls such as ABAC can use that metadata for enforcement.

Why is classification useful for ABAC?

It allows policies to target what data actually contains instead of relying entirely on manually maintained object lists.

What is a governed tag?

A controlled metadata attribute whose key/value usage is governed centrally.

What is the difference between classification and a business tag?

Classification identifies sensitive content; business tags describe organizational concepts such as domain, ownership or criticality.

Why combine classification with lineage?

Classification tells you what is sensitive; lineage tells you where that sensitive information flows.

How would you govern 100,000 columns?

Automate classification and metadata collection, establish a controlled tag taxonomy, connect classification tags to ABAC, monitor policy coverage, and reserve human review for high-impact or ambiguous cases.

---

30. Production checklist

• Data Classification coverage is defined.
• Sensitive classes are mapped to enterprise requirements.
• System classification tags are separated from business tags.
• Tag taxonomy is documented.
• Classification results have owners.
• High-risk detections have review workflows.
• ABAC policies consume approved classifications.
• Sensitive data is traced through lineage.
• Sensitive-data access is audited.
• Classification coverage metrics exist.
• False positives/negatives are monitored.
• Custom classifiers are validated before production.
• Beta/preview features are not treated as regulatory dependencies without approval.
• Classification workload and cost are monitored.
• Legal/compliance interpretation remains with the appropriate business function.

---

31. Final mental model

The architect should remember:

              DISCOVER
                 ↓
             CLASSIFY
                 ↓
                TAG
                 ↓
              GOVERN
                 ↓
              ENFORCE
                 ↓
               AUDIT
                 ↓
              IMPROVE

Data Classification answers: "What sensitive data do we have?"

Unity Catalog metadata answers: "What do we know about it?"

Lineage answers: "Where did it come from and where does it go?"

ABAC answers: "What policy should apply?"

Row filters and masks answer: "What can this user actually see?"

Audit answers: "What happened?"

That complete loop is the foundation of automated enterprise data governance.