13. Enterprise CI/CD & Declarative Automation Bundles
Learning objective: move from manually editing Databricks workspace objects to an enterprise release process where code, jobs, pipelines, permissions and configuration are versioned, validated, tested, promoted and recoverable.
A data platform can have excellent Spark transformations, Delta architecture, Unity Catalog governance and Lakeflow pipelines — and still be operationally immature if production changes are made manually.
Enterprise delivery needs repeatability.
The central question is:
How do we prove that the exact version reviewed by the team is the version deployed to the correct workspace, under the correct identity, with the expected configuration and a recoverable release history?
Databricks recommends Declarative Automation Bundles for CI/CD. They were formerly called Databricks Asset Bundles. This rename is non-breaking: existing configuration and the databricks bundle CLI command remain familiar.
Official overview: https://docs.databricks.com/aws/en/dev-tools/bundles/
1. Why CI/CD matters in data engineering
Consider a developer making a production change directly in the workspace:
OPEN NOTEBOOK
|
v
EDIT CODE
|
v
CLICK RUN
|
v
CHANGE JOB IN UI
|
v
PRODUCTION
What can go wrong?
- nobody reviewed the change;
- the source repository does not contain the production version;
- the development and production jobs differ;
- a user accidentally changes the wrong catalog;
- a library version is not reproducible;
- deployment order is undocumented;
- rollback is unclear;
- audit evidence is incomplete.
Now compare an automated release:
GIT COMMIT
|
v
PULL REQUEST
|
v
REVIEW + TEST
|
v
BUNDLE VALIDATION
|
v
DEV → STAGING → PROD
|
v
MONITOR + RELEASE RECORD
The goal is not to eliminate engineering judgement. It is to make the release process controlled, testable and repeatable.
2. What CI/CD means
Continuous Integration
Frequently integrate code changes into a shared source repository.
Typical activities:
- pull request review;
- linting;
- static analysis;
- unit tests;
- dependency checks;
- bundle validation;
- merge protection.
Continuous Delivery
Keep every accepted change deployable and promote it through controlled environments.
Typical activities:
- build artifacts;
- deploy to development;
- run integration tests;
- deploy to staging;
- obtain approval;
- deploy to production;
- run smoke tests.
Continuous Deployment
Automatically deploy qualifying changes to production without a manual release approval.
This may be appropriate for low-risk applications, but many enterprise data products require explicit approvals because a deployment can affect financial reporting, regulated data or downstream business processes.
Important distinction
CI/CD is the release process.
Declarative Automation Bundles are a Databricks-native project and deployment definition used within that process.
GitHub Actions, Azure DevOps, GitLab CI and Jenkins can execute the automation.
A bundle does not replace the CI platform.
3. Animated release architecture
How to read the flow: a feature branch enters review and quality gates. The same reviewed release is promoted to development, staging and production. Identity federation, release traceability and recovery controls apply across the entire lifecycle.
The most important architectural principle is:
Build and verify once, then promote the same version through environments. Do not create slightly different business logic for DEV, TEST and PROD.
4. What are Declarative Automation Bundles?
A Declarative Automation Bundle (DAB) is an end-to-end, source-controlled definition of a Databricks project.
A bundle can include:
- source code;
- notebooks;
- SQL files;
- Python packages and build configuration;
- Lakeflow Jobs;
- Lakeflow Declarative Pipelines;
- dashboards;
- model-related resources;
- permissions;
- environment targets;
- tests and deployment metadata.
A bundle is a project-level deployment unit.
Conceptually:
BUNDLE
|
+── SOURCE CODE
|
+── RESOURCE DEFINITIONS
| |
| +── JOBS
| +── PIPELINES
| +── DASHBOARDS
| +── OTHER SUPPORTED RESOURCES
|
+── ENVIRONMENT CONFIGURATION
|
+── TESTS
|
+── BUILD / DEPLOYMENT SETTINGS
The Databricks CLI uses the bundle configuration to validate, deploy and run those resources.
Official reference: https://docs.databricks.com/aws/en/dev-tools/bundles/
5. Why the name changed
The former name was Databricks Asset Bundles.
The current name is Declarative Automation Bundles.
The new name better communicates that the bundle defines and automates a complete project lifecycle, rather than merely collecting assets.
The rename is non-breaking:
- the databricks bundle CLI command remains;
- existing bundle configuration does not need to be rewritten merely because of the rename;
- older documentation, tutorials and community examples may still say Asset Bundles.
When reading legacy examples, understand that both names refer to the same evolving bundle capability.
Official FAQ: https://docs.databricks.com/aws/en/dev-tools/bundles/faqs
6. Bundle vs Git repository vs CI platform
These are different responsibilities.
| Component | Responsibility |
|---|---|
| Git repository | Stores source history, reviews and approved changes |
| Declarative Automation Bundle | Defines the Databricks project and deployable resources |
| Databricks CLI | Validates, deploys and runs bundle resources |
| CI/CD platform | Executes tests, approvals and deployment stages |
| Databricks workspace | Hosts deployed Databricks resources |
| Unity Catalog | Governs data, storage and permissions |
Think of the relationship this way:
GIT
|
+---- source code
|
+---- databricks.yml
|
+---- resource YAML
|
+---- tests
|
v
CI/CD PLATFORM
|
v
DATABRICKS CLI
|
v
DECLARATIVE AUTOMATION BUNDLE
|
v
TARGET WORKSPACE
A Git repository alone does not create a safe deployment process.
A bundle alone does not create an approval workflow.
The complete architecture needs both.
7. Which resources belong in a bundle?
Common resource families include:
| Resource | Why manage it as code? |
|---|---|
| Lakeflow Job | Version task definitions, schedules and permissions |
| Lakeflow Pipeline | Version pipeline configuration and deployment |
| Notebook / Python / SQL | Version business logic |
| Dashboard | Version supported dashboard definitions |
| Model resources | Version supported ML resources and deployment configuration |
| Permissions | Review intended access changes |
| Libraries / artifacts | Make dependencies and executable versions traceable |
The exact set of supported resources evolves with the Databricks CLI and platform.
Check the current resource reference before committing to a resource type.
Official resource reference: https://docs.databricks.com/aws/en/dev-tools/bundles/resources
8. What should not be placed in Git?
Version control almost all project source and configuration, but never commit secrets or sensitive runtime data.
Good candidates for Git
- Python, SQL and notebook source;
- databricks.yml;
- resource YAML;
- test code;
- dependency manifests;
- CI workflow files;
- environment configuration templates;
- documentation.
Do not commit
- access tokens;
- passwords;
- OAuth client secrets;
- cloud credentials;
- private keys;
- production data extracts;
- files containing PII;
- large compiled build output when it should be uploaded separately.
Use a secret manager and the platform's supported identity and secret integrations.
Official best practices: https://docs.databricks.com/aws/en/developers/best-practices/
9. Recommended project structure
A simple enterprise project can look like this:
retail-lakehouse/
|
+── databricks.yml
|
+── resources/
| |
| +── jobs.yml
| +── pipelines.yml
|
+── src/
| |
| +── ingestion/
| +── transformations/
| +── quality/
| +── shared/
|
+── notebooks/
|
+── sql/
|
+── tests/
| |
| +── unit/
| +── integration/
| +── data_quality/
|
+── scripts/
|
+── .github/
| |
| +── workflows/
| |
| +── databricks-cicd.yml
|
+── requirements.txt
+── pyproject.toml
+── .gitignore
This structure is illustrative. Teams can organize projects differently, but the bundle configuration must clearly include the intended resources and files.
10. The main configuration file: databricks.yml
A bundle must have one root configuration file named:
databricks.yml
This file defines the bundle and can include other configuration files.
A minimal example:
bundle:
name: retail_lakehouse
targets:
dev:
default: true
The bundle name is important because it participates in deployment identity and state management.
The targets section defines deployment environments such as development, staging and production.
Official configuration reference: https://docs.databricks.com/aws/en/dev-tools/bundles/settings
11. The bundle lifecycle
A bundle follows a repeatable lifecycle:
CREATE
|
v
DEVELOP
|
v
VALIDATE
|
v
DEPLOY
|
v
RUN
|
v
MONITOR
|
v
UPDATE / DESTROY WHEN APPROPRIATE
The core commands include:
databricks bundle init
databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run <resource-key> -t dev
databricks bundle summary
The CLI also provides resource generation, binding, deployment history and other management commands.
Do not run destructive commands such as bundle destroy against production as routine cleanup.
12. Initialize a new bundle
The Databricks CLI provides starter templates.
Start with:
databricks bundle init
The CLI presents available templates and generates a starter project.
You can also specify a custom template:
databricks bundle init <template-location>
A generated template is a starting point, not an enterprise architecture by itself.
Review:
- project structure;
- source file paths;
- job definitions;
- cluster or serverless configuration;
- package dependencies;
- target environments;
- permissions;
- tests;
- deployment identity.
Then customize it according to your platform standards.
13. Environment targets
A target represents a deployment configuration for a particular environment.
Typical targets:
dev
staging
prod
Each target can define or override:
- workspace host;
- deployment root path;
- variables;
- resource settings;
- permissions;
- deployment mode;
- run identity;
- environment-specific configuration.
The same business logic can be deployed to different workspaces.
Example
| Target | Workspace | Catalog |
|---|---|---|
| dev | Development workspace | dev_catalog |
| staging | Staging workspace | staging_catalog |
| prod | Production workspace | prod_catalog |
The code should remain consistent.
The environment-specific configuration changes.
14. Separate workspaces vs separate catalogs
These provide different levels of isolation.
Separate catalogs
Useful for logical separation of data domains and lifecycle boundaries within a workspace.
Separate workspaces
Provide stronger isolation for:
- permissions;
- workspace resources;
- compute policies;
- network configuration;
- operational blast radius;
- user access;
- release independence.
For small teams, development and production workspaces may be a reasonable start.
For growing teams, a development, staging and production workspace model is stronger.
Regulated environments may need additional account and network isolation.
Do not assume that separate catalogs alone provide the same isolation as separate workspaces.
15. Development deployment mode
A development target can use:
targets:
dev:
mode: development
default: true
Development mode applies development-oriented behavior.
Examples include resource name prefixes and development tags for supported resources.
This helps multiple developers deploy the same bundle into a shared development workspace without colliding as easily.
It is a convenience mode, not a substitute for access control or workspace isolation.
16. Production deployment mode
A production target commonly uses:
targets:
prod:
mode: production
Production mode applies production-oriented defaults for supported resources.
The settings can also be customized using presets.
Architect caution
Do not assume that setting production mode automatically makes the entire deployment safe.
You still need:
- reviewed code;
- correct target workspace;
- protected CI environment;
- production permissions;
- service identity;
- data contracts;
- integration tests;
- controlled deployment approval.
Deployment modes are optional convenience behaviors.
Official reference: https://docs.databricks.com/aws/en/dev-tools/bundles/deployment-modes
17. Example: multi-environment bundle configuration
The following is a simplified starting pattern:
bundle:
name: retail_lakehouse
include:
- resources/*.yml
variables:
catalog_name:
description: Target Unity Catalog catalog
default: dev_catalog
targets:
dev:
mode: development
default: true
workspace:
host: https://<dev-workspace-host>
variables:
catalog_name: dev_catalog
staging:
mode: production
workspace:
host: https://<staging-workspace-host>
variables:
catalog_name: staging_catalog
prod:
mode: production
workspace:
host: https://<prod-workspace-host>
variables:
catalog_name: prod_catalog
run_as:
service_principal_name: <runtime-service-principal-name>
Replace placeholders with actual workspace hosts and the configured runtime principal.
This is an example, not a complete deployment file for every organization.
Before using it, confirm that the resource definitions, permissions, paths and workspace identities match your workspace setup.
18. Workspace root path and bundle identity
This is one of the most important operational concepts.
A bundle deployment has an identity based on:
- bundle name;
- target;
- workspace;
- deployment identity and associated state location.
The workspace root path controls where deployment files and state are stored.
The CLI needs to identify which deployed resources belong to a specific bundle deployment.
Conceptually:
WORKSPACE
|
+── bundle deployment state
|
+── synchronized source files
|
+── deployed artifacts
|
+── resource references
A production deployment should use a stable, unique, team-owned location.
Architect rule
Use a unique root path for each bundle and target, and keep the deployment identity stable.
If two independent bundle deployments accidentally share the same identity, they can interfere with each other's state and resources.
The default root path is based on the deployer identity, bundle name and target. Explicitly design this for CI/CD rather than allowing an accidental shared identity.
Official CLI reference: https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands
19. Deployment identity vs runtime identity
Do not confuse these two identities.
Deployment identity
The identity used by CI/CD or the developer to:
- authenticate to the workspace;
- deploy resources;
- update bundle state;
- create or modify supported resources.
Runtime identity
The identity used by a deployed job or pipeline when it executes.
A production job should normally run as an appropriately governed service principal rather than the human developer who created it.
Conceptually:
GITHUB ACTIONS
|
v
CI SERVICE PRINCIPAL
|
v
DEPLOY BUNDLE
|
v
PRODUCTION JOB
|
v
RUNTIME SERVICE PRINCIPAL
|
v
UNITY CATALOG DATA
The deployment principal needs deployment privileges.
The runtime principal needs only the data and execution permissions required by the workload.
They may be different identities.
20. Resource definitions
A bundle can define resources such as jobs and pipelines.
Example job resource structure:
resources:
jobs:
orders_pipeline_job:
name: orders-pipeline
tasks:
- task_key: process_orders
notebook_task:
notebook_path: ../notebooks/process_orders.py
The resource key orders_pipeline_job is a programmatic identifier used by bundle commands.
The deployed display name can be environment-specific.
A larger project should split resource definitions into separate files under resources/ and include them from databricks.yml.
21. Variables and configuration
Do not hard-code environment-specific values throughout the source code.
Examples of values that should be configurable:
- catalog;
- schema;
- warehouse;
- environment;
- source path;
- target path;
- notification destination;
- compute policy;
- runtime service principal.
Use bundle variables and target overrides.
Conceptually:
SAME SOURCE CODE
|
+── DEV VALUES
|
+── STAGING VALUES
|
+── PROD VALUES
The business transformation should not need a separate code branch for every environment.
Keep secrets outside bundle YAML and retrieve them through supported secret integrations.
22. Deployment state and reconciliation
Bundles track the resources they manage.
A subsequent deployment compares the bundle configuration with the tracked deployed resources.
Conceptually:
DECLARED RESOURCES
|
v
COMPARE WITH DEPLOYMENT STATE
|
+── CREATE MISSING
|
+── UPDATE EXISTING
|
+── REMOVE MANAGED RESOURCES
NO LONGER DECLARED
This is powerful — and potentially destructive.
Critical warning
If a resource was previously deployed and then removed from the bundle configuration, the next deployment can remove that resource from the target workspace.
Therefore:
- review resource deletion in pull requests;
- understand deployment state;
- separate shared resources from application-owned resources;
- avoid casually deleting YAML sections;
- test destructive changes in development first.
A green bundle validate result does not mean the deployment has no destructive impact.
23. Bundle validate
This is the core pre-deployment validation command:
databricks bundle validate -t dev
It checks bundle configuration and resource definitions against the available schemas.
The command also reports the bundle identity and target workspace information.
Use it to catch:
- YAML/configuration errors;
- invalid resource properties;
- unresolved or incorrect target configuration;
- deployment identity mistakes.
But validation does not prove that:
- the source data is correct;
- the query logic is correct;
- the job will complete;
- the production SLA will be met;
- downstream consumers are compatible.
Validation is one gate in a larger testing strategy.
Official command reference: https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands
24. Bundle deploy
Deploy the target:
databricks bundle deploy -t dev
Then inspect the deployed resources:
databricks bundle summary -t dev
Use the summary to verify:
- bundle identity;
- target;
- deployed resource inventory;
- deep links to Databricks resources.
Deployment is not a substitute for testing.
A successfully deployed job can still fail when it runs.
25. Bundle run
Run a resource by its bundle key:
databricks bundle run orders_pipeline_job -t dev
The resource key is the key under resources.jobs, not necessarily the user-facing job display name.
A pipeline resource can also be run using the supported pipeline task and CLI workflow.
For jobs, use the run command to validate the deployed resource and then examine task results in the workspace.
Production principle
Do not equate:
DEPLOY SUCCESS
with:
DATA PRODUCT SUCCESS
A release should be tested through actual execution and business validation.
26. Bundle summary and deployment history
Use:
databricks bundle summary -t prod
This gives the identity and inventory of the deployed bundle.
Deployment history and state-management commands also support operational troubleshooting.
Keep a release record connecting:
- Git commit SHA;
- bundle name;
- bundle target;
- deployment identity;
- deployment timestamp;
- deployed resources;
- test results;
- approval record.
This enables an operator to answer:
What exact change is running in production, and how did it get there?
27. A complete testing strategy
Enterprise testing should happen at multiple levels.
Level 1 — Static quality
- formatting;
- linting;
- type checks where appropriate;
- SQL style and syntax checks;
- dependency checks;
- secret scanning.
Level 2 — Unit tests
Test business logic locally where possible.
For Python:
pytest tests/unit
Level 3 — Bundle validation
databricks bundle validate -t dev
Level 4 — Development integration tests
Deploy to a development workspace and run the affected resources.
Level 5 — Staging tests
Test the actual deployed configuration in a production-like workspace.
Level 6 — Data regression
Validate:
- schema;
- row counts;
- uniqueness;
- reconciliation totals;
- quality expectations;
- critical business metrics.
Level 7 — Production smoke tests
After deployment, verify resource health and critical output readiness.
No single test type proves everything.
28. What should block a release?
A release should stop when a critical quality gate fails.
Examples:
UNIT TEST FAIL
|
v
BLOCK MERGE
BUNDLE VALIDATION FAIL
|
v
BLOCK DEPLOYMENT
STAGING RECONCILIATION FAIL
|
v
BLOCK PRODUCTION APPROVAL
PRODUCTION SMOKE TEST FAIL
|
v
INCIDENT / RECOVERY PROCESS
Avoid treating every warning as a production blocker. Classify rules by severity and business impact.
But do not allow a critical correctness failure to be overridden merely to meet a release date.
29. Git branching strategy
Databricks recommends a trunk-based workflow for many teams.
A practical pattern:
main
|
+── feature branch
|
v
pull request
|
v
review + tests
|
v
merge to main
|
v
staging
|
v
production
Keep feature branches short-lived.
Protect main with:
- required reviews;
- status checks;
- code owners;
- restricted direct pushes;
- required successful CI;
- security scanning.
The main branch should represent a deployable state.
This reduces long-lived branch drift and merge complexity.
30. Environment promotion
A robust release should promote the same reviewed commit.
COMMIT abc123
|
v
DEV
|
v
STAGING
|
v
PROD
Do not:
- rebuild different business logic for production;
- manually edit production notebooks;
- change the source commit after staging tests;
- silently substitute a different package version.
Environment-specific settings may change, but the source release should remain traceable.
This is the foundation of reproducibility.
31. GitHub Actions architecture
For a team using GitHub, the workflow can be:
PULL REQUEST
|
v
UNIT + STATIC TESTS
|
v
BUNDLE VALIDATION
|
v
MERGE TO MAIN
|
v
DEPLOY TO DEV
|
v
INTEGRATION TESTS
|
v
STAGING
|
v
APPROVAL
|
v
PRODUCTION
|
v
SMOKE TEST + MONITOR
Databricks provides a GitHub Actions integration pattern and recommends workload identity federation for CI/CD authentication.
Official references:
- https://docs.databricks.com/aws/en/dev-tools/ci-cd/github
- https://docs.databricks.com/aws/en/dev-tools/auth/provider-github
32. GitHub Actions authentication
The CI runner needs an identity that can authenticate to the Databricks workspace.
A basic but weaker approach is storing a long-lived token in a repository secret.
A stronger production pattern is:
Workload identity federation using GitHub OIDC.
The flow is:
GITHUB ACTIONS RUNNER
|
v
GITHUB OIDC TOKEN
|
v
DATABRICKS FEDERATION POLICY
|
v
DATABRICKS SERVICE PRINCIPAL
|
v
SHORT-LIVED OAUTH ACCESS
Benefits:
- no long-lived Databricks PAT stored in GitHub;
- no manually rotated client secret for this flow;
- identity can be restricted to a specific repository and environment;
- access is granted to a service principal rather than a human user.
The federation policy must match the GitHub OIDC claims used by the workflow.
33. GitHub OIDC configuration
A GitHub Actions job can use:
permissions:
contents: read
id-token: write
The deployment job can set:
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: REPLACE_WITH_GITHUB_ENVIRONMENT_VARIABLE
DATABRICKS_CLIENT_ID: REPLACE_WITH_SERVICE_PRINCIPAL_APP_ID
The DATABRICKS_CLIENT_ID here is the Databricks service principal application ID, not a client secret.
Use GitHub environment variables for the workspace host and application ID.
Configure the federation policy on the Databricks service principal so that only the intended repository and GitHub environment can authenticate.
Databricks recommends matching the environment claim for stronger deployment scoping.
Do not copy a sample federation subject without matching it to your actual GitHub organization, repository and environment names.
34. A GitHub Actions workflow example
The following is a reference starting point. Replace the project-specific test commands and resource keys. Configure the Databricks service-principal federation policy and GitHub environments before enabling production deployment.
name: Databricks CI/CD
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
contents: read
id-token: write
jobs:
quality:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install project dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txt
- name: Unit tests
run: pytest tests/unit
- name: Install Databricks CLI
uses: databricks/setup-cli@main
- name: Validate bundle
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: REPLACE_WITH_GITHUB_ENVIRONMENT_VARIABLE
DATABRICKS_CLIENT_ID: REPLACE_WITH_SERVICE_PRINCIPAL_APP_ID
run: databricks bundle validate -t dev
deploy-dev:
if: github.event_name == 'push'
needs: quality
runs-on: ubuntu-latest
environment: dev
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Databricks CLI
uses: databricks/setup-cli@main
- name: Deploy development target
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: REPLACE_WITH_GITHUB_ENVIRONMENT_VARIABLE
DATABRICKS_CLIENT_ID: REPLACE_WITH_SERVICE_PRINCIPAL_APP_ID
run: |
databricks bundle validate -t dev
databricks bundle deploy -t dev
- name: Run development workflow
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: REPLACE_WITH_GITHUB_ENVIRONMENT_VARIABLE
DATABRICKS_CLIENT_ID: REPLACE_WITH_SERVICE_PRINCIPAL_APP_ID
run: databricks bundle run orders_pipeline_job -t dev
Production hardening
For a real enterprise deployment:
- pin GitHub Actions to approved versions or immutable commit SHAs;
- pin and regularly review the Databricks CLI version;
- use protected GitHub environments;
- configure required production reviewers;
- use separate service principals and federation policies where appropriate;
- set concurrency controls;
- keep production deployment out of pull-request workflows;
- add staging integration tests;
- add a production smoke test;
- preserve deployment logs and test evidence.
This example shows the essential flow, not every organization's security policy.
35. Add staging and production approval gates
Create GitHub environments such as:
dev
staging
prod
Use repository settings to configure:
- environment-specific variables;
- deployment restrictions;
- required reviewers;
- approval rules;
- allowed branches.
The production job can declare:
environment: prod
A required reviewer can then approve the deployment before the production job begins.
The Databricks federation policy should be configured for the same GitHub environment identity.
The approval gate belongs in the CI platform; do not confuse a protected GitHub environment with Databricks resource permissions.
36. Build artifacts and Python wheels
Not every project is just a notebook.
A mature project may package reusable Python code as a wheel.
Conceptually:
src/
retail_transform/
__init__.py
orders.py
customers.py
Build:
python -m build
Test the package:
pytest tests/unit
Then publish the versioned artifact to an approved artifact store or Unity Catalog volume.
Tie the artifact version to:
- Git commit SHA;
- semantic version;
- release identifier.
Do not commit generated binary artifacts into Git as a replacement for a reproducible build process.
Databricks CI/CD guidance describes building artifacts, uploading them and then deploying the bundle with the artifact reference.
37. Artifact traceability
Imagine a production job uses:
retail_transform-1.4.2.whl
How do you know which source code produced it?
Use versioning:
Git commit
|
v
Build artifact
|
v
Artifact version / checksum
|
v
Bundle deployment
|
v
Production run
A production release record should connect the deployed job to the exact artifact and source commit.
Never rely on an unversioned latest-wheel convention for a critical production release.
38. Declarative Automation Bundles vs Terraform
Both are Infrastructure-as-Code tools, but their common responsibilities differ.
| Concern | Declarative Automation Bundles | Terraform |
|---|---|---|
| Databricks jobs | Strong fit | Supported |
| Lakeflow pipelines | Strong fit | Supported |
| Project source files | Integrated | Usually managed separately |
| Workspace project deployment | Strong fit | Not its primary project model |
| Account/workspace infrastructure | Limited relative to full IaC | Strong fit |
| Cloud networking | Not the primary tool | Strong fit |
| Reusable project templates | Native bundle templates | Terraform modules |
| Data project lifecycle | Strong fit | Infrastructure focus |
A common enterprise split:
TERRAFORM
|
+── workspace
+── account configuration
+── networking
+── cloud infrastructure
+── identity foundations
|
v
DECLARATIVE AUTOMATION BUNDLES
|
+── jobs
+── pipelines
+── project files
+── dashboards
+── supported resources
Use Terraform for the infrastructure layer when that is the organization's established model.
Use Bundles for the application/project deployment lifecycle.
Avoid managing the same resource in both Terraform and Bundles unless ownership and state boundaries are deliberately designed.
39. Declarative deployment is not a database migration
A bundle deploy updates supported workspace resources.
It does not automatically make every data change safe.
For example:
- changing a job definition is a resource deployment;
- changing a pipeline definition is a resource deployment;
- changing a table schema may have data compatibility implications;
- deleting a table or resetting a pipeline can have data-loss consequences;
- changing a transformation can change business results.
Separate release management from data migration management.
A production data release may need:
- backward-compatible schema evolution;
- expand-and-contract migration;
- dual-read or dual-write transition;
- backfill;
- reconciliation;
- downstream consumer coordination.
The bundle is one part of the release, not a guarantee that the data change is safe.
40. Existing workspace resources: generate and bind
Many organizations already have manually created jobs and pipelines.
You may want to bring them under bundle management.
The CLI supports resource generation from supported existing workspace resources.
Conceptually:
EXISTING WORKSPACE RESOURCE
|
v
databricks bundle generate
|
v
RESOURCE YAML + SOURCE FILES
|
v
REVIEW / CLEAN UP CONFIG
|
v
BIND TO EXISTING RESOURCE
|
v
CONTROLLED DEPLOYMENT
Important warning
Generating bundle configuration without binding the existing resource can cause a later deployment to create a new resource instead of managing the existing one.
Use the supported bind option or the bundle deployment bind command when the goal is to manage the existing resource.
Verify the binding and resource identity before the first production deployment.
41. Drift management
Configuration drift occurs when the workspace changes outside source control.
Example:
GIT JOB CONFIG
|
| intended state
v
PRODUCTION JOB
Someone edits the production job manually
|
v
WORKSPACE CONFIG ≠ GIT CONFIG
This creates two competing sources of truth.
A mature operating model should:
- make Git the authoritative source for managed resources;
- restrict manual production changes;
- review workspace changes;
- reconcile drift through a controlled deployment;
- log emergency modifications;
- back-port emergency fixes into Git immediately.
Do not silently overwrite a workspace change you do not understand.
Investigate it first.
42. Bundle deployment identity and state safety
Bundle deployment state is used to track which workspace resources belong to the deployment.
Remember:
- same bundle name, target and workspace identify the deployment boundary;
- CI runners should use a stable identity and consistent target root path;
- separate bundles should not accidentally share a root path;
- changing the identity can make the CLI treat the deployment differently;
- resource names alone are not sufficient to determine ownership.
Before changing a bundle name, target name or root path in production, plan the state transition.
Do not casually rename the bundle after it has deployed resources.
43. Destructive operations: deploy vs destroy
These commands have very different implications.
Deploy
databricks bundle deploy -t dev
Creates or updates resources managed by that bundle deployment.
A previously managed resource removed from the configuration can also be removed from the workspace during reconciliation.
Destroy
databricks bundle destroy -t dev
Permanently deletes resources and artifacts previously deployed by that bundle.
This is not a rollback command.
It is a destructive cleanup operation.
Production rule
Never use:
databricks bundle destroy --auto-approve
as a generic CI cleanup step against production.
Review deployment identity, target and resource inventory before any destroy operation.
Official command reference: https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands
44. Rollback architecture
A code rollback and a data rollback are different.
Code/configuration rollback
Revert to a previously validated commit and redeploy the same bundle identity and target.
BAD RELEASE
|
v
REVERT TO KNOWN-GOOD COMMIT
|
v
VALIDATE
|
v
DEPLOY PREVIOUS VERSION
Data rollback
If the release has already changed data, a code redeployment may not reverse the data.
You may need:
- Delta time travel or restore, where appropriate;
- reprocessing from Bronze;
- backfill;
- compensating transformation;
- downstream reconciliation.
Architect principle
Rollback must cover both the deployed code and the data state affected by that code.
Design this before production deployment.
45. Release strategies
Choose a release strategy based on the workload.
Standard release
- deploy during an approved window;
- run smoke tests;
- monitor;
- complete the release.
Blue/green style
Where the architecture supports separate resources or serving endpoints:
- deploy a new version alongside the old one;
- test it;
- switch consumers;
- retire the old version after validation.
Canary style
Where the resource and traffic model supports it:
- expose a limited consumer group;
- observe results;
- increase adoption;
- roll back if quality degrades.
Not all Databricks jobs or pipelines support a true traffic-based canary model. Do not claim a deployment strategy is available unless the resource and serving architecture support it.
46. Permissions and least privilege
CI/CD requires carefully scoped access.
Separate:
CI deploy principal
Can deploy the intended resources to the intended workspace.
Runtime principal
Can read and write only the data required by the deployed job or pipeline.
Human developers
Can author and test in their development environment.
Production operators
Can monitor and perform approved recovery actions.
Use Unity Catalog and workspace permissions to keep access explicit.
Do not give a deployment principal workspace-admin privileges simply because it is convenient.
The permissions required depend on the resource types and deployment operations in the bundle.
47. Custom bundle templates
Large organizations often need consistent defaults.
A custom template can standardize:
- directory structure;
- resource YAML;
- Python package setup;
- test framework;
- CI workflow;
- environment targets;
- service identity conventions;
- tags and permissions;
- naming standards;
- documentation.
Conceptually:
ORGANIZATION TEMPLATE
|
+── BUNDLE STRUCTURE
+── SECURITY DEFAULTS
+── TEST FRAMEWORK
+── TARGETS
+── CI/CD
|
v
NEW DATA PRODUCT
This reduces variation across teams.
A template should encode current supported patterns and should itself be maintained as code.
48. Monorepo vs multiple repositories
A monorepo can simplify:
- shared code;
- shared CI;
- common templates;
- cross-project changes;
- consistent developer tooling.
Multiple repositories can be appropriate when:
- deployment ownership differs;
- data domains are isolated;
- confidentiality requirements demand separation;
- independent release cycles are important;
- teams have different access boundaries.
The number of repositories should follow ownership and security boundaries, not fashion.
Databricks developer guidance generally recommends a single repository for related source and configuration, with exceptions for regulated confidentiality requirements.
49. Production operating model
Every production release should have an identifiable owner.
| Area | Responsibility |
|---|---|
| Developer | Code and unit tests |
| Reviewer | Design and code review |
| Data product owner | Business correctness and SLA |
| Platform team | Templates, identity and deployment standards |
| CI service principal | Automated deployment |
| Runtime service principal | Scheduled execution |
| Release approver | Production authorization |
| Operations / on-call | Monitoring and incident response |
| Governance | Access and audit policy |
This avoids the common situation where a deployment succeeds but nobody owns the data product.
50. Common anti-patterns
1. Manual production deployment
No repeatable deployment history.
2. User PAT in GitHub secrets
Long-lived credentials create rotation and exposure risk. Prefer workload identity federation.
3. Different logic per environment
The production behavior is no longer the behavior that staging tested.
4. Bundle validation treated as full testing
A valid YAML file can still deploy incorrect business logic.
5. Shared root path
Independent deployments can interfere with one another.
6. Multiple owners for the same resource
Terraform, Bundles and manual workspace edits all fight over the same resource.
7. Removing a resource definition without understanding deletion behavior
A later deployment can remove the managed resource.
8. Using destroy as rollback
Destroy is permanent cleanup, not a release rollback.
9. Unversioned build artifact
You cannot prove which code produced the deployed binary.
10. No post-deployment smoke test
A successful deployment does not prove the workload actually runs or produces correct results.
11. No data rollback plan
Code is reverted, but corrupted or incompatible data remains.
12. Unprotected production environment
A pull request or unauthorized branch can reach production without the intended approval gate.
51. Hands-on lab — initialize and deploy a bundle
Objective
Create a source-controlled Databricks job and deploy it to a development workspace.
Step 1 — initialize
databricks bundle init
Select a suitable Python job template.
Step 2 — inspect the project
Identify:
- databricks.yml;
- resource definitions;
- notebooks or source files;
- tests;
- environment settings.
Step 3 — validate
databricks bundle validate -t dev
Step 4 — deploy
databricks bundle deploy -t dev
Step 5 — inspect
databricks bundle summary -t dev
Step 6 — run
databricks bundle run <job-resource-key> -t dev
Step 7 — test
Inspect:
- job run;
- task execution;
- logs;
- output table;
- quality checks.
The goal is to understand the lifecycle, not simply to get a green command result.
52. Hands-on lab — add environment targets
Configure:
DEV
STAGING
PROD
For each target, define:
- workspace;
- catalog;
- resource settings;
- runtime identity;
- permissions;
- deployment mode.
Then run:
databricks bundle validate -t dev
databricks bundle validate -t staging
databricks bundle validate -t prod
Ensure each target resolves to the expected workspace and identity before deploying.
53. Hands-on lab — build GitHub Actions CI
Implement:
- Pull request trigger.
- Unit tests.
- Static checks.
- Databricks CLI installation.
- Bundle validation.
- Branch protection.
- Merge to main.
- Development deployment.
- Integration test.
- Staging deployment.
Then simulate:
- failed unit test;
- invalid bundle YAML;
- expired or mismatched federation policy;
- failed integration test.
Confirm that each failure blocks the next promotion stage.
54. Hands-on lab — production approval
Configure a protected GitHub environment for production.
Require:
- approved source branch;
- authorized reviewer;
- correct federation identity;
- production workspace variables;
- deployment service principal;
- runtime service principal.
Run a test release and verify that production deployment cannot begin before approval.
55. Hands-on lab — rollback and deployment-state safety
In a development workspace only:
- Deploy version A.
- Confirm the bundle summary.
- Change a job configuration and deploy version B.
- Remove a managed resource definition in a test branch.
- Observe the deployment plan and documented removal behavior.
- Restore the resource definition.
- Deploy the known-good commit.
- Verify the resource inventory.
Do not perform these destructive experiments in production.
56. Troubleshooting guide
| Symptom | Likely cause | Investigation |
|---|---|---|
| CLI cannot authenticate | OIDC policy mismatch, host or client ID issue | Check GitHub subject, environment and service principal |
| Bundle validation fails | Invalid YAML, resource schema or target | Run validate and inspect the exact resource path |
| Deployment targets wrong workspace | Target or host resolution mistake | Inspect target and bundle summary |
| Resources appear duplicated | Existing resource not bound or wrong bundle identity | Inspect resource IDs and deployment binding |
| Deployment unexpectedly deletes a resource | Resource removed from managed configuration | Review Git diff and deployment state |
| Two deployments interfere | Shared name, target or root path | Inspect deployment identity |
| Job deploys but cannot access data | Runtime principal permissions | Review run_as and Unity Catalog grants |
| Production differs from staging | Different commit, artifact or environment overrides | Compare release records |
| Wheel missing at runtime | Artifact build/upload/reference problem | Inspect build logs and artifact path |
| Rollback does not fix output data | Code rollback mistaken for data rollback | Run data recovery/reconciliation plan |
Never start by deleting bundle state or destroying resources. First establish the exact deployment identity and the resources it manages.
57. Enterprise release checklist
Source control
- Source code is versioned.
- Bundle configuration is versioned.
- Resource YAML is reviewed.
- Secrets and PII are excluded.
- Main branch is protected.
Bundle
- One root databricks.yml exists.
- Resource includes are correct.
- Targets are defined.
- Variables are environment-aware.
- Root path is unique and stable.
- Resource ownership is clear.
- Destructive reconciliation is reviewed.
Identity
- CI deploy identity is a service principal.
- Workload identity federation is configured.
- Federation claims are scoped.
- Runtime identity is explicit.
- Permissions follow least privilege.
- No long-lived secrets are committed.
Testing
- Unit tests run on pull requests.
- Static checks are automated.
- Bundle validation passes.
- Development integration test passes.
- Staging regression passes.
- Business reconciliation passes.
- Production smoke test exists.
Promotion
- Same commit is promoted.
- Artifact versions are traceable.
- Staging is production-like.
- Production approval is configured.
- Environment variables are correct.
- Concurrent deployments are controlled.
Operations
- Deployment logs are retained.
- Bundle summary is available.
- Job and pipeline monitoring is enabled.
- Rollback is documented.
- Data recovery is separate from code rollback.
- Resource deletion is reviewed.
- Ownership and on-call are assigned.
58. Enterprise architect exercise
Design CI/CD for a global Databricks platform with:
- 12 development teams;
- 40 Lakeflow pipelines;
- 80 Lakeflow Jobs;
- shared Python libraries;
- Unity Catalog governance;
- development, staging and production workspaces;
- production change approvals;
- strict audit requirements;
- multiple cloud accounts or subscriptions.
Your architecture must answer:
- Monorepo or multiple repositories?
- How many bundles and deployment lifecycles?
- How are common templates maintained?
- How are bundle root paths separated?
- Which identity deploys each target?
- Which identity runs each job?
- How is OIDC federation scoped?
- What blocks a pull request?
- What integration tests run in staging?
- How are artifacts versioned?
- How are resources bound if they already exist?
- How is manual configuration drift handled?
- How are destructive changes reviewed?
- How are production approvals enforced?
- How is rollback performed?
- How are affected Delta tables recovered?
- How are release records audited?
- How are costs allocated?
Then defend why your approach is safer than deploying notebooks manually from a user's workspace.
59. Interview questions
Fundamentals
- What is CI/CD?
- What is a Declarative Automation Bundle?
- How is a bundle different from a Git repository?
- What is databricks.yml?
- What is a deployment target?
- What does bundle validate verify?
Development and deployment
- What is the lifecycle of a bundle?
- How do you initialize a new bundle?
- How do you deploy a job?
- What is bundle deployment state?
- Why is workspace root path important?
- What happens when a managed resource is removed from YAML?
- How do you inspect a bundle deployment?
Security
- Why use a service principal for CI/CD?
- What is workload identity federation?
- What is the difference between deployment identity and runtime identity?
- How does GitHub OIDC authenticate to Databricks?
- How do you scope permissions for production deployments?
Testing and reliability
- What belongs in a unit test?
- What does bundle validation not prove?
- How do you test a Lakeflow pipeline in staging?
- How do you version a Python wheel?
- What is configuration drift?
- How do you manage destructive resource changes?
- How do you roll back a failed release?
Architect
- When would you use Terraform instead of Bundles?
- How would you manage 100 data products across multiple teams?
- How would you design environment promotion?
- How would you prevent two bundles from managing the same resource?
- How would you guarantee release traceability?
- How would you design CI/CD for regulated finance data?
- How would you separate code rollback from data rollback?
60. The enterprise architect mental model
Remember the six release questions:
1. WHAT EXACTLY IS THE RELEASE?
|
v
2. HAS IT BEEN REVIEWED AND TESTED?
|
v
3. WHICH IDENTITY DEPLOYS IT?
|
v
4. WHICH ENVIRONMENT RECEIVES IT?
|
v
5. HOW DO WE VERIFY IT WORKS?
|
v
6. HOW DO WE RECOVER IF IT DOES NOT?
A mature Databricks platform has:
- Git as the source of truth;
- Bundles as the project deployment definition;
- CI/CD as the automation and approval engine;
- service principals and federation as the identity model;
- environment targets as the promotion model;
- tests as evidence of correctness;
- monitoring as production assurance;
- rollback and data recovery as the safety net.
61. Final enterprise takeaway
Declarative Automation Bundles are not simply a way to replace manual notebook deployment with YAML.
They are the foundation for treating Databricks data and AI projects as software products.
The engineering maturity shift is:
MANUAL WORKSPACE EDITS
|
v
SOURCE-CONTROLLED PROJECT
|
v
AUTOMATED VALIDATION
|
v
REPEATABLE DEPLOYMENT
|
v
CONTROLLED PROMOTION
|
v
OBSERVED + RECOVERABLE PRODUCTION
Architect principle: Every production data product must be traceable to a reviewed source version, deployable through a controlled identity, validated in an isolated environment, promoted without changing its logic, and recoverable without confusing code rollback with data rollback.
That is enterprise Databricks CI/CD.
Official references
- Databricks — What are Declarative Automation Bundles?
- Databricks — CI/CD on Databricks
- Databricks — CI/CD workflows
- Databricks — Develop Declarative Automation Bundles
- Databricks — Bundle configuration
- Databricks — Deployment modes
- Databricks — CLI bundle commands
- Databricks — Bundle resources
- Databricks — Developer best practices
- Databricks — Workload identity federation for GitHub Actions
- Databricks — Declarative Automation Bundles FAQs
- Databricks — Bundle tutorials