20. MLOps & Model Serving
Learning objective: turn ML engineering into a repeatable production discipline: environments, CI/CD, training, evaluation, governed promotion, serving, monitoring, retraining, rollback, security, cost, and enterprise operating models.
Phase 19 taught reproducible ML with MLflow and governed models with Unity Catalog. Phase 20 answers the next question:
How do we operate ML as a reliable production system?
Databricks currently describes MLOps as the combination of DevOps, DataOps, and ModelOps. Its guidance emphasizes development, staging, production, code promotion, automated workflows, monitoring, and retraining.
20.1 The production ML mental model
A beginner thinks:
Train → deploy → predict
An architect thinks:
Business objective
↓
Data + feature contract
↓
Version-controlled code
↓
Automated tests
↓
Training
↓
Evaluation gate
↓
Governed model
↓
Promotion policy
↓
Serving / batch inference
↓
Observability
↓
Drift + business monitoring
↓
Retraining decision
↓
Controlled replacement
ML systems change in several dimensions at once:
- source data
- feature logic
- training code
- dependencies
- model versions
- serving configuration
- business behavior
MLOps makes those changes repeatable, observable, auditable, and reversible.
20.2 Development, staging, production
A strong baseline is:
DEV → STAGING → PRODUCTION
Development
Used for:
- experimentation
- feature development
- model development
- debugging
- rapid iteration
Staging
Used for:
- integration tests
- deployment validation
- realistic workflow tests
- release qualification
Production
Used for:
- business execution
- controlled training
- serving
- monitoring
- governed model lifecycle
Databricks recommends distinct development, staging, and production concerns with tighter production controls.
Environment separation is not merely three folders. Consider:
- identities
- permissions
- compute
- catalogs
- network boundaries
- secrets
- deployment targets
- operational ownership
Architect rule:
Separate environments where blast-radius reduction matters.
20.3 Production ownership
A practical operating model:
| Role | Responsibility |
|---|---|
| Data scientist | Experimentation, features, model behavior |
| Data engineer | Data pipelines and quality |
| ML engineer | Production ML pipelines |
| MLOps engineer | Deployment, automation, observability |
| Platform engineer | Workspace, identity, networking |
| Product owner | Business outcomes |
| Security/governance | Access, audit, compliance |
Databricks' MLOps guidance places production ownership primarily with ML engineers while giving data scientists visibility into production evidence.
Key principle:
Production visibility should be broader than production write access.
20.4 Git is the source of truth for change
Version-control:
- Python
- SQL
- notebooks where required
- tests
- configuration
- bundle definitions
- dependencies
- deployment policies
- documentation
Avoid manually editing production notebooks as the authoritative source.
Git
↓
Review
↓
Test
↓
Package
↓
Deploy
Git answers:
What changed?
MLflow answers:
What happened when it ran?
Unity Catalog answers:
What governed model asset exists?
20.5 Deploy code, not models
Databricks currently recommends promoting training and pipeline code between environments in most situations. Production then trains the approved code on production data.
Model promotion:
DEV model
↓
copy artifact
↓
PROD
Code promotion:
DEV code
↓
STAGING code + tests
↓
PROD approved code
↓
production data
↓
production model
Benefits:
- production model uses production data
- retraining is safer
- training code is reviewed
- supporting pipelines follow the same release path
- security boundaries are easier to reason about
20.6 When model promotion is valid
Model-artifact promotion can still make sense when:
- training is extremely expensive
- training is difficult to reproduce
- training occurs outside Databricks
- a controlled staging environment produces the final artifact
- a legacy architecture must be supported
Databricks documents a hybrid approach in which code is promoted to staging, the model is trained there, and the resulting artifact is promoted to production.
| Situation | Pattern |
|---|---|
| Frequent retraining | Deploy code |
| Production data isolation | Deploy code |
| Reproducible training | Deploy code |
| Extremely expensive training | Consider model promotion |
| External training | Consider model promotion |
| Transitional legacy | Hybrid |
20.7 CI/CD for ML
CI validates changes before release.
Typical CI checks:
- linting
- unit tests
- data-contract tests
- schema tests
- package validation
- security checks
- bundle validation
CD moves approved changes through controlled environments:
Pull request
↓
CI
↓
STAGING
↓
Integration tests
↓
Policy / approval
↓
PRODUCTION
The objective is not automation for its own sake.
Every production change should have evidence.
20.8 Declarative Automation Bundles
Databricks currently calls its source-controlled deployment framework Declarative Automation Bundles, formerly Databricks Asset Bundles.
Bundles describe project resources and deployment configuration as source files and can be validated, deployed, and run programmatically.
A bundle can represent:
- notebooks
- Python code
- jobs
- pipelines
- model-related resources
- serving endpoints
- environment configuration
- tests
Conceptually:
Git repository
↓
Declarative Automation Bundle
↓
validate
↓
deploy
↓
Databricks workspace
20.9 Bundles versus Terraform
Do not force one IaC tool onto every layer.
| Concern | Typical choice |
|---|---|
| ML workflows | Declarative Automation Bundles |
| Jobs and pipelines | Declarative Automation Bundles |
| Model serving resources | Declarative Automation Bundles |
| Workspace infrastructure | Terraform |
| Network infrastructure | Cloud IaC / Terraform |
| Enterprise cloud foundation | Cloud IaC |
Databricks currently recommends Declarative Automation Bundles for data and AI resources and Terraform for broader infrastructure such as workspaces and networks.
20.10 Current bundle terminology and engine
Older documentation may say:
Databricks Asset Bundles
Current documentation says:
Declarative Automation Bundles
The rename is non-breaking.
Current Databricks CLI versions support a direct deployment engine. New bundles created with CLI 1.3.0 and later use the direct engine by default; older configurations may reference Terraform-based deployment.
Before production deployment verify:
- CLI version
- bundle engine
- target workspace
- authentication
- deployment permissions
- environment variables
20.11 Repository architecture
A useful project structure:
supplier-risk/
├── databricks.yml
├── resources/
│ ├── training_job.yml
│ ├── inference_job.yml
│ └── serving.yml
├── src/
│ ├── features/
│ ├── training/
│ ├── evaluation/
│ └── inference/
├── tests/
│ ├── unit/
│ ├── integration/
│ └── data/
├── configs/
│ ├── dev.yml
│ ├── staging.yml
│ └── prod.yml
└── README.md
The directory names are not sacred.
The architectural principle is:
Code, infrastructure, tests, and environment intent should be versionable.
20.12 Lakeflow Jobs
Lakeflow Jobs orchestrates multi-step ML workflows.
Training:
Validate
↓
Prepare data
↓
Build features
↓
Train
↓
Evaluate
↓
Register
↓
Promotion gate
↓
Deploy
↓
Smoke test
Inference:
Read features
↓
Load approved model
↓
Score
↓
Write predictions
↓
Quality check
↓
Publish
Databricks currently positions Lakeflow Jobs as the workflow layer for automated ML pipelines.
20.13 Training pipeline contract
A production training pipeline should explicitly define:
- source data
- feature version
- training window
- validation window
- algorithm
- hyperparameters
- random seed
- model signature
- evaluation criteria
- registration policy
Avoid hidden dependencies such as:
- notebook state
- manually changed SQL
- local packages
- untracked feature logic
- undocumented environment variables
20.14 Idempotent training and inference
If a job retries, its output must remain understandable.
Define:
- run identifier
- input snapshot
- model version
- output location
- duplicate policy
- overwrite policy
For batch inference:
business_date + entity_id
↓
idempotency key
This prevents retry storms from creating duplicate predictions.
20.15 Evaluation gates
Training success does not equal production readiness.
Example:
Data quality PASS
AND
Precision ≥ target
AND
Recall ≥ target
AND
Business KPI ≥ target
AND
No critical segment regression
AND
Latency ≤ SLO
AND
Security checks PASS
A gate is a policy encoded into automation.
20.16 Offline evaluation
Compare candidate and current models on the same held-out data:
Champion ──┐
├── same evaluation data → comparison
Challenger ─┘
Evaluate:
- precision
- recall
- F1
- RMSE / MAE
- calibration
- segment performance
- business metrics
- error distribution
Do not select a model because it was the most recent run.
20.17 Online evaluation
Offline tests do not always predict production behavior.
Possible patterns:
- A/B testing
- gradual rollout
- canary
- shadow evaluation
Databricks' MLOps guidance explicitly discusses offline and online champion/challenger comparisons.
20.18 Champion and challenger
CHAMPION
│
│ baseline
↓
comparison
↑
│
CHALLENGER
│
evaluation
/ \
reject promote
│ │
↓ ↓
keep old new champion
The champion is the current production baseline.
The challenger is the candidate.
Architect rule:
Never replace a known-good baseline without evidence.
20.19 Model serving mental model
Model Serving is the runtime interface.
Client
↓
Serving endpoint
↓
Approved model
↓
Prediction
↓
Response
The endpoint is not the model.
It is the controlled runtime interface through which consumers invoke the model.
Databricks currently supports managed scalable model serving for real-time inference.
20.20 Real-time inference
Use real-time serving when the decision requires immediate response.
Examples:
- fraud scoring
- personalization
- recommendation
- interactive risk scoring
Flow:
Application
↓
HTTPS request
↓
Serving endpoint
↓
Feature lookup / preprocessing
↓
Model
↓
Prediction
↓
Response
Define:
- latency SLO
- timeout
- concurrency
- payload limits
- authentication
- fallback
- monitoring
20.21 Batch inference
Use batch when immediate response is unnecessary.
Daily source data
↓
Inference job
↓
Prediction table
↓
Quality checks
↓
SAC / BI
For enterprise analytics, batch inference is often simpler and cheaper than real-time serving.
Decision rule:
Choose the lowest-complexity serving mode that satisfies the business latency requirement.
20.22 Streaming inference
Streaming sits between batch and synchronous APIs:
Event stream
↓
stream processing
↓
features
↓
model inference
↓
prediction stream
↓
downstream action
Use it when decisions must continuously follow events.
20.23 Serving API contract
Define:
- required fields
- data types
- null policy
- allowed ranges
- authentication
- timeout
- error schema
- compatibility policy
- model/deployment identity
Example request:
{
"supplier_id": "SUP-1001",
"annual_spend": 1250000,
"late_delivery_rate": 0.14
}
Example response:
{
"risk_score": 0.82,
"risk_class": "HIGH"
}
A real contract also defines validation and failure behavior.
20.24 Serving failure policy
| Failure | Possible response |
|---|---|
| Invalid payload | Reject with validation error |
| Timeout | Controlled retry / fallback |
| Dependency unavailable | Business-defined fallback |
| Model unavailable | Fail closed or fallback |
| Unknown schema | Reject |
| Rate limit | Backoff |
Do not invent fallback behavior during an incident.
20.25 Feature dependencies
A model endpoint can be healthy while its features are wrong.
Endpoint = UP
Model = UP
Feature table = STALE
Prediction = technically valid
Business decision = WRONG
Therefore monitor:
- feature freshness
- feature completeness
- feature distributions
- feature computation failures
20.26 Monitoring pyramid
Business outcome
/---------------\
Model performance
/-------------------\
Data quality + drift
/-----------------------\
Endpoint / infrastructure
Infrastructure
- availability
- latency
- throughput
- errors
Data
- freshness
- nulls
- schema
- distributions
Model
- prediction drift
- accuracy when labels arrive
- calibration
- segment behavior
Business
- revenue
- loss
- conversion
- operational workload
Databricks currently provides data profiling and monitoring capabilities for data quality, model performance, and prediction drift.
20.27 Data drift
Data drift changes the input distribution.
Example:
Training average spend = 1.2M
Production average spend = 2.8M
Possible causes:
- market change
- customer mix
- source-system change
- seasonality
- upstream defect
Schema stability does not guarantee distribution stability.
20.28 Concept drift
Concept drift changes the relationship between input and target.
Example:
Historical:
late delivery → high supplier risk
New reality:
logistics policy changed
late delivery is less predictive
The input schema may remain perfect while model usefulness falls.
20.29 Delayed labels
Many models receive ground truth later:
Prediction today
↓
business outcome
↓
label after 30 days
Separate:
- immediate proxy metrics
- delayed model metrics
- business outcome metrics
Do not call a model healthy only because its endpoint is available.
20.30 Retraining strategies
Scheduled
Weekly
↓
Train
↓
Evaluate
↓
Promote if passed
Triggered
Drift / quality signal
↓
Train
↓
Evaluate
↓
Promote if passed
Hybrid
Scheduled baseline
+
Triggered retraining
Databricks recommends beginning with scheduled periodic retraining and moving toward triggered retraining when appropriate.
20.31 Retraining does not mean automatic promotion
Unsafe:
Drift
↓
Train
↓
Deploy automatically
Safer:
Drift
↓
Train
↓
Evaluate
↓
Compare
↓
Policy gate
↓
Promote
Automatic training can coexist with controlled promotion.
20.32 Deployment strategies
Blue/green
BLUE = current
GREEN = candidate
Traffic → BLUE
Validate GREEN
Switch traffic
Good for:
- fast rollback
- strong isolation
Trade-off:
- additional infrastructure
Canary
Candidate traffic:
5% → 10% → 25% → 50% → 100%
Good for:
- reducing blast radius
Trade-off:
- more complex monitoring
Shadow
Request
├── Champion → business response
└── Challenger → evaluation only
Good for production comparison without changing decisions.
A/B
Different populations receive different models.
Good for controlled business experimentation.
20.33 Choosing a deployment strategy
| Situation | Strategy |
|---|---|
| Low-risk internal model | Direct replacement may suffice |
| Moderate risk | Canary |
| High-risk customer decision | Canary + strong rollback |
| Need production comparison | Shadow |
| Business experiment | A/B |
| Instant rollback requirement | Blue/green |
The rollback point should be defined before deployment.
20.34 Rollback
A rollback may target:
- previous model version
- previous serving configuration
- previous feature version
- previous code release
A rollback plan must answer:
- Who can trigger it?
- What evidence triggers it?
- How fast can it execute?
- What happens to in-flight requests?
- How is the incident recorded?
- How is accidental re-promotion prevented?
20.35 Rollback versus roll-forward
Rollback is not always correct.
If the previous model uses the same broken feature, rollback can reproduce the problem.
Use:
Incident
↓
identify failure layer
↓
data?
feature?
code?
model?
serving?
business rule?
↓
rollback OR fix-forward
Architect rule:
Rollback is a mechanism, not an incident strategy.
20.36 Human approval
Human approval may be required for:
- financial decisions
- regulated use cases
- customer-impacting models
- high-impact classification
- material business changes
Workflow:
Automated evaluation
↓
Policy checks
↓
Human approval
↓
Deployment
Current Databricks documentation also describes MLflow 3 deployment jobs that can automate evaluation, approval, and deployment around Unity Catalog models; this capability is currently documented as Public Preview.
Preview features should be assessed for enterprise support and risk before becoming mandatory dependencies.
20.37 Security and identities
Production pipelines should not depend on personal credentials.
Prefer:
- service principals
- workload identity federation where supported
- cloud-native workload identities
- scoped permissions
- secret-management systems
Databricks CI/CD guidance recommends service principals and supports workload identity federation to reduce stored secrets.
20.38 Least privilege
A training identity may need:
- read training data
- write experiment evidence
- register candidate models
A deployment identity may need:
- deploy approved resources
- update serving configuration
A monitoring identity may need:
- read telemetry
- write monitoring results
Do not grant every identity unrestricted lakehouse access.
20.39 Secrets versus configuration
Version-control normal configuration:
environment = prod
catalog = ml_prod
schedule = daily
threshold = 0.85
Never version-control secrets such as:
password
private token
API credential
20.40 Testing pyramid
E2E
/------\
Integration
/------------\
Data contract
/----------------\
Unit tests
Unit
Feature functions and transformations.
Data
Schema, nulls, ranges, uniqueness, freshness.
Integration
Data access, model loading, orchestration, endpoint contracts.
End-to-end
Source → features → training → registration → deployment → inference → monitoring.
20.41 Model quality tests
Test:
- minimum metric thresholds
- maximum error
- segment-level thresholds
- prediction ranges
- model signature
- distribution expectations
Example:
if recall < target:
reject
if critical_segment_recall < minimum:
reject
if prediction_rate is abnormal:
investigate
Not every metric must be a hard gate. Some are better as warnings.
20.42 Data quality gates
A model can be correct while input data is wrong.
Example:
freshness < 2 hours
null_rate < 1%
schema = expected
row_count = expected range
distribution = acceptable
Critical failures should stop publication rather than publish bad predictions.
20.43 Release evidence
A production release should allow engineers to answer:
Which code?
Which data?
Which features?
Which model?
Which configuration?
Which endpoint?
Which deployment?
Version independently:
- source code
- feature definitions
- model versions
- deployment configuration
- infrastructure
- data contracts
This is the foundation of production lineage.
20.44 Incident response
A model incident is not automatically a model problem.
Use:
Prediction issue
↓
Endpoint healthy?
├── NO → serving investigation
└── YES
↓
Input data valid?
├── NO → data incident
└── YES
↓
Features correct?
├── NO → feature incident
└── YES
↓
Model performance degraded?
├── YES → model investigation
└── NO → business process investigation
This prevents blind retraining.
20.45 Incident severity
Example policy:
| Severity | Example | Response |
|---|---|---|
| Sev 1 | Critical model unavailable | Immediate fallback/rollback |
| Sev 2 | Material quality degradation | Incident + mitigation |
| Sev 3 | Drift warning | Investigation |
| Sev 4 | Non-critical anomaly | Backlog |
Align levels with enterprise SRE policy.
20.46 Cost engineering
ML cost includes:
- training compute
- hyperparameter search
- inference
- serving
- storage
- monitoring
- data processing
- retries
- idle resources
Total ML cost
=
training
+
inference
+
serving
+
data
+
monitoring
+
platform overhead
Optimize the lifecycle, not just the training cluster.
20.47 Cost-aware retraining
If retraining cost is X and expected business benefit is Y:
Retrain when expected benefit
>
training + validation + deployment cost
Drift thresholds should not create uncontrolled retraining.
Use:
- persistence windows
- cooldowns
- business impact
- evaluation gates
20.48 MLOps observability
ML SYSTEM
│
┌──────────┼──────────┐
↓ ↓ ↓
Platform Data Model
metrics metrics metrics
│ │ │
└──────────┼──────────┘
↓
Observability
↓
Alert / Ticket
↓
Runbook / Action
Monitoring without an action path is only telemetry.
20.49 Actionable alerts
Weak:
Model metric changed.
Strong:
Supplier-risk recall fell below the approved threshold for two evaluation windows; investigate drift and compare the champion with the previous release.
An alert should contain:
- metric
- threshold
- duration
- model
- environment
- owner
- runbook
20.50 Automated retraining architecture
Monitoring
↓
Drift / quality threshold
↓
Trigger
↓
Lakeflow Job
↓
Training
↓
Evaluation
↓
Promotion gate
/ \
FAIL PASS
│ │
Alert Register
↓
Deploy
↓
Smoke test
Databricks documents scheduled and triggered retraining patterns using jobs and monitoring signals.
20.51 Enterprise ML platform
Source systems
↓
Lakehouse / feature data
↓
Git + MLflow
↓
CI/CD + Bundles
↓
DEV → STAGING → PROD
↓
Unity Catalog
↓
Evaluation + promotion
↓
┌──────────────┬──────────────┐
│ Real-time │ Batch │
│ serving │ inference │
└──────────────┴──────────────┘
↓
Business consumers
↓
Monitoring
↓
Retraining
This is a platform, not a notebook collection.
20.52 Multi-team architecture
Suppose an enterprise has:
- fraud
- procurement
- finance
- marketing
- supply chain
Create one governed platform:
Enterprise ML Platform
│
┌──────┼──────┬──────┐
↓ ↓ ↓ ↓
Fraud Procure Finance Marketing
Standardize:
- project templates
- bundle patterns
- CI/CD
- governance
- monitoring
- release controls
Allow domains to customize:
- features
- algorithms
- business metrics
- retraining cadence
20.53 Platform versus product ownership
Platform team
Owns:
- shared tooling
- workspace patterns
- CI/CD framework
- governance
- observability
- templates
- security controls
ML product team
Owns:
- model behavior
- features
- business metrics
- training logic
- model-specific runbooks
This is a platform operating model.
20.54 SAP analytics integration
For an SAP analytics architect:
SAP S/4HANA
↓
ODP / SLT / APIs / approved integration
↓
Databricks ingestion
↓
Delta Lake
↓
Conformed business data
↓
ML features
↓
Model
↓
Prediction tables
↓
SAC / BI / applications
Do not treat SAP BW, Datasphere, and Databricks as interchangeable.
Ask:
- Which system owns the business definition?
- Where should transformation occur?
- Where should ML training occur?
- Where should analytical consumption occur?
- What latency is required?
- Which data product is authoritative?
20.55 SAP procurement example
Business problem:
Predict supplier delivery risk.
S/4HANA purchasing
↓
PO + supplier + delivery data
↓
Databricks lakehouse
↓
Feature engineering
↓
Training dataset
↓
MLflow
↓
Unity Catalog model
↓
Production scoring
↓
Supplier risk table
↓
SAC dashboard
The model adds predictive intelligence. It does not replace transactional truth.
20.56 SAC consumption architecture
For analytical consumption:
Databricks
↓
Curated prediction data
↓
Governed analytical serving
↓
SAC / BI
Separate:
- model runtime
- prediction persistence
- analytical serving
- dashboard consumption
A dashboard should not depend directly on a fragile training workflow.
20.57 Model serving versus analytical serving
Model serving
Give me a prediction now.
Analytical serving
Show me predictions across millions of records.
Do not make a dashboard call a real-time endpoint once per row.
Prefer:
Batch inference
↓
Prediction table
↓
Analytical serving
↓
Dashboard
unless synchronous prediction is genuinely required.
20.58 Production runbook
Every production model should document:
Ownership
- business owner
- ML owner
- platform owner
Dependencies
- source tables
- feature tables
- model
- endpoint
- jobs
SLOs
- freshness
- latency
- availability
- model quality
Recovery
- rollback version
- fallback behavior
- escalation path
Monitoring
- dashboards
- alerts
- thresholds
Change process
- release
- approval
- rollback
20.59 Deployment checklist
Before production:
- code reviewed
- tests passed
- bundle validated
- staging passed
- data contract validated
- model signature validated
- evaluation passed
- security approved
- permissions verified
- monitoring enabled
- rollback tested
- runbook published
- owner confirmed
20.60 Troubleshooting matrix
| Symptom | Likely layer | First investigation |
|---|---|---|
| Endpoint unavailable | Serving | Endpoint status |
| High latency | Serving/features | Request path |
| Predictions suddenly shift | Data/model | Drift and upstream changes |
| Quality drops | Model/data | Labels and drift |
| Training fails | Pipeline | Task logs |
| Duplicate predictions | Inference | Idempotency |
| Wrong environment deployed | CI/CD | Bundle target |
| Unauthorized access | Governance | Identity and grants |
| Retraining loop | Automation | Trigger and policy |
| Cost spike | Compute | Retries and utilization |
20.61 Anti-pattern: manual production notebooks
Problem:
- no reproducibility
- weak review
- difficult rollback
- hidden state
Better:
Git → CI → Bundle → Production
20.62 Anti-pattern: train once and forget
Problem:
Data and business behavior change.
Better:
Monitor → Evaluate → Retrain when justified → Promote
20.63 Anti-pattern: retrain on every signal
Problem:
- cost
- noise
- unstable models
- operational churn
Better:
- thresholds
- persistence windows
- cooldown
- business impact
- evaluation gates
20.64 Anti-pattern: accuracy-only promotion
Better:
Technical metrics
+
Business metrics
+
Risk
+
Operational constraints
20.65 Anti-pattern: real-time by default
Ask:
- What latency is required?
- How many predictions?
- What is the cost?
- Does the consumer need synchronous response?
Then choose batch, streaming, or real-time.
20.66 Anti-pattern: every team builds its own platform
Better:
Central platform
├── templates
├── policies
├── CI/CD
└── observability
↓
Domain teams
├── features
├── models
└── business metrics
20.67 Lab 1 — Reproducible training
Build a project with:
- Git
- MLflow
- training code
- evaluation
- Unity Catalog model registration
Document:
- input data
- features
- metrics
- signature
- lineage
- reproducibility evidence
20.68 Lab 2 — DEV/STAGING/PROD
Design:
DEV → STAGING → PROD
Implement environment-specific configuration and identify what prevents an engineer from accidentally deploying to production.
20.69 Lab 3 — Declarative Automation Bundle
Create a bundle defining:
- training job
- inference job
- environment targets
Practice:
validate → deploy → run → inspect
Verify the current CLI and bundle engine before applying production commands.
20.70 Lab 4 — Champion/challenger
Create:
Champion = Version A
Challenger = Version B
Compare:
- precision
- recall
- business KPI
- segment performance
Then define a promotion gate.
20.71 Lab 5 — Model serving
Design:
- request schema
- response schema
- latency SLO
- timeout
- authentication
- monitoring
- rollback
Then justify whether real-time serving is necessary.
20.72 Lab 6 — Batch inference
Build:
Source
↓
Inference job
↓
Prediction table
↓
Quality checks
↓
SAC-ready dataset
Add idempotency and replay handling.
20.73 Lab 7 — Drift-driven retraining
Design:
Monitor
↓
Drift threshold
↓
Trigger
↓
Train
↓
Evaluate
↓
Promote only if passed
Add a cooldown so repeated alerts cannot create an uncontrolled loop.
20.74 Lab 8 — Failure injection
Simulate:
- stale feature table
- schema change
- endpoint failure
- bad model
- wrong environment deployment
- failed training task
For each document:
- detection
- diagnosis
- mitigation
- rollback
- prevention
20.75 Enterprise architect exercise
Design an enterprise ML platform with:
- 20 models
- 5 domains
- daily batch inference
- 3 real-time models
- regulated finance use cases
- SAC consumption
- dev/staging/prod
- automated retraining
- audit requirements
- multi-region requirements
Include:
Platform
- workspace topology
- identity
- networking
- governance
Data
- ingestion
- features
- quality
ML
- MLflow
- Unity Catalog models
- training
- evaluation
- promotion
Delivery
- Git
- CI/CD
- Declarative Automation Bundles
- Lakeflow Jobs
Serving
- batch
- real-time
- rollback
Operations
- monitoring
- alerting
- retraining
- incident response
Then defend:
Why this architecture and not a simpler one?
20.76 Architect decision framework
Ask:
1. What is the business SLO?
Without it, technical architecture is guesswork.
2. What changes most often?
Code, data, model, or serving?
3. What is the failure cost?
Low-risk recommendation and regulated finance should not have identical controls.
4. Where should training occur?
Development, staging, or production?
5. What should be promoted?
Code, model, or both?
6. How is quality measured?
Technical + business + operational.
7. How is rollback performed?
If unclear, production readiness is incomplete.
8. What is automated?
Automate toil without removing necessary controls.
20.77 MLOps maturity model
Level 1 — Notebook ML
Notebook → model
Level 2 — Tracked ML
Notebook → MLflow
Level 3 — Governed ML
MLflow → Unity Catalog
Level 4 — Automated ML
Git → CI/CD → Jobs → model
Level 5 — Production MLOps
CI/CD
+
training
+
evaluation
+
serving
+
monitoring
+
retraining
Level 6 — Enterprise ML platform
Multi-team
+
governance
+
SLOs
+
FinOps
+
HA/DR
+
operating model
Your target is Level 6.
20.78 Interview questions
- What is MLOps?
- Why separate environments?
- Why deploy code rather than models?
- When is model promotion justified?
- What are Declarative Automation Bundles?
- Where does Terraform fit?
- What does Lakeflow Jobs do?
- How does CI/CD work for ML?
- What is champion/challenger?
- What is canary deployment?
- When would you use shadow deployment?
- When is batch better than real-time?
- What is data drift?
- What is concept drift?
- How do delayed labels affect monitoring?
- How do you automate retraining safely?
- How do you design rollback?
- How do you secure production ML?
- How do you integrate predictions with SAC?
- What makes an ML platform enterprise-grade?
20.79 Final mental model
ENTERPRISE MLOps
BUSINESS OBJECTIVE
↓
DATA CONTRACT
↓
FEATURE LOGIC
↓
GIT
↓
CI / TEST
↓
TRAINING JOB
↓
MLflow
↓
EVALUATION
↓
UNITY CATALOG MODEL
↓
PROMOTION POLICY
/ \
reject promote
│ │
↓ ↓
iterate serving
│
┌────────┴────────┐
↓ ↓
REAL-TIME BATCH
│ │
└────────┬────────┘
↓
MONITORING
↓
DRIFT / QUALITY
↓
RETRAINING
↓
EVALUATION
↓
CONTROLLED UPDATE
The architect-level insight:
MLOps is not a deployment pipeline. It is the operating system for continuously changing machine-learning products.
20.80 Production readiness checklist
Business
- objective defined
- business KPI defined
- owner assigned
- risk classification completed
Data
- contracts defined
- freshness monitored
- quality gates implemented
- feature lineage understood
Model
- reproducible training
- MLflow tracking
- model signature
- Unity Catalog registration
- evaluation thresholds
- champion/challenger strategy
Delivery
- Git source control
- CI tests
- staging environment
- production automation
- bundle validated
- identity separated
Serving
- batch/real-time decision justified
- latency SLO
- error handling
- authentication
- dependency monitoring
- rollback
Operations
- dashboards
- alerts
- drift monitoring
- incident runbook
- retraining policy
- cost monitoring
Governance
- least privilege
- auditability
- lineage
- regulated-use controls where applicable
20.81 Official references
- Databricks MLOps workflows
- Model deployment patterns
- MLOps Stacks
- CI/CD for ML
- Machine learning lifecycle
- Declarative Automation Bundles
- Databricks CI/CD
- MLflow 3 deployment jobs
Version note: Databricks changes product names, APIs, deployment behavior, and preview capabilities. This phase uses current terminology as of November 2026. Verify cloud- and version-specific commands before production use.
20.82 Phase completion standard
You are ready for Phase 21 only when you can design and defend:
Git
↓
CI/CD
↓
Declarative Automation Bundles
↓
Lakeflow Jobs
↓
MLflow
↓
Unity Catalog
↓
Evaluation
↓
Promotion
↓
Model Serving / Batch
↓
Monitoring
↓
Retraining
↓
Rollback
and explain why each boundary exists, what failure it prevents, what it costs, and when you would choose a different architecture.