19. MLflow & Machine Learning Engineering
Learning objective: take a machine-learning problem from reproducible experimentation through evaluation, governed model registration, deployment preparation, monitoring, rollback, and retraining using Databricks and MLflow.
Phase 18 taught you to design the enterprise lakehouse platform.
Phase 19 introduces a new production object:
The model.
A production model is not merely a Python file. It is a governed asset connected to data, code, experiments, metrics, artifacts, lineage, permissions, evaluation evidence, deployment configuration, monitoring, and an owner.
Databricks currently recommends Models in Unity Catalog for governing and deploying models. MLflow 3 uses the Unity Catalog model registry by default through the databricks-uc registry URI. The older workspace model registry is a legacy path.
19.1 The ML engineering problem
A beginner thinks:
Train model → save model → predict
An enterprise architect thinks:
Business objective
↓
Data definition
↓
Feature / training dataset
↓
Experiment
↓
Reproducible training
↓
Evaluation
↓
Model registration
↓
Approval / promotion
↓
Deployment
↓
Inference
↓
Monitoring
↓
Retraining
↓
Controlled replacement
The engineering problem is not merely model accuracy.
It is:
How do we repeatedly produce trustworthy model versions and safely move them into production?
19.2 MLflow mental model
MLflow provides lifecycle capabilities rather than being a machine-learning algorithm.
| Capability | Purpose |
|---|---|
| Tracking | Record experiments |
| Runs | Capture one training/evaluation execution |
| Parameters | Record configuration |
| Metrics | Record measured outcomes |
| Artifacts | Store files produced by a run |
| Logged Models | Identify model artifacts and metadata |
| Model Registry | Govern reusable production models |
| Evaluation | Compare model behavior |
| Deployment | Connect models to inference |
| Tracing | Observe modern AI/ML applications |
Databricks-managed MLflow integrates tracking, model governance, lineage, and deployment with the broader Databricks platform.
19.3 Experiment, run, and model
Experiment
An experiment organizes related work.
procurement_supplier_risk
Run
A run represents one execution.
Run 001
Run 002
Run 003
Each run can have different parameters, feature versions, training data, code commits, and metrics.
Model
A model is a trained artifact intended for reuse.
Experiment
├── Run 001 → candidate
├── Run 002 → candidate
└── Run 003 → candidate
The winning model should be selected using evidence, not because it was the last run.
19.4 What a reproducible run must capture
Inputs
- training dataset reference
- validation dataset reference
- feature definitions
- filters
- sampling rules
Code
- Git commit
- source package or notebook
- dependency versions
Configuration
- algorithm
- hyperparameters
- random seed
- relevant compute configuration
Outputs
- metrics
- model artifact
- evaluation reports
- plots
- feature importance
- error analysis
Governance
- owner
- experiment
- business use case
- data classification
- approval evidence
A model without reproducibility is difficult to defend during an incident.
19.5 First MLflow experiment
A minimal Python pattern:
import mlflow
with mlflow.start_run():
mlflow.log_param("max_depth", 8)
mlflow.log_param("learning_rate", 0.05)
accuracy = 0.91
mlflow.log_metric("accuracy", accuracy)
mlflow.log_artifact("evaluation_report.html")
The important lesson is not the API syntax.
The important lesson is that the run becomes an evidence record.
19.6 Parameters, metrics, and artifacts
Parameters
Inputs to an experiment.
Examples:
- learning rate
- tree depth
- regularization
- batch size
Metrics
Measured outcomes.
Examples:
- accuracy
- precision
- recall
- F1
- RMSE
- MAE
- AUC
Artifacts
Files produced by the run.
Examples:
- model files
- confusion matrices
- feature importance plots
- evaluation reports
- serialized preprocessing objects
Architect rule:
If a future engineer needs it to understand why a model won, consider logging it.
19.7 MLflow and Git are complementary
Do not use MLflow as a replacement for source control.
Git
↓
source code + configuration + tests
↓
MLflow
↓
run evidence + metrics + artifacts + model
↓
Unity Catalog
↓
governed model lifecycle
Git answers:
What code did we write?
MLflow answers:
What happened when we executed it?
Unity Catalog answers:
What governed model asset are we allowed to use and where did it come from?
Databricks documents lineage connecting model versions to training runs, datasets, environments, and Git commits.
19.8 Feature engineering architecture
Separate:
- raw source data
- cleaned/conformed data
- feature computation
- training dataset
- inference features
SAP / operational sources
↓
Lakehouse data
↓
Feature engineering
↓
Feature datasets
↙ ↘
Training Inference
↓ ↓
Model ←───────┘
The key architectural concern is training-serving consistency.
A feature calculated one way during training and another way during inference can create silent model degradation.
19.9 Training-serving skew
Suppose training calculates:
supplier_avg_delay = average(delay_days)
But production calculates the same metric using:
- a different time window
- a different null policy
- a different timezone
- incomplete source data
The model sees a different feature distribution.
That is training-serving skew.
Prevent it with:
- reusable feature definitions
- shared transformation logic
- versioned datasets
- data contracts
- validation
- lineage
19.10 Model evaluation
Accuracy alone is rarely sufficient.
For classification, evaluate:
- precision
- recall
- F1
- ROC AUC
- PR AUC
- confusion matrix
- calibration
For regression:
- MAE
- RMSE
- R-squared
- residual analysis
- error distribution
For ranking/recommendation:
- precision at K
- recall at K
- NDCG
- business conversion metrics
For every model define:
Technical metric
+
Business metric
+
Risk threshold
A model can be technically excellent and commercially useless.
19.11 Evaluation must be decision-oriented
Suppose a fraud model has 99 percent accuracy and fraud prevalence is 1 percent.
A trivial model predicting "not fraud" can appear excellent.
Therefore ask:
- What is the cost of false positives?
- What is the cost of false negatives?
- What is the operational capacity for investigation?
- What threshold optimizes business value?
- Is model performance stable across segments?
Architecture is about translating model metrics into business decisions.
19.12 Model signatures
A model signature describes expected inputs and outputs.
INPUT
supplier_age: integer
annual_spend: double
late_delivery_rate: double
OUTPUT
risk_score: double
risk_class: string
Signatures reduce ambiguity between training and serving.
Databricks requires models in Unity Catalog to have a model signature, making the input/output contract explicit.
19.13 Registering a model in Unity Catalog
Modern Databricks architecture should use the Unity Catalog model registry.
Typical naming:
catalog.schema.model
Example:
ml_prod.procurement.supplier_risk
The model becomes a governed catalog asset with controls around:
- access
- discovery
- lineage
- sharing
- lifecycle
- audit
Databricks recommends Models in Unity Catalog for model governance and deployment.
19.14 Legacy Workspace Model Registry
Older tutorials may use the Workspace Model Registry.
Treat it as a legacy architecture.
Current Databricks guidance says workspaces enabled for Unity Catalog should use Models in Unity Catalog instead. MLflow 3 defaults to the Unity Catalog registry.
You still need to understand the legacy registry because enterprise migrations may contain it.
Architect rule:
Learn legacy behavior for migration and troubleshooting, not as the default for new architecture.
19.15 Model versioning
Think in versions:
supplier_risk
├── Version 1
├── Version 2
├── Version 3
└── Version 4
A new version should exist because something meaningful changed:
- training data
- features
- algorithm
- hyperparameters
- code
- dependency
- business policy
19.16 Champion and challenger
A robust production pattern is:
Current production
CHAMPION
│
│ compare
▼
CHALLENGER
│
evaluation
│
┌─────┴─────┐
│ │
fails passes
│ │
reject promote
The challenger must beat an explicit baseline.
Databricks MLOps guidance uses champion/challenger comparisons and supports automated or controlled promotion workflows.
19.17 Promotion is a policy decision
Do not promote because:
"The model has better accuracy."
Define a promotion gate.
Precision ≥ target
Recall ≥ target
Business KPI ≥ target
No critical segment regression
Data quality ≥ threshold
Latency ≤ SLO
Security checks passed
Reproducibility verified
Promotion is therefore an engineering control.
19.18 Model Serving architecture
Databricks Model Serving provides a managed interface for real-time and batch inference and exposes served models through REST APIs. Current documentation describes automatic scaling for serving infrastructure.
Application
│
│ HTTPS / REST
▼
Model Serving Endpoint
│
▼
Registered Model
│
▼
Unity Catalog
│
├── lineage
├── permissions
└── model versions
The endpoint is not the model.
It is the runtime interface through which a model is consumed.
19.19 Real-time versus batch inference
Real-time
Use when the decision is needed immediately.
Examples:
- fraud scoring
- recommendation
- application risk score
Request → endpoint → prediction → response
Batch
Use when predictions can be calculated periodically.
Examples:
- monthly churn score
- supplier risk
- financial forecasting
Data → inference job → predictions table → BI
Do not build real-time serving when the business only needs daily predictions.
19.20 Inference architecture
A production inference path should define:
- input schema
- validation
- model version
- timeout
- retry behavior
- fallback
- logging
- security
- monitoring
- rollback
For batch inference also define:
- partition strategy
- idempotency
- replay
- backfill
- reconciliation
19.21 Model monitoring
Production monitoring has at least four dimensions.
System health
- endpoint availability
- latency
- throughput
- errors
- saturation
Data health
- missing values
- schema changes
- freshness
- distribution changes
Model health
- prediction distribution
- drift
- confidence
- performance where labels are available
Business health
- conversion
- revenue
- loss rate
- fraud capture
- operational workload
A healthy endpoint does not guarantee a healthy model.
19.22 Data drift
Data drift means the input distribution changes.
Training supplier spend
↓
Average = 1.2M
Production supplier spend
↓
Average = 2.8M
Monitor:
- distribution
- missingness
- cardinality
- category changes
- ranges
- feature correlations
19.23 Concept drift
Concept drift is more serious.
The relationship between inputs and outcomes changes.
Before:
high spend → high risk
After:
high spend → strategic supplier
The feature distribution can remain similar while the prediction relationship changes.
Concept drift usually requires outcome labels and business monitoring.
19.24 Model decay
A model can decay because:
- customer behavior changes
- suppliers change
- market conditions change
- regulations change
- source systems change
- feature definitions change
Define retraining triggers:
Scheduled
OR
Drift threshold exceeded
OR
Performance threshold breached
OR
Business policy changed
OR
Feature contract changed
19.25 Retraining architecture
A production retraining loop:
Production predictions
↓
Monitoring
↓
Drift / performance signal
↓
Training dataset refresh
↓
Training job
↓
MLflow experiment
↓
Evaluation
↓
Candidate model
↓
Promotion gate
↓
Registry
↓
Deployment
The loop must be controlled.
Automatic retraining without evaluation can automate failure.
19.26 MLOps architecture
A mature Databricks ML platform can combine:
- notebooks or source repositories
- MLflow
- Unity Catalog
- Lakeflow Jobs
- Declarative Automation Bundles
- CI/CD
- Model Serving
- monitoring
Databricks MLOps Stack documentation uses this general combination for model development, registry, serving, orchestration, infrastructure-as-code, CI/CD, and monitoring.
Git
│
▼
CI/CD validation
│
▼
Databricks workspace
│
┌─────────┴─────────┐
▼ ▼
Training job Evaluation job
│ │
└─────────┬─────────┘
▼
MLflow
│
▼
Unity Catalog Model
│
Promotion gate
│
┌─────────┴─────────┐
▼ ▼
Batch inference Model Serving
│ │
└─────────┬─────────┘
▼
Monitoring
│
▼
Retraining
19.27 CI/CD for ML
Application CI/CD asks:
Does the software behave correctly?
ML CI/CD must also ask:
Does the model behave correctly on representative data?
A production pipeline can include:
- linting
- unit tests
- data contract tests
- feature tests
- training
- evaluation
- model validation
- security validation
- deployment
- post-deployment verification
A model can pass every software test and still be a bad model.
19.28 ML testing pyramid
Unit tests
Test deterministic transformations.
Data tests
Test:
- schema
- nulls
- ranges
- uniqueness
- freshness
Feature tests
Test feature definitions and leakage.
Model tests
Test:
- signature
- prediction shape
- expected ranges
- serialization
Evaluation tests
Test business and statistical thresholds.
Integration tests
Test model with serving dependencies.
Production tests
Test real inference behavior and rollback.
19.29 Data leakage
Data leakage occurs when training receives information unavailable at prediction time.
Example:
Predict whether a supplier will be late.
Bad feature:
final_delivery_status
because the final status is only known after delivery.
The model may achieve excellent offline accuracy while being useless in production.
Architect rule:
Every feature must be available at the exact prediction timestamp.
19.30 Reproducibility
To reproduce a model, know:
Code
+
Data
+
Features
+
Dependencies
+
Parameters
+
Randomness
+
Environment
Reproducibility is especially important for:
- audits
- incidents
- regulatory review
- model comparison
- rollback
- retraining
19.31 Model rollback
Never deploy without a rollback strategy.
Production
│
Version 7
│
incident
▼
Version 6
Rollback should define:
- who can execute it
- how quickly
- what evidence triggers it
- how traffic is switched
- how the incident is recorded
- how the failed model is isolated
19.32 Model security
Protect:
- training data
- model artifacts
- features
- endpoints
- credentials
- inference payloads
- prediction outputs
Use:
- Unity Catalog permissions
- service principals
- least privilege
- network controls
- audit logs
- secret management
- controlled deployment identities
19.33 Model governance
A production model should have:
| Governance field | Example |
|---|---|
| Owner | Procurement Data Science |
| Business purpose | Supplier risk |
| Model type | Gradient boosting |
| Training dataset | Versioned governed table |
| Registered model | Unity Catalog |
| Criticality | Tier 1 |
| Data classification | Confidential |
| SLO | p95 latency target |
| Review cadence | Quarterly |
| Rollback version | Previous approved version |
This makes the model an enterprise-managed asset.
19.34 Model lifecycle
Use a lifecycle such as:
Idea
↓
Experiment
↓
Candidate
↓
Validated
↓
Approved
↓
Production
↓
Monitored
↓
Deprecated
↓
Archived
The exact labels can vary.
The important principle is that promotion and retirement are governed transitions.
19.35 MLflow 3 mental model
MLflow 3 expands the model lifecycle beyond a simple run-and-register workflow.
Current Databricks documentation describes MLflow 3 improvements that expose model parameters and metrics in the Unity Catalog model registry and connect evaluation information with model versions.
Think:
Experiment
↓
Logged Model
↓
Evaluation evidence
↓
Registered Model
↓
Deployment
↓
Production evidence
This creates a stronger bridge between development evidence and operational evidence.
19.36 Architecting ML data products
A mature architecture connects:
Data product
│
├── source contract
├── quality
├── feature definitions
├── training dataset
├── model
├── inference output
└── business KPI
Example:
PROCUREMENT
│
├── Supplier master
├── Purchase orders
├── Goods receipts
├── Invoice history
│
└── Supplier Risk ML Product
├── features
├── model
├── predictions
└── risk dashboard
This is where enterprise data architecture and ML engineering converge.
19.37 SAP-to-ML architecture example
Suppose the business wants supplier late-delivery prediction.
Sources:
- SAP purchasing documents
- goods receipts
- supplier master
- historical delivery performance
Architecture:
SAP S/4HANA
│
▼
Ingestion / CDC
│
▼
Lakehouse
│
▼
Conformed procurement data
│
▼
Feature engineering
│
▼
Training dataset
│
▼
MLflow training
│
▼
Unity Catalog model
│
├── batch predictions
│ ↓
│ Delta
│ ↓
│ SAC
│
└── real-time endpoint
↓
Procurement app
This does not require pretending BW, Datasphere, and Databricks have identical semantics. Map capabilities, not product names.
19.38 Common ML architecture anti-patterns
Anti-pattern 1 — Notebook as production
Training, deployment, and business logic live in one notebook.
Problem: difficult to test, version, automate, and recover.
Anti-pattern 2 — Model file on a shared folder
Problem: no governed lifecycle.
Anti-pattern 3 — Accuracy-only promotion
Problem: ignores business cost and segment behavior.
Anti-pattern 4 — No model signature
Problem: ambiguous serving contract.
Anti-pattern 5 — Training-serving skew
Problem: production features differ from training.
Anti-pattern 6 — No rollback
Problem: incidents become emergency rebuilds.
Anti-pattern 7 — Automatic retraining without gates
Problem: bad data can automatically produce a bad production model.
Anti-pattern 8 — Treating MLflow as Git
Problem: code history and experiment evidence solve different problems.
19.39 Hands-on lab — first MLflow experiment
Objective
Build a reproducible classification experiment.
Steps
- Load a governed dataset.
- Split train and validation data.
- Train at least three candidate configurations.
- Log parameters.
- Log metrics.
- Log evaluation artifacts.
- Compare runs.
- Identify the winning candidate.
- Record why it won.
Deliverable:
| Run | Parameters | Accuracy | Precision | Recall | Decision |
|---|---|---|---|---|---|
| 001 | baseline | ||||
| 002 | tuned | ||||
| 003 | tuned |
19.40 Hands-on lab — governed model
Objective
Register the selected model in Unity Catalog.
Experiment
↓
Winning run
↓
Model signature
↓
Unity Catalog model
↓
Permissions
↓
Model consumer
Questions:
- Who can create the model?
- Who can modify it?
- Who can deploy it?
- Who can invoke it?
- Which catalog owns it?
- What is the rollback version?
19.41 Hands-on lab — champion/challenger
Build:
Champion
│
├── evaluation
│
Challenger
│
└── comparison
Define:
- one technical metric
- one business metric
- one safety threshold
Automatically reject a challenger that fails any critical gate.
19.42 Hands-on lab — production monitoring
Design monitoring for:
- latency
- prediction distribution
- missing features
- drift
- model performance
- business KPI
Then define:
What condition causes investigation?
What condition causes rollback?
What condition causes retraining?
19.43 Architect exercise — global ML platform
Scenario
A global enterprise has:
- 30 data domains
- SAP source systems
- 100+ analytical datasets
- 20 production ML models
- regional regulatory requirements
- SAC dashboards
- real-time applications
- batch scoring
- centralized platform engineering
Design
Create an architecture covering:
- ML workspace strategy
- Unity Catalog model organization
- experiment organization
- feature architecture
- training pipelines
- MLflow tracking
- model registry
- evaluation gates
- deployment
- batch inference
- real-time serving
- monitoring
- retraining
- rollback
- security
- FinOps
- model ownership
Defend
Explain:
- Why is the model registry centralized?
- How do you prevent unauthorized deployment?
- How do you reproduce a production model?
- How do you detect training-serving skew?
- What happens when a production model degrades?
- Who owns the model?
- Who pays for inference?
- How does the design work across regions?
19.44 Troubleshooting matrix
| Symptom | Likely cause | First investigation |
|---|---|---|
| Model cannot be registered | UC permissions or registry configuration | Check catalog/schema/model privileges |
| Model works in notebook but not serving | Dependency/signature/environment issue | Validate signature and serving dependencies |
| Offline accuracy is high but production poor | Leakage or skew | Compare feature availability and distributions |
| New model is slower | Model/runtime change | Compare latency and dependency profile |
| Predictions suddenly shift | Data drift | Compare feature distributions |
| Endpoint healthy but business KPI falls | Concept drift | Compare outcomes and segments |
| Retraining creates unstable models | Weak promotion gate | Review evaluation policy |
| Cannot reproduce old model | Missing code/data/environment evidence | Trace run, model, dataset, and Git commit |
| Rollback takes too long | No operational runbook | Test rollback path |
| Costs spike | Inference/workload scaling | Analyze serving and batch usage |
19.45 Production readiness checklist
Experimentation
- Experiments organized
- Parameters logged
- Metrics logged
- Artifacts retained
- Code version captured
- Data version identified
- Reproducibility tested
Model
- Model signature defined
- Evaluation criteria defined
- Business metric defined
- Model registered in Unity Catalog
- Ownership defined
- Permissions defined
Deployment
- Batch/real-time choice justified
- Serving contract defined
- Security configured
- Rollback tested
- CI/CD implemented
Monitoring
- Endpoint health monitored
- Feature drift monitored
- Model performance monitored
- Business KPI monitored
- Alert thresholds defined
- Retraining policy defined
Governance
- Lineage available
- Auditability verified
- Model review cadence defined
- Retirement policy defined
- Regulatory requirements documented
19.46 Interview questions
- What problem does MLflow solve?
- What is the difference between an experiment and a run?
- What should be logged in MLflow?
- Why are Git and MLflow both required?
- What is a model signature?
- Why should new Databricks architectures use Unity Catalog models?
- What is the difference between the workspace model registry and Unity Catalog model registry?
- How would you implement champion/challenger?
- How do you prevent data leakage?
- What is training-serving skew?
- What is data drift?
- What is concept drift?
- When should you use batch inference instead of online serving?
- What belongs in an ML CI/CD pipeline?
- How do you roll back a model?
- How would you design model monitoring?
- How would you reproduce a model from six months ago?
- How would you integrate SAP data into an ML platform?
- How would you govern 100 models across multiple domains?
- How would you control model deployment in production?
19.47 Architect decision framework
Use this order:
Business decision
↓
Prediction requirement
↓
Latency requirement
↓
Data availability
↓
Feature design
↓
Training strategy
↓
Evaluation criteria
↓
Governance
↓
Deployment pattern
↓
Monitoring
↓
Retraining
↓
Rollback
↓
Cost
Never start with:
"Which MLflow API should I call?"
Start with:
What decision are we automating, what evidence proves the model is safe and useful, and how will we operate it after deployment?
19.48 Final mental model
The production ML lifecycle is:
DATA
↓
FEATURES
↓
EXPERIMENT
↓
RUNS
↓
EVALUATION
↓
MODEL
↓
UNITY CATALOG
↓
PROMOTION
↓
DEPLOYMENT
↓
INFERENCE
↓
MONITORING
↓
RETRAINING
↓
CONTROLLED REPLACEMENT
The enterprise architect's responsibility is not to make a model once.
It is to build the system that can repeatedly produce, govern, deploy, observe, recover, and improve models safely.