04.12 PySpark Project — Customer 360 Capstone
This is the capstone for the PySpark module.
You will build a production-style Customer 360 & Sales Analytics pipeline that deliberately combines the skills from 04.1–04.11:
- DataFrames
- explicit schemas
- filtering and projection
- aggregations
- joins
- windows
- null handling
- data quality
- advanced PySpark
- performance engineering
- Delta Lake patterns
The objective is not to produce a notebook that merely runs.
The objective is to produce a pipeline that you can explain as an engineer and defend as an architect.
1. Business Scenario
A retail company receives customer and order data from operational systems.
The source data is imperfect:
- customers can appear more than once
- attributes can change
- orders can arrive late
- some records contain nulls
- customer identifiers can be malformed
- duplicate orders can occur
- one customer can have thousands of orders
- source files can contain unexpected columns
The analytics team needs a trusted Gold dataset for:
- revenue reporting
- customer ranking
- customer segmentation
- order KPIs
- BI dashboards
- downstream ML features
Business question
For every valid customer, what is their current profile, order activity, revenue, rank and latest order information?
2. Target Architecture
Use a simple Medallion-style flow:
SOURCE
Customers + Orders
|
v
+----------------+
| BRONZE |
| raw source data|
+----------------+
|
v
+----------------+
| SILVER |
| clean + typed |
| deduplicated |
| validated |
+----------------+
|
v
+----------------+
| GOLD |
| Customer 360 |
| revenue + rank |
| latest order |
+----------------+
|
+----------> BI / SQL / Analytics
|
+----------> ML / Feature Engineering
The important separation is:
Bronze preserves source reality. Silver establishes trusted records. Gold establishes business meaning.
3. Project Repository Structure
A professional implementation should separate configuration, transformations and tests.
Recommended structure:
pyspark-customer360/
│
├── notebooks/
│ ├── 01_bronze_ingestion
│ ├── 02_silver_customers
│ ├── 03_silver_orders
│ └── 04_gold_customer360
│
├── src/
│ ├── schemas/
│ │ ├── customer_schema.py
│ │ └── order_schema.py
│ │
│ ├── transformations/
│ │ ├── customers.py
│ │ ├── orders.py
│ │ └── customer360.py
│ │
│ └── quality/
│ └── rules.py
│
├── tests/
│ ├── test_customers.py
│ ├── test_orders.py
│ └── test_customer360.py
│
└── README.md
For a learning project, notebooks are acceptable.
For production, move reusable logic into source modules and keep notebooks thin.
4. Source Data Contract
Customers
Suggested fields:
| Column | Type | Meaning |
|---|---|---|
| customer_id | string | Business customer identifier |
| customer_name | string | Customer name |
| string | ||
| country | string | Country |
| segment | string | Customer segment |
| updated_at | timestamp | Source update timestamp |
| source_system | string | Source identifier |
Orders
| Column | Type | Meaning |
|---|---|---|
| order_id | string | Order identifier |
| customer_id | string | Customer identifier |
| order_date | date | Business order date |
| amount | decimal(18,2) | Order value |
| status | string | Order status |
| updated_at | timestamp | Source update timestamp |
| source_system | string | Source identifier |
5. Data Quality Problems to Inject
Do not build a perfectly clean dataset.
Inject realistic defects:
Customers
- duplicate customer IDs
- null customer IDs
- blank names
- invalid email addresses
- multiple versions of the same customer
Orders
- duplicate order IDs
- null customer IDs
- negative amounts
- invalid statuses
- future order dates
- late-arriving records
This makes the project useful for real-world learning.
6. Generate Test Data
A small local dataset is enough to understand the logic.
Example:
customers_data = [
("C001", "Alice", "alice@example.com", "IN", "Gold", "2026-01-10 10:00:00", "CRM"),
("C001", "Alice Sharma", "alice@example.com", "IN", "Platinum", "2026-02-10 10:00:00", "CRM"),
("C002", "Bob", None, "US", "Silver", "2026-01-12 09:00:00", "CRM"),
(None, "Invalid Customer", "bad@example.com", "IN", "Bronze", "2026-01-01 08:00:00", "CRM")
]
Orders:
orders_data = [
("O001", "C001", "2026-02-01", 1500.00, "COMPLETED", "2026-02-01 12:00:00", "ERP"),
("O002", "C001", "2026-02-15", 2500.00, "COMPLETED", "2026-02-15 12:00:00", "ERP"),
("O003", "C002", "2026-02-20", 900.00, "CANCELLED", "2026-02-20 12:00:00", "ERP")
]
For the performance lab later, scale this to millions of rows.
7. Define Explicit Schemas
Never let an enterprise pipeline depend blindly on inference.
from pyspark.sql.types import (
StructType, StructField,
StringType, DateType,
TimestampType, DecimalType
)
customer_schema = StructType([
StructField("customer_id", StringType(), True),
StructField("customer_name", StringType(), True),
StructField("email", StringType(), True),
StructField("country", StringType(), True),
StructField("segment", StringType(), True),
StructField("updated_at", TimestampType(), True),
StructField("source_system", StringType(), True)
])
Orders should use a decimal type for monetary values:
StructField("amount", DecimalType(18, 2), True)
Avoid using floating-point types for financial measures when exact decimal semantics are required.
8. Bronze Layer
Bronze should preserve the source as faithfully as practical.
Typical responsibilities:
- ingest
- add ingestion metadata
- retain source columns
- record source system
- capture ingestion timestamp
- avoid destructive business transformations
Example:
from pyspark.sql import functions as F
bronze_customers = (
customers_raw
.withColumn("_ingested_at", F.current_timestamp())
.withColumn("_source_file", F.input_file_name())
)
Bronze is not the place to silently “fix” business meaning.
9. Silver Customer Transformation
The Silver layer establishes trusted customer records.
Step 1 — Remove invalid identifiers
customers_clean = (
bronze_customers
.filter(
F.col("customer_id").isNotNull() &
(F.trim("customer_id") != "")
)
)
Step 2 — Standardize attributes
customers_clean = (
customers_clean
.withColumn("customer_name", F.trim("customer_name"))
.withColumn("country", F.upper(F.trim("country")))
.withColumn("email", F.lower(F.trim("email")))
)
10. Deduplicate to the Latest Customer Record
This is a key window-function exercise.
from pyspark.sql.window import Window
customer_window = (
Window
.partitionBy("customer_id")
.orderBy(F.col("updated_at").desc())
)
customers_latest = (
customers_clean
.withColumn("_rn", F.row_number().over(customer_window))
.filter(F.col("_rn") == 1)
.drop("_rn")
)
Why use a window?
Because the business rule is:
Keep the most recently updated record for each customer.
This is not the same as simply calling distinct.
11. Silver Order Transformation
Validate the order population.
orders_clean = (
bronze_orders
.filter(F.col("order_id").isNotNull())
.filter(F.col("customer_id").isNotNull())
.filter(F.col("amount").isNotNull())
)
Standardize status:
orders_clean = orders_clean.withColumn(
"status",
F.upper(F.trim("status"))
)
Define accepted statuses:
valid_statuses = [
"COMPLETED",
"CANCELLED",
"RETURNED",
"PENDING"
]
orders_clean = orders_clean.filter(
F.col("status").isin(valid_statuses)
)
12. Duplicate Order Handling
A duplicate order should not inflate revenue.
Use the latest source version:
order_window = (
Window
.partitionBy("order_id")
.orderBy(F.col("updated_at").desc())
)
orders_latest = (
orders_clean
.withColumn("_rn", F.row_number().over(order_window))
.filter(F.col("_rn") == 1)
.drop("_rn")
)
The business rule must be documented.
Never deduplicate merely because duplicate rows “look wrong.”
13. Data Quality Rules
Create explicit quality rules.
| Rule | Action |
|---|---|
| customer_id is null | Reject/quarantine |
| order_id is null | Reject/quarantine |
| amount is null | Reject/quarantine |
| amount < 0 | Business-rule review |
| invalid status | Reject/quarantine |
| invalid email | Flag |
| duplicate order | Keep latest version |
| unknown customer | Exception handling |
A mature pipeline distinguishes:
invalid record from business exception.
14. Quarantine Instead of Silent Deletion
Create a rejected dataset:
invalid_orders = orders_raw.filter(
F.col("order_id").isNull() |
F.col("customer_id").isNull() |
F.col("amount").isNull()
)
Keep:
- original record
- rejection reason
- ingestion timestamp
- source
- pipeline run identifier
Example:
invalid_orders = invalid_orders.withColumn(
"_reject_reason",
F.when(
F.col("order_id").isNull(),
F.lit("MISSING_ORDER_ID")
).when(
F.col("customer_id").isNull(),
F.lit("MISSING_CUSTOMER_ID")
).when(
F.col("amount").isNull(),
F.lit("MISSING_AMOUNT")
)
)
This creates an auditable data-quality path.
15. Define the Revenue Rule
Do not simply sum every order.
For this project:
COMPLETED → revenue
CANCELLED → no revenue
RETURNED → no positive revenue
PENDING → excluded from realized revenue
Then:
completed_orders = orders_latest.filter(
F.col("status") == "COMPLETED"
)
Aggregate:
customer_revenue = (
completed_orders
.groupBy("customer_id")
.agg(
F.sum("amount").alias("total_revenue"),
F.countDistinct("order_id").alias("order_count"),
F.max("order_date").alias("latest_order_date")
)
)
16. Build Customer 360
Join the trusted customer profile with order metrics.
customer360 = (
customers_latest
.join(
customer_revenue,
"customer_id",
"left"
)
)
Handle customers with no orders:
customer360 = customer360.fillna({
"total_revenue": 0,
"order_count": 0
})
Do not accidentally turn the left join into an inner join later in the transformation.
17. Customer Ranking
Rank customers by revenue.
rank_window = Window.orderBy(
F.col("total_revenue").desc()
)
customer360 = customer360.withColumn(
"revenue_rank",
F.dense_rank().over(rank_window)
)
For ranking within countries:
country_rank_window = (
Window
.partitionBy("country")
.orderBy(F.col("total_revenue").desc())
)
customer360 = customer360.withColumn(
"country_revenue_rank",
F.dense_rank().over(country_rank_window)
)
18. Latest Order Details
Revenue metrics are not enough.
Create latest-order information.
latest_order_window = (
Window
.partitionBy("customer_id")
.orderBy(
F.col("order_date").desc(),
F.col("updated_at").desc()
)
)
latest_orders = (
orders_latest
.withColumn("_rn", F.row_number().over(latest_order_window))
.filter(F.col("_rn") == 1)
.select(
"customer_id",
F.col("order_id").alias("latest_order_id"),
F.col("order_date").alias("latest_order_date"),
F.col("status").alias("latest_order_status")
)
)
Join it into Customer 360.
19. Final Gold Dataset
Recommended columns:
| Column | Purpose |
|---|---|
| customer_id | Business key |
| customer_name | Current customer name |
| Current email | |
| country | Geography |
| segment | Current segment |
| total_revenue | Realized revenue |
| order_count | Completed order count |
| latest_order_id | Latest order |
| latest_order_date | Latest order date |
| latest_order_status | Latest order status |
| revenue_rank | Global rank |
| country_revenue_rank | Country rank |
This is now a business-facing Gold dataset.
20. Performance Engineering
Do not wait until the end to think about performance.
Apply the 04.11 framework.
First: reduce input
- select only required columns
- filter invalid/unneeded records early
Second: inspect joins
Ask:
- Is customer data small?
- Is broadcast safe?
- Is the join key skewed?
- Are there duplicate keys?
Third: inspect the plan
customer360.explain("formatted")
Look for:
- Exchange
- Sort
- BroadcastHashJoin
- SortMergeJoin
- Window
- HashAggregate
- Python execution
Fourth: measure
Record:
- runtime
- input bytes
- shuffle
- task distribution
- spill
- output file count
21. Avoid Accidental Join Explosion
Before joining:
customers_latest.groupBy("customer_id").count().filter("count > 1")
This should return zero rows.
Likewise:
orders_latest.groupBy("order_id").count().filter("count > 1")
If a supposedly unique side contains duplicates, a join can multiply rows.
Example:
Customer table
C001 → 2 rows
Orders
C001 → 10 rows
Join result
2 × 10 = 20 rows
That can silently inflate metrics.
22. Performance Lab — Scale the Data
The small dataset teaches correctness.
The large dataset teaches engineering.
Scale the order dataset to:
10–50 million rows.
Then deliberately create:
- a skewed customer
- unnecessary columns
- a late filter
- an unnecessary repartition
- a non-selective join
Measure the baseline.
Then optimize one issue at a time.
Required evidence
BEFORE
Runtime:
Input:
Shuffle:
Spill:
Longest task:
Output files:
AFTER
Runtime:
Input:
Shuffle:
Spill:
Longest task:
Output files:
The learner should explain why each metric changed.
23. Add Delta Lake
Persist the layers as Delta tables.
Conceptually:
bronze_customers
bronze_orders
silver_customers
silver_orders
gold_customer360
Example:
(
customer360
.write
.format("delta")
.mode("overwrite")
.saveAsTable("analytics.gold_customer360")
)
In an enterprise implementation, use governed Unity Catalog objects rather than unmanaged ad-hoc paths.
24. Incremental Design
A production pipeline should not recalculate all history for every run.
Think in terms of:
New / changed source records
↓
Incremental Silver
↓
Affected customers
↓
Recompute affected Gold metrics
Possible incremental inputs:
- source update timestamp
- ingestion date
- CDC sequence
- source change version
The important design question is:
What is the smallest set of data that must be recomputed to produce a correct result?
25. Late-Arriving Data
Suppose an order from January arrives in March.
A simple “process only today's records” pipeline can miss its effect on customer metrics.
The design must account for:
- event date
- ingestion date
- source update timestamp
- watermark/checkpoint strategy when streaming
- affected customer recalculation
This is why business time and processing time are different concepts.
26. Testing Strategy
Create tests for:
Schema
- required columns exist
- types are correct
Data quality
- invalid IDs rejected
- invalid statuses rejected
- duplicate orders resolved
Business logic
- cancelled orders do not contribute to revenue
- customer rank is correct
- latest customer version is selected
Join correctness
- customer cardinality is preserved
- unknown customers are handled intentionally
Regression
Run the same test data after every transformation change.
27. Example Assertions
assert customers_latest.filter(
F.col("customer_id").isNull()
).count() == 0
Check uniqueness:
assert (
customers_latest
.groupBy("customer_id")
.count()
.filter("count > 1")
.count()
== 0
)
Check revenue:
unexpected = customer360.filter(
F.col("total_revenue") < 0
)
assert unexpected.count() == 0
In production, use a proper testing framework and CI rather than relying on notebook assertions alone.
28. Observability
A production implementation should publish pipeline metrics.
At minimum:
records_read
records_valid
records_rejected
customers_deduplicated
orders_deduplicated
unknown_customer_orders
gold_rows
runtime_seconds
input_bytes
shuffle_bytes
output_files
These metrics turn a pipeline from a black box into an observable system.
29. Failure Scenarios to Test
Deliberately test:
- empty source
- malformed records
- schema drift
- duplicate customer versions
- duplicate orders
- unknown customer IDs
- highly skewed customer
- huge input volume
- late-arriving order
- failed Gold write
- partial rerun
- repeated execution
A production pipeline is defined as much by its failure behavior as by its happy path.
30. Idempotency
The pipeline should be safe to rerun.
Ask:
If the same source batch is processed twice, do I create duplicate business records?
Idempotency can be designed through:
- deterministic deduplication
- stable business keys
- batch identifiers
- merge semantics
- transactional writes
Never use “the job usually runs once” as an idempotency strategy.
31. Architect-Level Design Review
At the end of the project, be able to answer:
Data
- What is the source contract?
- What is the business key?
- What is the grain of each table?
Transformation
- Why is the customer window required?
- Why is the order window required?
- Where are nulls handled?
Performance
- Where does shuffle occur?
- Can the customer table be broadcast?
- What happens under skew?
- What evidence proves the optimization?
Reliability
- Is the pipeline idempotent?
- How are failed runs recovered?
- How are rejected records retained?
Incremental processing
- How are changed customers detected?
- How are late orders handled?
- What data must be recomputed?
Governance
- Who can access Gold?
- Which fields contain sensitive customer information?
- How would lineage be established?
32. Interview Challenge
Imagine the interviewer says:
“Your Customer 360 pipeline takes 45 minutes. The business wants it below 10 minutes without doubling cost.”
Answer in this order:
1. Establish baseline
↓
2. Inspect Spark UI
↓
3. Inspect physical plan
↓
4. Identify scan/shuffle/skew bottleneck
↓
5. Reduce data movement
↓
6. Validate join strategy
↓
7. Improve table layout
↓
8. Validate AQE / Photon
↓
9. Right-size compute
↓
10. Benchmark cost + runtime
Do not answer:
“Increase the cluster.”
That is a resource response, not an engineering diagnosis.
33. Final Deliverables
A complete project submission should contain:
Code
- Bronze ingestion
- Customer Silver transformation
- Order Silver transformation
- Gold Customer 360 transformation
- Reusable transformation functions
Data quality
- Explicit schema
- Validation rules
- Quarantine logic
- Duplicate handling
Performance
- Physical plan
- Spark UI evidence
- Before/after benchmark
- Skew analysis
- Join-strategy explanation
Production engineering
- Delta tables
- Incremental design
- Idempotency
- Observability
- Tests
- Failure scenarios
Architecture
- Architecture diagram
- Data-flow explanation
- Design decisions
- Trade-offs
- Cost considerations
34. Capstone Success Criteria
You have completed this project successfully when you can:
- Build the pipeline from raw data.
- Explain every transformation.
- Prove data quality.
- Explain the physical plan.
- Identify shuffle.
- Diagnose skew.
- Explain the join strategy.
- Optimize the workload using evidence.
- Persist trusted Delta datasets.
- Explain how the design becomes incremental.
- Make the pipeline idempotent.
- Design failure recovery.
- Explain the architecture to an interviewer or review board.
36. Production Implementation Blueprint
The learning project should be implemented in two modes.
Mode A — learning mode
Use notebooks and a small deterministic dataset so every transformation can be inspected.
Mode B — production mode
Move reusable logic into Python modules, parameterize environments, execute through Databricks Jobs, and store governed Delta tables in Unity Catalog.
A practical production flow is:
┌──────────────────────────────┐
│ Source systems / file drops │
└──────────────┬───────────────┘
│
ingestion + contract
│
v
┌──────────────────────────────┐
│ Bronze Delta │
│ raw + metadata + replay │
└──────────────┬───────────────┘
│
validation / DQ
│
┌──────────┴──────────┐
│ │
v v
accepted records rejected records
│ │
v v
Silver Quarantine
│
dedup + business rules
│
v
Gold 360
│
┌─────────┼──────────┐
v v v
BI SQL ML/AI
This pattern aligns with current Databricks guidance to use Delta for reliable lakehouse processing, Unity Catalog for governance, and medallion layers for raw, curated, and business-ready data.
37. Data Grain — The Most Important Design Decision
Before writing transformations, document the grain.
| Dataset | Grain |
|---|---|
| Bronze customers | one source record/version |
| Silver customers | one current record per customer |
| Bronze orders | one source order/version |
| Silver orders | one current record per order |
| Customer revenue | one row per customer |
| Latest order | one row per customer |
| Gold Customer 360 | one row per customer |
This prevents a common enterprise failure:
joining datasets with different grains and then trusting the resulting aggregate.
For every transformation, ask:
- What is the input grain?
- What is the output grain?
- Which keys define uniqueness?
- Can this operation multiply rows?
38. Transformation Contract
Each reusable transformation should have a small contract.
Example:
def build_customer_silver(bronze_customers):
"""
Input:
one row per source customer version
Output:
one current row per customer_id
Guarantees:
- customer_id is non-null
- customer_id is unique
- latest updated_at wins
- country and email are normalized
"""
This is more valuable than a notebook full of undocumented cells.
For every function, document:
- input schema
- output schema
- input grain
- output grain
- business rules
- rejected-record behavior
- performance considerations
39. Make the Pipeline Parameterized
Avoid hard-coding environment-specific values.
Example:
dbutils.widgets.text("catalog", "dev")
dbutils.widgets.text("run_date", "")
catalog = dbutils.widgets.get("catalog")
run_date = dbutils.widgets.get("run_date")
bronze_table = f"{catalog}.sales.bronze_orders"
silver_table = f"{catalog}.sales.silver_orders"
gold_table = f"{catalog}.sales.gold_customer360"
The same transformation logic should be deployable to:
dev
↓
test
↓
prod
without editing business logic.
40. Production Readiness Checklist
Before calling the capstone production-ready, verify:
Correctness
- Every table has a documented grain.
- Business keys are defined.
- Revenue rules are explicit.
- Window ordering is deterministic.
- Join cardinality is tested.
Reliability
- Bronze data can be replayed.
- Invalid records are retained.
- The pipeline is idempotent.
- Partial failures can be recovered.
- Late-arriving data has a defined strategy.
Performance
- Required columns are projected early.
- Filters are pushed as early as practical.
- Join strategy is understood.
- Skew is measured rather than guessed.
- Shuffle is measured.
- Before/after benchmarks are recorded.
Governance
- Production tables are governed through Unity Catalog.
- Access is granted to groups/service principals rather than individual developers.
- Sensitive customer fields are identified.
- Lineage and audit requirements are considered.
Operations
- Pipeline metrics are published.
- Failure notifications exist.
- Runbooks exist.
- Rerun behavior is documented.
- Data-quality thresholds are defined.
41. Architecture Review Exercise
Present this project to an architecture review board.
You have five minutes.
Explain:
1. Why Bronze?
Because source history and replayability are valuable.
2. Why Silver?
Because downstream consumers need trusted, typed, deduplicated records.
3. Why Gold?
Because business consumers should not repeatedly reconstruct business logic.
4. Why Delta?
Because transactional storage and reliable batch/stream processing simplify the lakehouse design.
5. Why Unity Catalog?
Because governance, access control, lineage and auditing should be part of the platform rather than bolted onto individual notebooks.
6. Why PySpark rather than pandas?
Because the workload is distributed and must scale beyond driver memory.
7. Why window functions?
Because the business rules require deterministic latest-version and ranking semantics.
8. What would you optimize first?
Measure the physical plan and Spark UI first. Optimize the largest source of data movement or skew rather than blindly increasing compute.
42. Final Capstone Challenge
Do not stop at the reference implementation.
Create a second version with these requirements:
Requirement A
Increase orders to at least 10 million rows.
Requirement B
Create one intentionally skewed customer with a very large number of orders.
Requirement C
Add 20 unused columns to the source.
Requirement D
Introduce duplicate order versions.
Requirement E
Introduce late-arriving orders.
Requirement F
Run the pipeline twice with the same input.
Requirement G
Compare:
baseline
vs
optimized
Record:
- runtime
- input bytes
- shuffle bytes
- spill
- task skew
- output files
- cost estimate
Requirement H
Explain every optimization.
If you cannot explain why the optimized version is faster, the optimization is not yet complete.
43. Current Databricks Design Guidance
For current Databricks implementations, prefer platform-native patterns where they fit the requirement:
- Unity Catalog for governance.
- Managed tables for most new governed table workloads.
- Delta Lake for transactional lakehouse storage where appropriate.
- Medallion layering when separate raw, curated and business-ready boundaries provide value.
- Lakeflow/Jobs for production orchestration as appropriate.
- Auto Loader or Lakeflow Connect for supported ingestion patterns.
- Avoid legacy DBFS mounts as the default storage-access pattern.
See the current Databricks guidance:
- Data engineering with Databricks
- Delta Lake best practices
- Data engineering best practices
- Unity Catalog
- Unity Catalog best practices
- Lakehouse architecture
- DBFS and Unity Catalog best practices
Feature availability and recommended defaults can change with Databricks Runtime and platform releases, so the implementation should always validate the target workspace/runtime before production rollout.
44. Module 04 Exit Gate
You should not move to SQL simply because you finished reading 04.12.
You should move when you can independently:
Read a source
↓
Define schema + grain
↓
Validate data
↓
Transform with PySpark
↓
Control joins/windows/shuffles
↓
Produce trusted Delta data
↓
Measure Spark execution
↓
Optimize with evidence
↓
Design incremental + idempotent processing
↓
Explain governance + operations
That is the real outcome of Module 04.
35. What You Have Learned in Module 04
You started with the DataFrame abstraction in 04.1.
You then learned:
04.1 DataFrames
↓
04.2 Schemas
↓
04.3 Select / Filter
↓
04.4 Aggregations
↓
04.5 Joins
↓
04.6 Windows
↓
04.7 UDFs
↓
04.8 Pandas API on Spark
↓
04.9 Nulls & Data Quality
↓
04.10 Advanced PySpark
↓
04.11 Performance Engineering
↓
04.12 END-TO-END PROJECT
The important transition is:
API knowledge → distributed execution understanding → production engineering.
That is the standard we want throughout the rest of this Databricks course.
Final Architect Takeaway
A PySpark developer asks:
“How do I write this transformation?”
A senior data engineer asks:
“How will this execute at scale?”
An architect asks:
“What is the right data contract, execution model, storage design, reliability model, governance boundary and cost profile for this workload?”
This project is designed to make that transition explicit.
Next module: Databricks SQL — starting with the SQL execution model and SQL Warehouses.