04.2 PySpark Schemas & Data Types

Schemas define the structure and data types of DataFrame columns.

1. Explicit schema

from pyspark.sql.types import (
    StructType, StructField, IntegerType, StringType, DecimalType
)

schema = StructType([
    StructField("customer_id", IntegerType(), False),
    StructField("name", StringType(), True),
    StructField("revenue", DecimalType(18, 2), True),
])

df = spark.createDataFrame(
    [(1, "Alice", 1000.50)],
    schema
)

2. Why explicit schemas matter

Inference is convenient for exploration but production ingestion should normally make data types intentional.

Benefits:

• Predictable pipelines.
• Better validation.
• Fewer accidental type changes.
• Clear contracts between ingestion and transformation.
• Easier schema evolution.

3. Common types

String, Boolean, Byte, Short, Integer, Long, Float, Double, Decimal, Date, Timestamp, Array, Map, and Struct are common Spark SQL types.

4. Architect rule

Treat a production schema as a data contract, not merely metadata.

5. Production schema model

A production schema should explicitly define field names, data types, nullability and, where useful, metadata. Use StructType and StructField for stable contracts. Complex enterprise data commonly uses ArrayType, MapType, and nested StructType. citeturn583419search0turn583419search4

6. Schema anatomy

StructType
 ├── StructField(name, dataType, nullable, metadata)
 ├── StructField(...)
 └── ...

Data type and nullability answer different questions: what is the value? and can it be NULL?

7. Explicit schema

from pyspark.sql.types import (
    StructType, StructField,
    IntegerType, StringType, DecimalType
)

schema = StructType([
    StructField("customer_id", IntegerType(), False),
    StructField("name", StringType(), True),
    StructField("revenue", DecimalType(18, 2), True),
])

df = spark.createDataFrame(
    [(1, "Alice", 1000.50)],
    schema=schema
)

StructField contains the field name, data type, nullability and optional metadata. citeturn583419search4

8. Primitive data types

Type	Typical use
BooleanType	flags
IntegerType	32-bit integers
LongType	large IDs/counts
DoubleType	floating point
DecimalType(p,s)	exact financial/numeric values
StringType	text
DateType	business dates
TimestampType	timestamps
TimestampNTZType	timestamps without timezone
BinaryType	binary data

Spark's current PySpark type catalog also includes CharType, VarcharType, interval types, VariantType, and spatial types. citeturn583419search3

Financial values

Prefer an appropriate DecimalType when exact decimal semantics matter. Do not default every monetary amount to DoubleType.

9. Complex types

Array

ArrayType(StringType(), containsNull=False)

Map

MapType(StringType(), StringType(), valueContainsNull=True)

Struct

StructType([
    StructField("city", StringType(), True),
    StructField("country", StringType(), True),
])

These are first-class Spark SQL types. citeturn583419search0turn583419search3

10. Nested enterprise schema

schema = StructType([
    StructField("customer_id", LongType(), False),
    StructField("name", StringType(), True),
    StructField(
        "address",
        StructType([
            StructField("city", StringType(), True),
            StructField("country", StringType(), True),
        ]),
        True
    ),
    StructField(
        "skills",
        ArrayType(StringType(), containsNull=False),
        True
    )
])

Conceptually:

customer_id
address
 ├── city
 └── country
skills[]
 └── element

11. Nullability

StructField("customer_id", LongType(), nullable=False)

For arrays:

ArrayType(StringType(), containsNull=False)

The first controls whether the field is nullable; the second controls whether array elements can be null. Map values have an analogous valueContainsNull property. citeturn583419search0

Do not confuse schema nullability with business-quality validation. Critical keys should still be validated explicitly at ingestion and transformation boundaries.

12. Inspecting and serializing schemas

df.printSchema()
df.schema
df.dtypes
df.columns

df.schema.simpleString()
df.schema.json()
df.schema.jsonValue()

StructType supports field access by name or position and machine-readable JSON representations. citeturn583419search0

13. DDL schema definitions

Current PySpark also supports DataType.fromDDL() for DDL-formatted type definitions. citeturn583419search4

from pyspark.sql.types import DataType

schema = DataType.fromDDL(
    "customer_id int, name string, revenue decimal(18,2)"
)

This can be useful when schemas are stored as configuration or metadata.

14. Casting and temporal types

from pyspark.sql import functions as F

clean = (
    df
    .withColumn("customer_id", F.col("customer_id").cast("long"))
    .withColumn("order_date", F.to_date("order_date"))
    .withColumn("created_at", F.to_timestamp("created_at"))
)

Before casting, ask:

1. What is the source type?
2. What is the target type?
3. Can every value be converted?
4. What happens to malformed values?
5. Do rejected records need quarantine?

A successful cast is not the same thing as valid business data.

15. Decimal precision and scale

For DecimalType(p,s):

• p = total digits
• s = digits after the decimal point

For example, DecimalType(18,2) allows 18 total digits with 2 decimal places. Choose precision and scale from business requirements and test rounding/overflow behavior.

16. Date vs timestamp

Prefer semantic types instead of storing all temporal fields as strings:

Business date   → DATE
Event time      → TIMESTAMP / TIMESTAMP_NTZ
Free-form text  → STRING

The timestamp choice should reflect source and business timezone semantics.

17. Inference vs explicit schema

Inference is useful for exploration:

df = spark.read.option("inferSchema", True).csv(path)

Production ingestion often benefits from explicit contracts:

df = (
    spark.read
    .schema(schema)
    .option("header", True)
    .csv(path)
)

Do not use the simplistic rule “never infer.” Use inference when discovery is the goal and explicit contracts when predictability is the goal.

18. Schema validation

required = {
    "order_id",
    "customer_id",
    "amount",
    "order_date"
}

missing = required - set(df.columns)

if missing:
    raise ValueError(f"Missing required columns: {sorted(missing)}")

For enterprise frameworks, validate field names and expected data types, then define a clear failure or quarantine path.

19. Schema drift and evolution

Schema drift means an upstream source changes structure or types.

Expected schema
      ↓
   Validate
      ↓
 ┌────┴─────┐
Pass       Drift
 ↓           ↓
Transform   Quarantine / Reject / Evolve

Typical changes include:

• New columns
• Removed columns
• Type changes
• Renames
• Nested-structure changes

Schema evolution should be intentional, tested, and governed. Do not let every source change silently become a production contract change.

20. Medallion schema contracts

Layer	Schema philosophy
Bronze	Preserve source fidelity; tolerate controlled variation
Silver	Strong validated business schema
Gold	Stable analytical contract
Semantic/BI	Consumer-oriented model

The farther downstream data travels, the stronger its contract generally becomes.

21. SAP-style example

Suppose an ERP source provides:

MANDT  VBELN  KUNNR  ERDAT  NETWR  WAERK

A Silver contract could map this to:

silver_schema = StructType([
    StructField("client", StringType(), False),
    StructField("sales_order_id", StringType(), False),
    StructField("customer_id", StringType(), True),
    StructField("order_date", DateType(), True),
    StructField("net_value", DecimalType(18, 2), True),
    StructField("currency", StringType(), True),
])

This separates source representation from the enterprise analytical contract.

22. Hands-on lab — explicit schema

Create a schema with order_id, customer_id, amount, and order_date. Load sample records and inspect printSchema() and show().

Then deliberately introduce:

• Missing field
• Wrong type
• Null customer ID
• Malformed date
• Unexpected decimal scale

For each case decide: reject, quarantine, coerce, or accept?

23. Hands-on lab — nested data

nested_schema = StructType([
    StructField("customer_id", LongType(), False),
    StructField(
        "address",
        StructType([
            StructField("city", StringType(), True),
            StructField("country", StringType(), True),
        ]),
        True
    ),
    StructField(
        "skills",
        ArrayType(StringType(), containsNull=False),
        True
    )
])

Practice:

df.select("address.city")
df.select(F.col("address.country"))

24. Architect decision framework

When defining a production schema, ask:

1. Who owns the source field?
2. What does the field mean?
3. What is the narrowest appropriate type?
4. Is NULL semantically valid?
5. Is exact numeric representation required?
6. What are the timezone semantics?
7. Should the structure remain nested?
8. What compatibility guarantees exist?
9. Which changes are allowed?
10. Who owns the contract?
11. What happens when validation fails?
12. How is drift observed?

25. Common mistakes

• Everything as STRING.
• Everything numeric as DOUBLE.
• Ignoring nullability.
• Treating inference as governance.
• Accepting schema drift blindly.
• Using timestamps without understanding timezone semantics.

26. Interview questions

1. What is StructType?
2. What is StructField?
3. What does nullable mean?
4. Array vs Map vs Struct?
5. Why use DecimalType?
6. Explicit schema vs inference?
7. How would you detect schema drift?
8. How would you handle malformed records?
9. How would you design Bronze/Silver/Gold schema contracts?
10. How would you evolve a schema without breaking consumers?
11. How would you define an enterprise data contract?
12. How would you integrate an SAP source whose representation changes over time?

27. Cheat sheet

from pyspark.sql.types import (
    StructType, StructField,
    StringType, LongType,
    DecimalType, DateType,
    ArrayType, MapType
)

schema = StructType([
    StructField("id", LongType(), False),
    StructField("name", StringType(), True),
    StructField("amount", DecimalType(18, 2), True),
    StructField("order_date", DateType(), True),
    StructField("tags", ArrayType(StringType(), containsNull=False), True),
])

df.printSchema()
df.schema.simpleString()
df.schema.json()

28. Key takeaways

1. A schema is a data contract.
2. Data type and nullability are separate concepts.
3. Explicit schemas improve production predictability.
4. Complex types are essential for enterprise semi-structured data.
5. Decimal, date and timestamp semantics must be deliberate.
6. Schema drift needs a defined response.
7. Schema evolution is a governance decision, not merely a technical switch.

Next

04.3 Select / Filter

We will go deeper into Spark column expressions, predicate pushdown, projection pruning, null-safe filtering, complex conditions, expression composition, and performance implications.

Official references

• PySpark Data Types
• PySpark StructType
• PySpark StructField