04.1 PySpark DataFrames

A PySpark DataFrame is a distributed collection of data organized into named columns. It provides a structured API over Spark's execution engine.

1. What is a PySpark DataFrame?\n\ntext\nPython DataFrame API → Logical plan → Optimized physical plan → Distributed execution → Partitions → Tasks → Executors\n\n\nA DataFrame is not a local Python list and does not mean all records are loaded into driver memory.\n\n## 2. Creating a DataFrame

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

data = [
    (1, "Alice", 1000),
    (2, "Bob", 1500),
]

df = spark.createDataFrame(data, ["customer_id", "name", "revenue"])
df.show()

3. Inspecting a DataFrame

df.printSchema()
df.columns
df.count()
df.show(10, truncate=False)

4. Why DataFrames?

DataFrames allow Spark to understand the structure of the data and optimize execution using Catalyst and the Spark SQL engine. \n\n## 6. DataFrame vs pandas\n\n| Capability | PySpark DataFrame | pandas DataFrame |\n|---|---|---|\n| Execution | Distributed | Local |\n| Lazy transformations | Yes | Mostly eager |\n| Partitions | Yes | No distributed partitioning |\n| Typical use | Data engineering / analytics | Local analysis |\n\ntoPandas() collects data to the driver; use it only for appropriately small results.\n\n## 7. Execution-plan mindset\n\nUse df.explain("formatted") to inspect scans, projections, filters, exchanges, joins, and other physical operators. Connect the plan to Spark UI runtime evidence when tuning.\n\n## 8. Core DataFrame operations\n\npython\nfrom pyspark.sql import functions as F\n\nselected = df.select("customer_id", "revenue")\nfiltered = df.filter(F.col("revenue") > 1500)\nenriched = df.withColumn("revenue_k", F.col("revenue") / 1000)\nrenamed = df.withColumnRenamed("customer_name", "customer")\ndropped = df.drop("temporary_flag")\n\n\nUse &, |, and ~ for Spark boolean expressions. DataFrame transformations return new DataFrame references rather than mutating the original.\n\n## 9. Lazy evaluation\n\npython\nresult = df.filter(F.col("revenue") > 1500).select("customer_id", "revenue")\nresult.show() # action triggers execution\n\n\nThe transformation chain describes computation; the action causes Spark to execute the required work.\n\n## 10. DataFrame and SQL\n\nThe same structured logic can be expressed through PySpark or Spark SQL. Choose the interface that best supports maintainability, testing, review, and governance.\n\n## 11. Reading and writing\n\npython\ndf = spark.read.format("delta").load("/path/to/delta")\n\ndf.write.format("delta").mode("append").saveAsTable("main.silver.orders")\n\n\nCommon write modes include append, overwrite, errorifexists, and ignore. Understand their production consequences.\n\n## 12. Production Bronze → Silver example\n\npython\nbronze = spark.table("main.bronze.orders")\n\nsilver = (\n bronze\n .select("order_id", "customer_id", "order_date", "amount", "status")\n .filter(F.col("order_id").isNotNull())\n .withColumn("amount", F.col("amount").cast("decimal(18,2)"))\n .withColumn("order_date", F.to_date("order_date"))\n)\n\nsilver.write.format("delta").mode("overwrite").saveAsTable("main.silver.orders")\n\n\nDesign pattern: read governed source → prune columns → validate keys → normalize types → persist clean Silver data.\n\n## 13. Anti-patterns\n\n- Large collect() or toPandas() calls.\n- Carrying unnecessary columns.\n- Blind schema inference in production.\n- Repeated expensive actions without understanding their cost.\n\n## 14. Debugging checklist\n\nFor correctness: inspect schema, samples, nulls, casts, filters, join keys, plan, and business-rule counts.\n\nFor performance: check input volume, columns, Exchange/shuffle, partition distribution, joins, spills/skew, and Spark UI metrics.\n\n## 15. Hands-on lab\n\npython\norders = [\n (101, 1, "2026-01-01", 120.0),\n (102, 1, "2026-01-02", 250.0),\n (103, 2, "2026-01-02", 80.0),\n (104, 3, "2026-01-03", 500.0),\n]\n\ndf = spark.createDataFrame(orders, ["order_id", "customer_id", "order_date", "amount"])\n\n\nBuild a result that converts the date, removes invalid customer IDs, creates amount_with_tax, filters orders above 100, and keeps only required columns. Then run result.explain("formatted") and result.show().\n\nPredict before running: which operations are narrow, whether a shuffle is required, what triggers execution, and which columns remain in the plan.\n\n## 16. Architect perspective\n\nA production DataFrame should have explicit answers for schema contract, business semantics, scale, execution cost, governance, reliability, observability, and testability.\n\n## 17. Interview questions\n\n1. What is a PySpark DataFrame?\n2. How does it differ from pandas?\n3. Why are transformations lazy?\n4. How do you inspect a physical plan?\n5. Why avoid collect() for large data?\n6. What is column pruning?\n7. repartition() vs coalesce()?\n8. How can a DataFrame transformation introduce a shuffle?\n9. How would you structure a Bronze-to-Silver DataFrame pipeline?\n10. How would you diagnose a pipeline that suddenly becomes 5× slower?\n\n## 18. Cheat sheet\n\npython\ndf.printSchema()\ndf.show()\ndf.select("id", "amount")\ndf.filter(F.col("amount") > 100)\ndf.withColumn("tax", F.col("amount") * 0.18)\ndf.withColumnRenamed("id", "customer_id")\ndf.drop("temporary_column")\ndf.explain("formatted")\ndf.rdd.getNumPartitions()\ndf.write.format("delta").mode("append").saveAsTable("main.silver.orders")\n\n\n## 19. Key takeaways\n\n1. A DataFrame is distributed and schema-aware.\n2. Transformations are lazy; actions trigger execution.\n3. The API describes computation; Spark chooses execution.\n4. Physical cost depends on partitions, data movement, and operators.\n5. Production PySpark means correctness, performance, governance, and observability.\n\n### Next\n\n04.2 PySpark Schemas\n\nWe will cover explicit schemas, primitive and complex data types, nullability, nested structures, safe casting, schema evolution, and production schema contracts.\n\n### Official references\n\n- Apache Spark — PySpark DataFrame API\n- Apache Spark — SQL Programming Guide\n- Apache Spark — Performance Tuning