Have you ever looked at a PySpark pipeline and assumed it would run top-to-bottom? You’d be surprised…it doesn’t. In reality, Spark builds and optimizes execution plans, and this mismatch can lead to subtle, production-grade bugs.
In this talk, I’ll walk through a real-world example where reading and updating the same tables produced inconsistent results in production. We’ll see why this happens, what Spark actually executes under the hood, and why code that looks correct can behave unexpectedly.
Join this session and you’ll learn practical design principles to make pipelines predictable: introducing explicit materializations, using declarative patterns, or imposing physical separation between sources and targets.
PySpark code often looks like it runs sequentially, but Spark does not execute pipelines line by line. Instead, it builds and optimizes execution plans, and this mismatch can lead to bugs and misunderstandings.
In this talk, I will present a real-world issue involving a pipeline that reads from and writes to the same tables. While the code appears straightforward, the final action triggers a separate execution plan that re-reads one of the inputs after it has been updated, producing inconsistent results. This behavior is counterintuitive because dataframes are logical abstractions: they do not hold data unless explicitly materialized, and each action can recompute the entire pipeline.
Starting from this example, I will explore some design principles that can systematically avoid this kind of issue. First, one can include explicit materializations after reading from sources, in order to save input data in a well-defined state. Another possibility is designing simple, atomic jobs with a single target table or even a single action, and then combine them sequentially by means of an orchestrator. Finally, if the underlying data platform has multiple layers, one can impose that every job reads from one layer and writes to the next, in order to physically separate sources and targets.
The talk will include code examples and is aimed at data engineers with some experience in PySpark who want to build cleaner and more reliable pipelines.
PySpark, data pipelines, execution plans