PySpark code often looks like it runs sequentially, but Spark does not execute pipelines line by line. Instead, it builds and optimizes execution plans, and this mismatch can lead to bugs and misunderstandings.

In this talk, I will present a real-world issue involving a pipeline that reads from and writes to the same tables. While the code appears straightforward, the final action triggers a separate execution plan that re-reads one of the inputs after it has been updated, producing inconsistent results. This behavior is counterintuitive because dataframes are logical abstractions: they do not hold data unless explicitly materialized, and each action can recompute the entire pipeline.

Starting from this example, I will explore some design principles that can systematically avoid this kind of issue. First, one can include explicit materializations after reading from sources, in order to save input data in a well-defined state. Another possibility is designing simple, atomic jobs with a single target table or even a single action, and then combine them sequentially by means of an orchestrator. Finally, if the underlying data platform has multiple layers, one can impose that every job reads from one layer and writes to the next, in order to physically separate sources and targets.

The talk will include code examples and is aimed at data engineers with some experience in PySpark who want to build cleaner and more reliable pipelines.