Pier Paolo La Pastina
I developed a passion for mathematics at an early age, which naturally led me to feel comfortable working with data and abstract thinking. I earned a PhD in Mathematics and later worked as an applied researcher at a startup, where I also developed strong skills in writing technical papers and speaking at international conferences.
I have been working as a data engineer for the past four years. Over this time, I have gained strong hands-on experience in several key areas of the field, including ETL, data warehousing, data quality, data lineage, data modeling, and master data management. I have designed, built, maintained, and optimized large-scale data pipelines using Python, Scala, Java, SQL, and several data processing engines, including Apache Spark. In my daily work, I take pride in writing clear and maintainable code, and I strongly support the continuous improvement of both technical and people processes.
Session
Have you ever looked at a PySpark pipeline and assumed it would run top-to-bottom? You’d be surprised…it doesn’t. In reality, Spark builds and optimizes execution plans, and this mismatch can lead to subtle, production-grade bugs.
In this talk, I’ll walk through a real-world example where reading and updating the same tables produced inconsistent results in production. We’ll see why this happens, what Spark actually executes under the hood, and why code that looks correct can behave unexpectedly.
Join this session and you’ll learn practical design principles to make pipelines predictable: introducing explicit materializations, using declarative patterns, or imposing physical separation between sources and targets.