Better logging in Python
10-18, 14:40–15:10 (Europe/Lisbon), Auditorium

You probably know, from experience, that catching the bug is 90% of the work of solving it. But how much thought and effort has gone into your application's logging structure? In this talk, we'll go over Python's stdlib logging library (and possible alternatives), how to configure it, good logging practices (also from a structural/code quality point of view), and finally we'll consider in which cases and for what reasons you'll want to use external service integrations.


There's a lot of talk about observability, reliability, fault tolerance. That seems to be all on the Ops side, though. For the developers, test methodologies and practices get talked about a lot, but logging, which might be even more important, doesn't get talked about enough. You're always gonna have bugs slip through no matter what you do, so ensuring a quick detection and response is worth an insane amount of time, money and trust, from clients and users. In this talk, I want to speak about the built-in logging module (this is something that other languages don't have!), its quirks and possible alternatives (it's certainly not the most ergonomic utility...), its setup, and then how to use it app-wide. When should one define different loggers, in which situations (if any ;)) one would want to include side effects in the logging action, where should the logging configuration be and in which format (it actually does not need to be configured through code, interestingly). Different log levels, what concretely to log and where (in the "bottom" calls, or as up in the main program as possible?) How much logging is too much logging, etc. Logging formats and what things you should always include in your logs no matter what you do (hint: times ;)) and why you'll need that information. Thoughts on how to make it parseable and easily interpretable. Finally, we'll touch upon why you'll want to hook this up to external services and how to handle those cases. Won't be the main point of the talk, as the log-as-a-service space is already packed, it's not nearly as interesting and I'm also not trying to publicize anything (much less proprietary solutions). I'll just mention the existence and use case of those tools (and i.e, in which situations you'll be able to roll your own vs having it as a service).

Expected roadmap:

Introduction (5 minutes) Welcome and set the context: Why is logging important? Touch on the challenges of debugging without proper logs. Time and cost savings, etc.

Python’s Built-in Logging Library (10 minutes) Explore the logging module from the Python standard library. Briefly discuss its features, log levels, and configuration options. Demonstrate basic usage and best practices. Introduce alternative libraries and why you might or might not want to use them.

Advanced Techniques (10 minutes) Log formatting: Customizing log messages. What things to include, what format should your logs use. Handlers and formatters: Choosing the right output channels. Filters: Controlling which logs get processed. Performance: Impact, Async, batching, log queues, etc. Rotating logs: Managing log files over time.

External Service Integrations (5 minutes): Why you'll probably want some centralized source for your logs. Briefly touch on integrating with logging platforms (i.e ELK), alerting platforms, or logs as a service.


Audience Level

Intermediate

What are the main topics of your talk?

logging, internal libraries, best practices, observability

Software engineer with experience in startups, now working as the lead engineer and main technical coordinator for Reckon Digital where he directs all the companies' projects, primarily in collaboration with the UN's World Food Programme (WFP).