1. Introduction

In 2022, in the aftermath of the COVID pandemic peak, the problem of absences in patient healthcare appointments in primary care was revisited. There was also a need to catch up to the backlog of appointments that were cancelled. In order to maximize attendance, it was decided that patients should get reminders of their scheduled appointments, by text message, as SPMS, the Shared Services of the Ministry of Health had had a great experience with them in the vaccination scheduling effort.

However, the team working on Primary Care was still fully committed to the vaccination scheduling system and other tasks. The deadline was also short, basically "As soon as possible". And I was the only one available to do it.

Who am I? Well, I'm a doctor, a family doctor to be more exact. A year and a half ago from starting this project, I was treating COVID patients in full protection suit. And as of that moment, I was a scientific consultant for Health apps. Doesn't mean I didn't spend a lot of time coding, as I had already some deployed internal apps, including a FastAPI service for your first vaccination schedule date.

1. Technical challenges

The task sounds simple when thinking about modern systems. Collect the upcoming appointments from the Electronic Health Record (EHR), and use a service to send the messages. However, the Portuguese primary care EHR is fragmented into multiple instances that serve a region. And the text message service used had to be a SOAP service. Given the timeframe and the fact that I still had to deliver on my other assignments, some compromises had to be made.

1. Shortcuts

The first and biggest shortcut to take has to be data gathering. I have to collect the data directly from the databases, as there was no time for anything else. This couples my app to the EHR data structure but it's a compromise we are willing to make.

I also can’t get an actual database for my app in a short time, but I can settle for Redis. It's fast to develop with Key Value databases, it won't be a bottleneck and I can log message sending successes or failures (if there is a need to retry).

Configuration can be made from environment variables.

And we'll host it all on a machine we already have, using a pipeline we're familiar with... that turned out to be kubernetes K3s.

Our enterprise ecosystem is made up mostly from Java apps, with legacy systems using PL-SQL. So of course I had to go with python.

Why python? Because we can interact with oracle databases with a robust package (cx-oracle, now oracle-db), use a SOAP service easily with Zeep, schedule tasks with Celery, create an API to access information with FastAPI, and ensure someone else can take over really fast if we fall ill with COVID. And we can do it in a week if we have to.

1. Choose well the corners you cut

Don’t cut testing. At some point you start thinking that this thing you're building will actually cost money. And if we send the wrong message, we can cost someone their appointment or worse, leak personal information. There are a lot of things that can go wrong. So we must make it bullet proof. That meant creating tests for the integrations that might fail, and all of the logic. And that allowed constant improvements of the not so good code to something that has been going non stop for a year.

Deployment. I can either deploy extremely fast today with a manual deployment and then suffer for eternity... or I can do it well, not perfectly but well enough. It's especially good when you can reuse tried and tested pipelines, so there is a compromise to be made here.
It can't fail, it can't wait for too long but it must also mean it's easy to redeploy new versions as often as needed.

1. Things you can't avoid

Everything in a large corporation takes longer than you wish. Want credentials to a service? Request, sign, wait. Access to databases? Wait for grants. Wait some more for firewall access. Clarification on the actual text of the message? Take a number. A pandemic is still happening and we need everything to both wait and happen simultaneously.
Everything feels both expedited and extremely slow in the corporate world. It's demoralising sometimes because it contrasts with the extreme speed with which you're developing with the snail pace of getting someone to stop what they're doing and solve a ticket for you. It goes with the privilege of serving a whole country. It pays off, but it takes time.

1. What happens the day after?

Well, after you deliver that extremely urgent project, you can actually start to make it good. The pilot is only sending a hundred messages, but to send thousands, you must use async or threads. But you can't just push everything at once, you'll crash the message service.
And the code is still terrible so it's time to refactor, because soon enough people will start to think of new things to put into the service. If everything is coupled, implementing new rules will start making big messes. If you're afraid to make changes, it's because you can't trust your tests to protect you.

The key takeaway is, every time you make a new feature, leave the code better than you found it. Everything can't be an emergency every time. This certainly wasn't. At some point you have to say "I can't do that in that amount of time". Compromises can be reached, but they must always include tech debt repayment warranties.

1. How's it going?

I'm fine, thank you. And the service is fine too!
The service is still only available to about half of the population, mostly in Lisbon. But as of the 21st of June 2023, we have sent more than five million messages. Some days we send more than 70 000. Not too bad for a service made by a doctor.