Introduction

When starting a Django project, you may decide that using a 32-bit integer field will be sufficient; or you don't even give it a thought. So you use an AutoField for a primary key or an IntegerField for some data. But over time, you may discover that your table is going to grow beyond 2 147 483 647 (2**31 – 1) rows, or your data has values larger than that number. If you had the ability to go back in time, you could change the original field type. Otherwise, you need a way to convert the fields and their underlying database columns from 32-bit to 64-bit. In this talk I'll describe two projects at Kraken Tech where we've done such bigint conversions.

The problem

The obvious way to change a field type in Django is to change the model definition, run makemigrations, and then apply the resulting migration. But that will require a full table rewrite, which could take hours or days (depending on the size of the table); all other access to the table will be blocked while that happens.

One solution is to work at the database level to add a new 64-bit column, backfill it with data from the 32-bit column it will replace, and then switch the columns over. That is the approach that we've used, but using Django migrations with RunSQL, rather than manually running DDL SQL commands.

Money fields

The Kraken software was originally written to support domestic retail energy customers. Representing money as a 32-bit integer number of pence (or cents) seemed reasonable, as balances or payments of over ~£21 million were not expected. But when we needed to support large corporate customers, we needed to increase that limit. That required converting all the money IntegerFields to BigIntegerFields for 29 fields on 14 models. At the time we did this project, Kraken had 12 clients; updating database tables manually one by one was out of the question, so we needed to use Django migrations.

Primary key fields

Django defaults to using an AutoField for primary keys, and Kraken models were originally created with that default. Over time, database tables grew, and the limit of ~2.1 billion rows became an issue on the largest Kraken instances. When we first encountered this issue, these tables had their primary key columns converted to bigint via manually running SQL. That was very labour intensive, so we've adopted a new process that uses Django migrations to apply the SQL, and does the conversion across all Kraken instances.

Conclusion

Avoid issues by starting your Django project with bigint fields. If it's too late for that, use Django migrations to convert fields to bigint. It can be more reliable and less labour-intensive than manually applying DDL SQL commands.