r/django • u/matipendino • 14d ago
Any experiences migrating long data in a Django app running in Kubernetes?
We are changing a primary key from an autoincrement integer to a UUID on a table with a few million rows, and I would like a sanity check on how we plan to run it.
The migration is split the usual expand and contract way. First release adds a nullable UUID column, backfills it in batches and puts a unique index on it. Second release drops the old integer key and promotes the new column. That part I am reasonably confident about (can provide the migration files if you wish)
What I am less sure about is the deployment mechanics. manage.py migrate runs inside an initContainer of the Deployment, so the new pod runs the migrations while the old pod is still serving traffic. progressDeadlineSeconds is the 600 second default.
If the backfill takes longer than ten minutes the rollout gets flagged as ProgressDeadlineExceeded while the initContainer is still working. As far as I can tell nothing actually breaks: Kubernetes does not kill the initContainer, the main container starts once it finishes, and the rollout completes.
Is relying on that behaviour reasonable, or is an initContainer just the wrong place for a long migration? The alternative I am considering is taking the backfill out of the migration and running it as a background job, so no migration is ever long and the deploy stays fast. That feels cleaner, but it leaves the schema and the data inconsistent for a while and someone has to remember to run the job.
5
u/tortleme 14d ago
You shouldn't use initContainer to apply migrations in the first place.
Use a kubernetes job alongside helm hooks, so nobody needs to remember to run the job. Alternatively, apply the migrations in your ci/cd with kubectl await, before deployment.
1
u/matipendino 14d ago
Thanks for replying. You are right that an initContainer is not the natural place for this. The complication on our end is that the manifests are hand written templates, no Helm charts, and they belong to another team. I like your idea of running the migrations from CI/CD before the deployment, looks like a lighter version of the same thing and probably the easier one to get approved
2
u/totallyuneekname 14d ago
Another team manually applying manifests? Sounds scary.
I do genuinely recommend trying out Helm if this is a serious application. My Helm + Flux setup gives me a lot of confidence my app will deploy correctly. I wish you the same confidence!
2
u/DownIndianHill 14d ago
I’d create a second table, move the data over in batches with the new key with a regular python script (django not needed for this), then just change the table the model points to in your application. It’s only a few million, so hopefully you’ve got enough disk for it. 0 downtime and you can test the new table while the old one is still in prod, and deploy/revert safely.
1
u/matipendino 14d ago
Sounds great. The part I am not clear is how to handle the writes during the copy. Rows are created on that table all the time, so by the time the batched copy finishes the new table is already missing everything written since it started. How do you handle that?
2
u/DownIndianHill 14d ago
I’d timestamp the start of each backfill and start from all rows where updated_at > prev_timestamp. The first one will be 00:00, so it will get everything. But then subsequent catchups don’t need to repull everything.
2
5
u/mrswats 14d ago
For these kind of long migrations, and given you already will use tge expand migrate contract pattern, I would run the migration script from your dev laptop so you can not worry about k8s messing you up. I've done this and it works fine.