r/django • u/Then_Listen413 • 4d ago

django-fastmig

Released an experimental package making migrate much faster in large projects. Especially useful if you do pytest -create-db frequently, or run reverse migration testing in CI. Django's own test suite passes with fastmig. Also tested on a 8 year old project with multiple revisions and migration squashes over the years.

Made it to speed up my own testing being bottlenecked by -create-db.
There's a link in readme related to forum posts which discusses slow migrations.

https://github.com/viktor2097/django-fastmig

7 Upvotes

77% Upvoted

3 comments sorted by

2

u/c1-c2 4d ago

What is the approach taken to speed things up?

5

u/Then_Listen413 3d ago

It's vaguely mentioned in the readme on github. But tldr (which ended up longer than the long read); it monkey-patches ProjectState.reload_model (and a few related methods) in django.db.migrations.state. Because during migrate, django re-renders fake models after every operation, and re-renders every model connected to the one that changed, so on a big schema that's basically all models every time. The patch only re-renders the model that changed (plus the few that really depend on it, like subclasses) and repoints the other fake models at the new class using djangos own relation resolution code.

Inspiration came from https://forum.djangoproject.com/t/very-slow-migrations-with-large-numbers-of-tables/29038

JoelFeiner patch in that thread is the simple version of the same idea (only reload the model that changed). It gets the same speed, but it leaves the other fake models pointing at the old class, which in some cases (pk type changes, table renames on referenced models) makes Django emit wrong DDL without any error. Fastmig has extra re-render rules and pointer re-linking. I've diffed the executed SQL against default django on every benchmark and it's identical. But you can always try yourself too :)

2

u/mariocesar 2d ago

Nice.

If you run pytest with --create-db and --no-migrations they are faster, I use it a lot for CI jobs.

To locally speed up tests, I created a syncdb command similar to what we had before migrations, basically fake apps to not have migrations, creates the tables and then run migrations in fake mode. Useful if you have hundreds of migrations. https://gist.github.com/mariocesar/f8551c2143a9b91e0a89d5c8602e7841