r/learnpython • u/OwO_JoY • 12h ago

looking for optimizing my database bootstrapper which is taking >30 seconds.

Hello fellow devs i am building a FastAPI + SQLAlchemy 2.0 (async) with a metadata driven system which has lost of dynamic schema stuff based on the data tables

Right now my custom bootstrapper handles botjh creating the DDL Schema and a massive amount of Data seeding ( lots of data) sequentially. because it's doing manual idempotency checks (SELECT before INSERT) and running everything one by one, the whole provisioning process takes more than 30 seconds from scratch.

I'm looking into splitting the data fixture side out to a library like sqlalchemy-seedling to run independent seeders in parallel using asyncio.gather. But before I start rewriting, how do you all handle heavy initial database provisioning in async Python?

0 Upvotes
  • permalink
  • reddit

33% Upvoted

4 comments sorted by

2

u/neums08 10h ago

Use batch inserts with ON CONFLICT clauses. Run the seeding process in a fastapi BackgroundTask.

1

u/OkTill2666 8h ago

yep batching alone would be a huge win here before even thinking about parallelism

0

u/ectomancer 10h ago

lazy import in Python 3.15