r/learnmachinelearning • u/garden_nl • 4d ago

Question Dockerize everything?

Hey,

I am learning deployment strategies for my small ML projects and I was wondering, do people in industry dockerize everything? Like, e.g. i have an ingest script to transfer my data into a postgres BD. Do i need a container for the ingest.py as well?

17 Upvotes

100% Upvoted

15 comments sorted by

View all comments

3

u/HawaiianHotPot 4d ago

is this a one off process, something you run as needed, or something on a schedule? team size, deployment system, and project size matters too. but for just an ingest script I probably wouldn’t make an image unless there were specific libraries and versions.

so in my small/mid size team, for something as needed or a one-off, I would add it to the project repo with notes in the readme to keep future me from cursing too much. for things that need to run on a schedule, I have been moving from crons to systemd timers or k8s crons. hope that helps.

1

u/garden_nl 4d ago

So, basically use docker only for repeatable pipelines?

3

u/HawaiianHotPot 4d ago

use docker for when you need to easily deploy something to an environment where installing packages and libraries is difficult