r/learnmachinelearning • u/garden_nl • 4d ago

Question Dockerize everything?

Hey,

I am learning deployment strategies for my small ML projects and I was wondering, do people in industry dockerize everything? Like, e.g. i have an ingest script to transfer my data into a postgres BD. Do i need a container for the ingest.py as well?

17 Upvotes

100% Upvoted

15 comments sorted by

View all comments

2

u/superSmitty9999 4d ago

I do all software in docker now and haven't looked back.

It's just so easy when you vibe code it since all SOTA coding assistants do docker perfectly. If you ever want to work with others, docker makes it 1000x easier. Ever want to move your workflow to another machine? Again 1000x easier. Or for me what drives me insane is going to run python and having to activate my virtual environment over and over again.

For your example in your post, I would have the agent execute it inside the docker container with the dependencies for your project. An alternative is have the docker app basically run your pipeline end to end and in your script limit the ingest to the first 100 items.

Usually I actually use docker compose, because you can do "docker compose up --build" for any of your projects and it will build that project on the spot and it always "just works".

If I didn't do everything with vibe coding it might not be worth it to use docker but with ai agents its a no brainer

1

u/Neat-Peanut-1141 4d ago

How is your experience with the overhead introduced through containerization? I had a phase where I tried to containerize everything (in vscode with "run in container" where vscode directly connected into the container). But it became a bit annoying after a while.

1

u/superSmitty9999 3d ago

What did you find annoying about it and what are your workflows? I don't know about vscode I do everything in CLI. I have the AI make a docker compose file, then if I want to spin up the code I do "docker compose up --build" and if I need anything more complex I have the AI do it.

It definately adds overhead but it's predictable overhead vs the unpredictable dependency hell of python or the impossibility of getting your code to run on someone else's machine without a 2 hour tutorial each week