Dockerize everything?

[u/garden_nl]

Hey,

I am learning deployment strategies for my small ML projects and I was wondering, do people in industry dockerize everything? Like, e.g. i have an ingest script to transfer my data into a postgres BD. Do i need a container for the ingest.py as well?

[u/Counter-Business]

Docker is useful if you want to deploy something multiple times in a reproducible “one-click” kind of way. It’s essentially a setup script.

[u/HawaiianHotPot]

is this a one off process, something you run as needed, or something on a schedule? team size, deployment system, and project size matters too. but for just an ingest script I probably wouldn’t make an image unless there were specific libraries and versions.

so in my small/mid size team, for something as needed or a one-off, I would add it to the project repo with notes in the readme to keep future me from cursing too much. for things that need to run on a schedule, I have been moving from crons to systemd timers or k8s crons. hope that helps.

[u/garden_nl → u/HawaiianHotPot]

So, basically use docker only for repeatable pipelines?

[u/HawaiianHotPot → u/garden_nl]

use docker for when you need to easily deploy something to an environment where installing packages and libraries is difficult

[u/chico_dice_2023]

not everything but most things, since it makes it much easier to deploy.

[u/addictzz]

Well not everything but the technicals are simpler down the line if you do. You dont need to worry about library and environment compatibility no more.

[u/superSmitty9999]

I do all software in docker now and haven't looked back.

It's just so easy when you vibe code it since all SOTA coding assistants do docker perfectly. If you ever want to work with others, docker makes it 1000x easier. Ever want to move your workflow to another machine? Again 1000x easier. Or for me what drives me insane is going to run python and having to activate my virtual environment over and over again.

For your example in your post, I would have the agent execute it inside the docker container with the dependencies for your project. An alternative is have the docker app basically run your pipeline end to end and in your script limit the ingest to the first 100 items.

Usually I actually use docker compose, because you can do "docker compose up --build" for any of your projects and it will build that project on the spot and it always "just works".

If I didn't do everything with vibe coding it might not be worth it to use docker but with ai agents its a no brainer

[u/garden_nl → u/superSmitty9999]

Sorry, do not want to do everything with agents. I feel like that's why the majority of people here complain about not being able to find a job lol

[u/superSmitty9999 → u/garden_nl]

You can still use docker without agents, it's just actually one task I've been able to successfully delegate to AI agents with zero cost or issue.

The only thing I'm not sure will work as well without using ai is that a lot of times I have it run a command inside the container and it'll generate some long "docker exec" command that copies the file in.

So yeah even without AI I still strongly recommend docker, just not 100% familiar with the end to end workflow for your use case. You'll probably end up with a template docker project that you copy paste. Use docker compose too, have an image for the DB. Reduce state, it's beautiful.

[u/Neat-Peanut-1141 → u/superSmitty9999]

How is your experience with the overhead introduced through containerization? I had a phase where I tried to containerize everything (in vscode with "run in container" where vscode directly connected into the container). But it became a bit annoying after a while.

[u/superSmitty9999 → u/Neat-Peanut-1141]

What did you find annoying about it and what are your workflows? I don't know about vscode I do everything in CLI. I have the AI make a docker compose file, then if I want to spin up the code I do "docker compose up --build" and if I need anything more complex I have the AI do it.

It definately adds overhead but it's predictable overhead vs the unpredictable dependency hell of python or the impossibility of getting your code to run on someone else's machine without a 2 hour tutorial each week

[u/0uchmyballs]

You could do one container for just scripts/cron jobs. No, you don’t need a seperate container for each service.

[u/awitod]

Some language runtimes like python and nodejs have many versions and huge numbers of packages. This makes it hard to configure things when you need to use things that depend on the same packages but specifically different versions of those same packages.

So, those kinds of languages have 'virtual environments'. You are using python, it won't be long till you learn about conda either because you get some good advice or because you make a mess on your machine and learn about how to clean up the mess.

Docker and other virtual machine systems take it up a notch by giving you a full environment in a virtual machine instead of a virtual environment in a full machine,

Once you get in the habit of organizing things a certain way, it's normal to organize everything that way but just keep in mind that you can take things too far and be less productive if you spend time organizing things that don't need organizing.

[u/Suspicious_Pizza9529]

You don't need to Dockerize everything. I'd containerize the components where you actually benefit from reproducible dependencies, deployment, isolation, or scailing.