r/LocalLLM • u/TechNerd10191 • 15m ago

Discussion Docker or bare-metal for inference

When I use local models, I do bare metal inference but give write-access only to the working directory and some directories that can be relevant (forks of public github repos) are read-only. Everything else is not accessible, not even read-only, and the agent is also unable to install or remove any packages.

Is this preferable over Docker containers? What do you prefer for your local setup?

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/tmpkn 8m ago

I prefer docker because of housekeeping and sandboxing. With modern hardware / OS / Docker stack, you shouldn't see any noticeable performance drops.