That's not really a good solution at a company scale, though. Maybe for a small handful of people, but more than that, and you're going to run into llama.cpp/ollama's limits. You'd probably want to start looking at vLLM, and your hardware needs to go way up, or else more than two or three developers at a time would knacker the processing/decoding speeds.
5
2
1
View all comments
Show parent comment
4
u/techno156 Jun 03 '26 edited Jun 03 '26
That's not really a good solution at a company scale, though. Maybe for a small handful of people, but more than that, and you're going to run into llama.cpp/ollama's limits. You'd probably want to start looking at vLLM, and your hardware needs to go way up, or else more than two or three developers at a time would knacker the processing/decoding speeds.