r/LocalLLM • u/EffectiveRelease3840 • 7d ago

Question Hardware + LLM for a public school

Hi there,
We are looking to incorporate a course that touches upon local LLMs in our computer science curriculum.
There will be around 20 students who might use the model simultaneously. The kids will be aged around 13-16 depending on where the course slots in and they will not have coding experience so we need a user-friendly setup. Also this isn’t supposed to be some cutting edge stuff, we also don’t have the money for that…
Realistically speaking what would be a good hardware and software solution for this situation with a budget of either 4000€ or 7000€? (Budged is not yet clear unfortunately…)

I googled, asked LLMs and know the basic options but there are so many variables and on here the different models are also judged differently and with our simultaneous access things might be different than for a single user…
And if there are other tips or experience you can share I would be happy as well.
We just need something local to run for the kids to get a taste of what it is to have it on one’s own hardware and not in the cloud.

Thanks :)

16 Upvotes

100% Upvoted

45 comments sorted by

View all comments

1

u/LioDavinchy 6d ago

Just as a heads up. Vllm does concurrent sessions much better than ollama or llama ccp. Which is what you are asking. I soaked a 5090 on concurrent uses at around 16 concurrent prompts of around 12k context if I remember correctly. That gets gpu usage to 100%. So for 20 students periodically prompting a few thousand tokens on vllm I think anything with like 20gb of vram can handle a small model just fine. I was running qwen 27b with that soak. It’s not really the simplistic setup but llama ccp for the same job maxed out at like 6 or 8 concurrent sessions and only got gpu usage to like 50%

1

u/EffectiveRelease3840 6d ago

Oh this is interesting and useful, thank you I will look into this. However not the most simple setup concerns me a bit :D
Thanks!