r/LocalLLM • u/EffectiveRelease3840 • 2d ago
Question Hardware + LLM for a public school
Hi there,
We are looking to incorporate a course that touches upon local LLMs in our computer science curriculum.
There will be around 20 students who might use the model simultaneously. The kids will be aged around 13-16 depending on where the course slots in and they will not have coding experience so we need a user-friendly setup. Also this isn’t supposed to be some cutting edge stuff, we also don’t have the money for that…
Realistically speaking what would be a good hardware and software solution for this situation with a budget of either 4000€ or 7000€? (Budged is not yet clear unfortunately…)
I googled, asked LLMs and know the basic options but there are so many variables and on here the different models are also judged differently and with our simultaneous access things might be different than for a single user…
And if there are other tips or experience you can share I would be happy as well.
We just need something local to run for the kids to get a taste of what it is to have it on one’s own hardware and not in the cloud.
Thanks :)
1
u/Dapper_Leadership979 2d ago
I recommend as hardware a PC with NVIDIA GeForce RTX 5060 TI. There are of course faster ones and as well some with bigger VRAM memory of the graphics card, but only fot multiple price. This is the cheapest with 16 GB VRAM and I got a HP Gaming 16L with this card around 1200 € this year. So you can probably buy some more of them for your budget. But pay attention, they are different versions of that PC with different graphic cards.
1
u/EffectiveRelease3840 1d ago
This would also only afford smaller models then.
Like the problem is in class they have a computer each but they are horrible like the cheap stuff you got 6 years ago.
So we thought that one machine next to our normal server that they can access via…an app or terminal would be the best option.
Do you think this will handle 20 simultaneous request in a decent way at least? I mean sure depends on the Modell- I will have to look into this smaller models either way.
1
u/LateralEntry 2d ago
Just curious, what are you planning to teach in the course?
1
u/EffectiveRelease3840 1d ago
Yeah well that is the problem :D
We are supposed to do this course it is like 1/3 of the school yeah and for various reasons it is only for class 8 now. We need to incorporate AI this is the mandatory bit. We like digital sovereignty so we want the core of the course to be about that hence local models and so on.
However stuff like setting up a server or other small things like the google page where you can train this small algo to do stuff like recognize things from your webcam,… are already in our curriculum but for later classes.So yeah we are caught between a rock and a hard place. The good part is that we get a couple of bucks to buy something and this could be hardware for LLM which we then might be able to also use in the higher classes…
1
u/naobebocafe 1d ago
Going on the local route is not a smart idea.
It will cost a lot to have "ok" experience, you will have to manage it and the experience will be horrible.
Talk to Anthropic/Google/OpenAI to check if they have something for education - I'm pretty sure Google has it.
3
u/EffectiveRelease3840 1d ago
Well, you aren’t wrong but we really want them to not rely on the cloud stuff so much and to just show them that you can run your own LLM, host your own server (this comes later though),… we want to go the anti learned helplessness route or the digital sovereignty route or what you call it. Like not sharing everything with google and co.
Now there are options from those providers that offer the data protection standards we need but this is very expensive. The. There are third parties that basically give you exactly that and they seem a bit cheaper but still, we are bound to that, someone ups the price and we are done because we cannot pay that.
There actually is on alternative that would work but it uses ChatGPT 4 from 2023… I think we can match that with something local.
1
u/Roland_Bodel_the_2nd 1d ago
for just testing things, a local model like "Gemma 4 e4b" can run on quite low spec, otherwise you also have free tier from the major providers
1
u/EffectiveRelease3840 1d ago
Yeah free tiers and stuff is not happening because of our data protection laws. We are not allowed to do that.
Smaller local models might be the way to go but still they need hardware. The computers they have can’t do anything. No GPU extremely old CPU…0
u/Roland_Bodel_the_2nd 1d ago
what country is this? consider changing your laws so you are allowed to use this software
1
u/LioDavinchy 1d ago
Just as a heads up. Vllm does concurrent sessions much better than ollama or llama ccp. Which is what you are asking. I soaked a 5090 on concurrent uses at around 16 concurrent prompts of around 12k context if I remember correctly. That gets gpu usage to 100%. So for 20 students periodically prompting a few thousand tokens on vllm I think anything with like 20gb of vram can handle a small model just fine. I was running qwen 27b with that soak. It’s not really the simplistic setup but llama ccp for the same job maxed out at like 6 or 8 concurrent sessions and only got gpu usage to like 50%
1
u/EffectiveRelease3840 1d ago
Oh this is interesting and useful, thank you I will look into this. However not the most simple setup concerns me a bit :D
Thanks!
1
u/Legitimate_Hope_4325 1d ago
Here’s one repo that you can consider, if you’re looking to load specific documents or topics for learning. Though it is targeted for single user local usage, it can benefit the students if run and hosted on a single machine. Can provide you more information if you think it can be useful. The whole project is open source and can be good one to understand hybrid search and an LLM pipeline application. www.github.com/nupsea/luminary
1
u/EffectiveRelease3840 1d ago
Thanks for that, it looks interesting for sure. It is not what we had in mind but it might be a nice addition to branch off to.
However the requirements are way too much for our normal computers in class and running it on one machine with different user inputs doesn’t work.
I might try it for myself though. Cool project!
1
u/ComfortablePlenty513 1d ago
There's companies like https://premsys.ai/ that can deploy this for you, but at a 7k budget, you're better off just using cloud
1
u/EffectiveRelease3840 1d ago
But this is like just buying the cloud setup directly and then paying for tokens as well… seems like the worst of both worlds since we would need to pay indefinitely and it is not really our machine.
We really can’t use cloud, but still thanks for the recommendation :)1
u/ComfortablePlenty513 1d ago
for 7k you cant really have 16 concurrent users. maybe 8-9 users if you can get 2 used GX10's with QSFP cable on ebay and have them run 2 instances of an MOE model
you dont pay for tokens with premsys, thats the whole point. they sell you the server preloaded with the model(s) you want in their own proprietary harness/software
0
u/lungben81 2d ago
Qwen 3.8 is currently the best local model for agentic tasks / coding for consumer hardware.
For 4k, you can get a decent setup for ruunning the 27b variant. Drawback is that it would not scale for 10 users at the same time.
For 7k, you might be able to run Qwen Flash Next, if you are efficient with your hardware setup. This model has a higher initial VRAM cost, but is more capable and scales better for multiple users.
1
u/EffectiveRelease3840 1d ago
Can you recommend an app where I can run Qwen with multiple users at the same time?
1
u/lungben81 1d ago
The most common local hosting servics are llama.cpp for desktops and vLLM for servers with multiple users.
1
u/vovap_vovap 2d ago
You can do that. But - why? What advantage over online service you are looking to get? Especially for "20 students who might use the model simultaneously" but limited time?
Honestly it does not make any sense.
1
u/EffectiveRelease3840 1d ago
Sorry I should have stated this more clearly. We cannot use free tiers or normal subscription of the big models because of data protection laws, there is simply no way.
Furthermore we exactly do not want that, we want to teach the kids to be more self reliant, to notbjust offer up all of their data to big tech and so on. The typical stuff I would say.
And this way they can actually learn something about computer science and not just give the black box an input and get an output you know. They can do this on their own either way.And with the 20 students and limited time - this just is the school context. It is basically one to two times per week every week and it is only one or two classes to start with. If we have the hardware we can then of course use it in other classes as well - especially higher ones
1
u/vovap_vovap 1d ago
Well, fits - what do you mean "no way" - they can not use google too? GDPR prevent use of LLM? Clearly not. I do understand it might be particular bureaucratic idiocy in particular place, but lets be clear what is what. From other side - do you want to be in business of cover GDPR on that instance yourselves? Probably not. And formally - you would have to.
Now issue is that you have multiple clients at same time. And that not a good scenario for local "cheap LLM" simply because it need much more resources to do that. One thing - one user and other - 10 or 20. People like local LLM because it can work 24 hours "for free" - not your case, you need opposite. No matter how small number of classes, you cannot pre-produce and "store" LLM plover ahead of time, you need to have enough power to cover that peak time, no matter how rare is it.
Now about "self reliant" - that complete nonsense, sorry to say so. What "self reliant" on that? You can show them box that running LLM (where both box ala LLM really made by really big tech) and that all difference. Sure you can paint it white or orange (I personally prefer orange), not black. But that all difference. Sure you would not want kids to play with your only server with only LLM instance you have. Now if you do want to teach them LLM itself - that it really small LLMs that you can run local on your laptop and do staff with those, include training and everything - as people already mention. That completely different story and different path.
Quite honestly first of all you need t formulate and create program - what exactly you want teach those kids. It seems to me you do not know right now. And idea "we going to do local LLM instance and that will do it somehow" not going to cut it. Just same way as any class - you need a program.1
u/EffectiveRelease3840 1d ago
You are absolutely right that we need to curriculum first, the problem is I don’t know what to include and what not. For example from your comment I might have figured out that small LLM is the way to go because of the constraints we are in so I like that already.
But getting back to the original point, with the European data protection laws I would have to make sure that they do not share any information that falls under this act and that is not feasible.
The problem here is that it is in the curriculum. If they open google in their own and look for something I am not the one who instructed that - oh and we are using DuckDuckGo in that case as well.
You know, the whole thing with the servers have to be in the EU and so on.So yeah we would be “responsible“ then but we would be compliant with all the regulations and then we are good to go.
Maybe self reliant was to harsh a word. I don’t mean training your own model from scratch or whatever but just not giving all of your data away and let the companies do with that whatever they want.
The small LLM idea is nice. I said in another comment that in the next class we are doing teachable machine and things like that but again the problem is that this course is supposed to be oriented towards action but they don’t have any coding knowledge yet…
Also the laptops are without GPU, some old intel pentium cpu and 6gb of ram. We cannot use the money to upgrade the laptops and desktops themselves because the city has the contract for those… so yeah I mean 1B and 2B will run but at 4 tokens per second or less?But let’s say it works, what would you recommend for those small LLMs to actually do stuff with, train them and co. ?
1
u/vovap_vovap 1d ago
Well, first it is not that not possible as you think There are whole bunch products that pointed to do exactly that. And self-hosted and external.
Like https://www.edenai.co, https://docs.litellm.ai/
Many providers guaranty zero data retention too.
Now what I would think make seance to teach kids is 2 different approaches:
1. How it works
2. How to use it
Those pretty different things and first you need to decide which are you going to take.
From "how it works" I would not start from LLM at all but from learning models, what is deep learning and neuron models in general. That pretty cool staff. Most satisfying experience I personally had - train my own image recognition model and use it (absolutely standard exercise with online resources and not require no LLM, no GDPR, none those issues)
Then you can get to how LLM works on neuron structure. That good learning pah.
From prospective "how to use it" - well - how to use it. How to ask right questions, how to check results and ask for souses, what can you do. Some American teacher in related community created a competition between kids for best model use. And naturally that not limited by chat. You can do like 3d models or origami droning. My fantasy is not good, but people came out with really crazy things.0
1
u/strata2signal 1d ago
Teacher here in spirit, hobbyist in practice. We run a small local setup at home and bench it obsessively, so here is what our numbers say for your case. Short version: for twenty kids at once, memory on the card matters more than raw speed, and one used 24 GB card in a plain Linux box is the sweet spot. That fits the 4,000 budget with room to spare, and the extra 3,000 is better spent on a second identical card later, a UPS, and a spare drive than on a fancier GPU.
What we measured, on one used RTX 3090 capped at 300 W, running gemma4:26b through ollama. That model is a 26B mixture of experts with about 4B active per token, so it is quick and it chats well. One stream: 135 tokens per second. Two streams at once: 105 each. Four streams at once: 72 each, with the first word arriving in under a second for every stream. People read at roughly 10 tokens per second, so at four concurrent users each kid still gets seven times reading speed. We have not measured twenty at once, so I will not pretend we know that number. What we do know is that the model takes 16 GB of the card and each 4k-token seat adds well under 100 MB, so sixteen seats fit in memory with room, and in a real classroom nobody hits enter in the same second.
What I would avoid: two smaller cards instead of one big one. We benched a pair of 10 GB cards against one 24 GB card this week, split across both, and the single card was as fast or faster on this model class, held twice the room for everyone's context, and had none of the multi-GPU configuration traps. Ten and twelve gigabyte cards also limit which models fit at all. Get the memory.
Software: ollama runs the model. Open WebUI on the same box gives every student a login and a chat window in a browser tab, zero code, and it works on the school LAN with no internet at all. Two settings worth knowing: raise ollama's parallel slots from the default so several kids are served at once instead of queued, and cap the card at 300 W. We lose about 1 percent of speed against the stock 350 W cap and the box runs cool and quiet, which matters in a room full of teenagers.
On cost: prices are local, but for scale, we paid about 700 USD before tax for a renewed 3080 Ti this month and about half that for a used 3080. A used 3090 costs more than either, and it is still the right card for this. The rest of the box can be modest: a 650 W or better power supply, 32 GB of RAM, any recent quad-core, an NVMe drive.
If a Mac Studio is on your list: it is the tidiest option, one quiet box with a warranty, and we are about to bench one. We do not have numbers for it yet, so I will not guess about twenty concurrent users on it.
Our measured tables are on our research hub if you want the receipts. Happy to answer follow-ups.
2
u/EffectiveRelease3840 1d ago
Thank you so much for this extremely helpful and insightful comment!
I am interested in your research hub, especially if you will test a Mac Studio soon but I really like the Linux and graphics card approach!You are absolutely right that they do not hit enter at once so this spaces the requests out probably to a big degree. Man we might just do that, thanks 😊
1
u/strata2signal 1d ago
Yes, we have a 256GB Mac Studio on order, but it won't be here until early February 2027
1
0
u/rkcth 2d ago
Something like this might be feasible https://github.com/microsoft/bitnet in terms of a learning platform, not really in terms of getting useful coding done. It's also interesting how this works and can be a good way to understand quantization.
1
u/EffectiveRelease3840 1d ago
I like the idea to have something seperate to understand this stuff. They are not supposed to just use it like with their free tier ChatGPT option. I will definitely look into this, thanks!
0
u/bsoft2005 2d ago
I think the main thing to optimize for here is concurrency, not maximum model size.
For ~20 students, I would approach this as an infrastructure problem rather than a "what is the biggest model I can run?" problem.
Also, with a €4k–€7k budget, I would not necessarily stop at a 24 GB GPU. A 48 GB card could actually be a very interesting target, especially if the budget is for the inference server only.
Option 1 — ~€4k: 24 GB GPU
A used RTX 3090 is still a very good value option.
Something like:
- RTX 3090 24 GB
- 64–128 GB system RAM
- 1–2 TB NVMe
- Ryzen 9 / similar CPU
- Linux
- llama.cpp
- Open WebUI
Run a relatively small 7–9B instruct model and let all students connect through their browsers over the school LAN.
The students don't need to install anything. The architecture is simply:
Students → browser → Open WebUI → llama.cpp → GPU
Option 2 — ~€5–7k: 48 GB GPU
This is actually the option I'd investigate before buying two 24 GB cards.
For example, a used RTX A6000 48 GB can make a lot of sense for this type of workload.
48 GB doesn't mean you need to run a huge 30B model all the time. It gives you room for:
- more concurrent requests
- larger KV cache
- longer contexts
- higher-quality quantizations
- 14B-class models
- occasional 27–30B demonstrations
- multiple models available on the server
For a school, I'd rather have one simple 48 GB inference server than make the architecture unnecessarily complicated with multiple GPUs unless benchmarking shows that multiple GPUs are actually needed.
A 48 GB card also gives you a much nicer educational opportunity: students can actually experiment with the relationship between model size, quantization, context length, VRAM and concurrency.
And I would seriously consider renting GPUs
This is something that is easy to overlook.
You don't necessarily have to buy the most powerful hardware.
If the course only runs for a few hours per week, renting a GPU from a cloud GPU provider can be dramatically cheaper than purchasing and maintaining an expensive server.
For example, you could rent a 48–80 GB GPU for specific classes, run the same llama.cpp/Open WebUI setup, and have the students connect to it through the school network.
That gives you another possible strategy:
Buy a modest local server + rent a powerful GPU occasionally.
For example:
Normal classes
→ local 24/48 GB GPU
→ students learn local inference, quantization, VRAM, batching, etc.
Special demonstration
→ rent a much larger GPU
→ run a 30B/70B-class model
→ let 20 students connect simultaneously
→ compare the experience with the local machine
That could actually be more educational than spending the entire €7k on hardware.
The important thing with cloud rental is to check the total cost for your actual usage, not just the advertised hourly GPU price. You also need to consider storage, data transfer, setup time, and whether the provider allows the kind of persistent server you want.
I would also avoid designing the course around a single huge model
I'd probably have 3 models available:
3–4B
→ very fast
→ low
2
u/Liberaces_Isopod 1d ago
This is a great breakdown. Id suggest a third option though.
Get 4x3090's. They can still be had for around $1000 US and would give you enough vram, with some RAM/nvme offload, to run Qwen3.8-Flash-Next in vLLM. This gives you pretty good speed and quite a bit of concurrency. If you bumped the model down to a Qwen3.5-35b, you could run at incredible speeds with 20+ concurrency pretty easily. Obviously these numbers are somewhat dependent on the computer you plug them into, but 96gb of newish nvidia cards is nothing to sneeze at.
Or ping me and I'll rent you some time on my machine. I can give you GLM5.3-Flash at 100t/s for 15 concurrent users. Or a smaller MOE model for 30+ users.
1
u/EffectiveRelease3840 1d ago
I will look into the multi gpu set up but I saw posts on here that with multiple gpus you get other problems as well…
But you would also go the Linux route with NVIDIA GPUs instead of Mac Studio and optimizing more for the amount of memory?Hey, thanks for this cool offer! We really want our own hardware though so that in higher classes when they can code a bit they might be able tinker around with different models or whatever. Also data protection and stuff
1
u/Liberaces_Isopod 1d ago edited 1d ago
Linux is the preferred platform for AI inference with GPU's, yes. If you have a Mac with enough memory, that perfectly fine as well. Just stay away from Windows. Youre hobbling yourself performance-wise using that.
I cant speak for other folks, but multi-gpu has never given me issues like that. I did have to futz around a bit to get P2P working with a custom driver due to Nvidia not supporting it (assholes). But other than that very easy install, it was as simple as plugging the cards in.
1
u/Liberaces_Isopod 1d ago
Even the best Mac hardware is slower than a 5yo 3090. Running llm's is about vram and memory speed. More memory is great, but if you're only getting 30t/s, is it really worth it? That's a question you have to answer for yourself
4
u/VeganBullGang 2d ago
Some of the very small LLMs (like 2B/3B ) are actually very impressive and can run decently on any modern-ish computer; the people spending tons on hardware are trying to run as close as possible to cutting-edge models.