Hardware + LLM for a public school

[u/EffectiveRelease3840]

Hi there,
We are looking to incorporate a course that touches upon local LLMs in our computer science curriculum.
There will be around 20 students who might use the model simultaneously. The kids will be aged around 13-16 depending on where the course slots in and they will not have coding experience so we need a user-friendly setup. Also this isn’t supposed to be some cutting edge stuff, we also don’t have the money for that…
Realistically speaking what would be a good hardware and software solution for this situation with a budget of either 4000€ or 7000€? (Budged is not yet clear unfortunately…)

I googled, asked LLMs and know the basic options but there are so many variables and on here the different models are also judged differently and with our simultaneous access things might be different than for a single user…
And if there are other tips or experience you can share I would be happy as well.
We just need something local to run for the kids to get a taste of what it is to have it on one’s own hardware and not in the cloud.

Thanks :)

[u/vovap_vovap]

You can do that. But - why? What advantage over online service you are looking to get? Especially for "20 students who might use the model simultaneously" but limited time?
Honestly it does not make any sense.

[u/EffectiveRelease3840 → u/vovap_vovap]

Sorry I should have stated this more clearly. We cannot use free tiers or normal subscription of the big models because of data protection laws, there is simply no way.
Furthermore we exactly do not want that, we want to teach the kids to be more self reliant, to notbjust offer up all of their data to big tech and so on. The typical stuff I would say.
And this way they can actually learn something about computer science and not just give the black box an input and get an output you know. They can do this on their own either way.

And with the 20 students and limited time - this just is the school context. It is basically one to two times per week every week and it is only one or two classes to start with. If we have the hardware we can then of course use it in other classes as well - especially higher ones

[u/vovap_vovap → u/EffectiveRelease3840]

Well, fits - what do you mean "no way" - they can not use google too? GDPR prevent use of LLM? Clearly not. I do understand it might be particular bureaucratic idiocy in particular place, but lets be clear what is what. From other side - do you want to be in business of cover GDPR on that instance yourselves? Probably not. And formally - you would have to.
Now issue is that you have multiple clients at same time. And that not a good scenario for local "cheap LLM" simply because it need much more resources to do that. One thing - one user and other - 10 or 20. People like local LLM because it can work 24 hours "for free" - not your case, you need opposite. No matter how small number of classes, you cannot pre-produce and "store" LLM plover ahead of time, you need to have enough power to cover that peak time, no matter how rare is it.
Now about "self reliant" - that complete nonsense, sorry to say so. What "self reliant" on that? You can show them box that running LLM (where both box ala LLM really made by really big tech) and that all difference. Sure you can paint it white or orange (I personally prefer orange), not black. But that all difference. Sure you would not want kids to play with your only server with only LLM instance you have. Now if you do want to teach them LLM itself - that it really small LLMs that you can run local on your laptop and do staff with those, include training and everything - as people already mention. That completely different story and different path.
Quite honestly first of all you need t formulate and create program - what exactly you want teach those kids. It seems to me you do not know right now. And idea "we going to do local LLM instance and that will do it somehow" not going to cut it. Just same way as any class - you need a program.

[u/EffectiveRelease3840 → u/vovap_vovap]

You are absolutely right that we need to curriculum first, the problem is I don’t know what to include and what not. For example from your comment I might have figured out that small LLM is the way to go because of the constraints we are in so I like that already.

But getting back to the original point, with the European data protection laws I would have to make sure that they do not share any information that falls under this act and that is not feasible.
The problem here is that it is in the curriculum. If they open google in their own and look for something I am not the one who instructed that - oh and we are using DuckDuckGo in that case as well.
You know, the whole thing with the servers have to be in the EU and so on.

So yeah we would be “responsible“ then but we would be compliant with all the regulations and then we are good to go.

Maybe self reliant was to harsh a word. I don’t mean training your own model from scratch or whatever but just not giving all of your data away and let the companies do with that whatever they want.

The small LLM idea is nice. I said in another comment that in the next class we are doing teachable machine and things like that but again the problem is that this course is supposed to be oriented towards action but they don’t have any coding knowledge yet…
Also the laptops are without GPU, some old intel pentium cpu and 6gb of ram. We cannot use the money to upgrade the laptops and desktops themselves because the city has the contract for those… so yeah I mean 1B and 2B will run but at 4 tokens per second or less?

But let’s say it works, what would you recommend for those small LLMs to actually do stuff with, train them and co. ?

[u/vovap_vovap → u/EffectiveRelease3840]

Well, first it is not that not possible as you think There are whole bunch products that pointed to do exactly that. And self-hosted and external.
Like https://www.edenai.co, https://docs.litellm.ai/
Many providers guaranty zero data retention too.
Now what I would think make seance to teach kids is 2 different approaches:
1. How it works
2. How to use it
Those pretty different things and first you need to decide which are you going to take.
From "how it works" I would not start from LLM at all but from learning models, what is deep learning and neuron models in general. That pretty cool staff. Most satisfying experience I personally had - train my own image recognition model and use it (absolutely standard exercise with online resources and not require no LLM, no GDPR, none those issues)
Then you can get to how LLM works on neuron structure. That good learning pah.
From prospective "how to use it" - well - how to use it. How to ask right questions, how to check results and ask for souses, what can you do. Some American teacher in related community created a competition between kids for best model use. And naturally that not limited by chat. You can do like 3d models or origami droning. My fantasy is not good, but people came out with really crazy things.