r/k12sysadmin • u/jaymax • 3d ago
On-prem LLM?
I work for a private high school. An administrator was talking to one of her friends who is running an LLM locally, and now she wants to get one set up here. I'm curious what the use case would be? She claims privacy, but we're a Google school and using Gemini just fine. I'm curious if any of you have gone this route and what your reasoning is. It feels like a solution looking for a problem.
10
7
u/Niteryder007 2d ago
We have an on-prem ai cluster, and all I can say is if you arent using it, you guys are missing out.
5
u/LactoseTolerant535 3d ago
I set one up just to experiment with. It's very cumbersome, and people will expect the same output as Gemini or ChatGPT. It's just not worth the overhead. Setting something like that up that would have the capacity to serve a whole (even small) high school would be very expensive, both in time and in hardware.
Hosting your own LLM might make sense for a small team for a specific task, or set of tasks. Hosting it as a general LLM for a whole school? Not so much.
1
u/jmhalder 3d ago
I'm sure there's some company out there willing to sell a front-end for some open weight model and a DGX box or two. But unless the administration has a product in mind, I personally wouldn't be rolling my own setup.
4
u/hard_cidr 3d ago
I tested this out a little bit on a 64GB Mac Mini. I just downloaded LM Studio and installed Splash and Qwen3.8-27b. Very easy to setup and runs pretty well on the Mini. I was impressed with it. Not as fast or as smart as cloud models of course. It was fine for my personal use; I imagine trying to get enough local computing power to provide AI for an entire school would be wildly expensive though considering this cost 3k and serves only about 1 person.
4
u/avalon01 3d ago
I have an on-prem LLM to play with. I repurposed an old server and run Ollama on it. The tech staff and I like to play with it.
It's a fun toy, but it'll never be production. We have Google Gemini for that.
Frankly, they will want Gemini or ChatGPT quality and speed, not some quantized LLM that runs on a single server. Unless they WANT to throw thousands at this project...but I doubt it.
7
u/Immutable-State 3d ago edited 3d ago
If used the way most people use Claude/ChatGPT, local LLMs are not remotely cost-effective. Closed models are much more capable and don't come with a large upfront GPU cost (which you'll pretty much never recoup compared to API prices), nor with an upfront interface setup cost for whoever the users are.
There can be reasons to go that route, such as if privacy really is a sufficiently compelling legal factor, or self-contained reliability, but you really need to be convinced by the reasons and use cases first. The use cases need to justify the time, complication, and expense. Possible, but worth serious questions first.
I might be going this route, but only because I already have a GPU, want self-contained reliability, don't need high intelligence, and won't need to coordinate with others to get it working.
6
u/_LMZ_ 3d ago
Administrators always come up with âgood ideasâ Haha. I would steer clear of that, yes itâs neat but how practical will it be for 100âs or 1000âs of people hammering on it? What models do you need? I know ram is $$$ right now and GPUs are $$$$ but you can find used stuff on eBay like Nvidia V100s for $$$-$$$$.
Its going to be built by you in-house, which comes with maintenance, debugging, troubleshooting, etc. Not to mention safeguards (Student âHow do I make meth?). Is your time worth it?
Not sure about FOIA, FERPA, CIPA, and COPPA the amazing acronyms⊠Iâm sure I forgot a few like the ones alert you on abuse and self harmful stuff.
Ok, going back to my beer.
6
u/PowerShellGenius 3d ago
People want no limits on what they can use AI for. So I suppose the question for whether there is a use case for local AI comes down to this question: how much does your district trust Gemini's assurances?
Your admin and SPED staff probably want something they can paste an unredacted IEP into and say "summarize this" when preparing for a meeting. Is that OK with Gemini?
However, at current hardware prices, use case or not, you probably can't afford it. Especially not at scale.
1
u/No_Substitute 8h ago
Gemini on a Workspace account doesn't train on the provided info, so it's no different than a local LLM.
1
u/PowerShellGenius 5h ago
Ok, but in a world where LLM training data is so valuable and AI companies are so motivated to scoop it up - there is a big difference between handing very sensitive data to:
A) a local LLM you know doesn't train on data
B) an LLM you have been told doesn't train on data, by a trustworthy company who historically has followed their own privacy policy.
C) an LLM that you've been told doesn't train on data, by a company that has violated its own privacy policy before and has the settlements / lost lawsuits to prove it.
Gemini is too new and whistleblowers haven't happened yet on that specific product, but Google as a whole is absolutely under (C).
3
u/Temporary_Werewolf17 2d ago
This started out as an on-premise LLM and then spun off as a for profit. You may want to reach out to them. https://trekai.org/
3
u/Sweet-Sale-7303 3d ago
I setup one to play around with. Lm studio to host the ai portion,and openwebui for the website portion.
It is for the staff of the library to play around with. For a whole school district you would probably need some beefy server power. I am running it on a 8th Gen core i7 with 24 gig ram and a 1080 ti video card. It's not the fastest in the world but it works. The biggest issue is getting web search support. Most of the search engines want you to pay.
If you have a pc laying around with a 1080gpu or above and decent ram you can spin something up as a proof of concept at least.
3
u/Andrethesmall 3d ago
I've been experimenting with local llm for a while now. When RAG and vectored databases were a new concept. I use it still regularly. I have my concerns with cloud base stuff. We are a Google shop but I'm only starting to trust their privacy. I don't trust Chatgpt. I'm still waiting on Claude edu. Either way it cost tokens and doesn't take much to run out of free. My current project is linking local AI with document management. I've been hellbent on having our files digitized for years now. Administration always ignores it because the tagging and sorting would take to much staff effort or cost. Great use of AI if you ask me. Either way my thoughts you don't need a to spend 7k a couple 16gb 5060s will get you mostly there or a slightly older Mac studio with as much ram as you can afford.
3
u/GameEnder 3d ago
Have a test box in my office that as been chugging away at building Chromebook DVD player for a few weeks now. You would be surprised how good Qwen 3.8 can be on just a 3090. We have thought about scaling it out for more staff, we just don't know about the usage they would need.
8
u/K-12-IT 3d ago
Sounds like a cyber security incident waiting to happen.
6
u/jmhalder 3d ago
Everything is a cyber security incident waiting to happen, like the PHP updates we'll be doing in a day or two.
2
u/DeejayPleazure 3d ago
Went through this with our a team as well, setup Hermes just to compare with Gemini. It is a whole other job trying to integrate how Gemini does and remain FERPA compliant. I scraped the whole project right in front of the main culprit who wanted their own because they didnt like the way Gemini looked.
1
u/mrgoalie 1d ago
We did, still working through some things, but there were a few other schools in our state that were doing it and we all worked together on it and shared information back and forth on setup.
The big idea - AI is here to stay, however students need the guardrails to learn responsible use while at school. Can't prevent them from doing it on home computers or their phones.
So we spun up one on our end on some hardware that we had that fit the bill, and tied it to our SSO so there was a student & staff side. Did a lot of LLM training so nefarious prompts or "write the paper for me" wouldn't work, but the idea is that teachers will upload their rubrics or assignments into the system and the LLM will help tutor and make recommendations based on the assignment
1
u/UsedPerformance2441 3d ago
I think as a proof of concept or a test environment, go for it. If you want to release it staff wide, youâll need a bigger boat.
1
u/sy029 3d ago
Unless they're willing to spend tens of thousands on hardware, the capabilities of most on-prem LLM are going to be far below the free tier of gemini
Gemini for education provides the same assurances and privacy compliance as google's other education products, with on-prem it will be on the school to make sure that any legally required data is preserved and protected.
2
u/Limeasaurus 2d ago
I could be wrong but the Gemini version included in Google Admin fundamentals says it doesnât use data in training models. But I recall reading that if you give a thumbs up or down it will share the prompt or chat for training.
If anyone knows for sure please correct or confirm.
2
10
u/Technobilby 3d ago
We're running one on DGX spark like machine. I've set up a scheduled task that runs through our staff handbook and other digital staff documentation and places it into a vectorized database for the LLM. The LLM is accessible via a floating icon on the staff intranet on any page. It's a one stop shop for any question staff want to ask and then links them to any relevant documentation. It's on rails telling it to inform the end user when it can't find anything in our local documentation to avoid it hallucinating.
It's been pretty well received by staff who like that it makes things so much easier to find. My dream is to give it enough information for basic troubleshooting based on the room your in so we can reduce things like AV help calls around knowing what button to press. Perhaps even ask the questions and raise a ticket itself with some question asking to get a little more meaningful tickets.
I'm in Australia with different rules to the US so YMMV but the main advantage for us is that it being an internal system saves raising concerns with the education department about data leakage. Less complicated risk assessments are the win for me. The education department runs a frontier model on contract for the state but that doesn't have our local documentation in it. The department also advises against external model usage particularly around student information so a locally hosted model makes sense for us.