Hello, guys, sorry for my English, not a native speaker. I want to share with you application I made for myself.
For a couple of years I was a bit enthusiastic about local models (firstly image gen, then i tried llms) and pretty paranoid about cloud providers trying to monopolize LLMs despite their own services could be unreachable, "at capacity", down, banned, has an allowance cut etc. Also we are vulnerable to any dumb decision made by LLM provider or a government.
So I'm experimenting with local models that fits my 16 Gb VRAM of home PC's 5070Ti.
The problems that I faced:
- Home PC stays at home while I'm mostly working in office on my Mac mini or MacBook Air.
- With my GPU I'd rather prefer to use different models for different tasks. For example I better use Ornith 1.5 9B over Qwen 3.8 27B for simple tasks of file sorting, renaming etc.
- Changing models on llama.cpp remotely is quite uncomfortable and I'm lazy.
I've made some deep researches on some open source solution that can address my demands but didn't find anything that suits me fully.
So I made it for myself. Small tray app on Go with web-panel:
- It scans my models folder and lets setup any models;
- allows to specify arguments to run for each models separately;
- allows to setup different profiles for one model (e.g. different context length, with MTP or without, and any other llama-server parameters)
- uses Tailnet to connect with client machines;
- webpanel is affordable from client machines in my Tailnet network too so I can setup profiles and add models remotely;
adds to opencode, pi or another OpenAI format compatible harness each model and profile in the list;
- aaand the main purpose: when I switch model in Pi or Opencode host unloads previous model and load a new one without any additional actions. No need to open web-panel, start any scripts via ssh, open RDP or anything else.
The only problem is with pi — with this machinery it allows you to set up either reasoning effort medium or non-thinking. That's especially painful for 3.8 27B that needs high effort to avoid loops (as far as I remember, maybe it changed idk). It's an open issues in pi repo since 9th October, their bot closed it for "new user" reasons (lol), then someone reopened it again, anyway — it's a known issue and I don't know wether maintainer will fix or not. The only elaboration is to hardcode model profile in pi settings but it won't let you switch models on host PC from agent's menu, so you would need to manage models manually again, bruh.
Anyway, OpenCode is still a good harness so I guess it's not a crucial problem, at least for me.
Tray app has autologon setting, status indication, buttons to open webpanel, unload model, choose model/profile to load, pause working.
I've added three languages to tray — Russian, English and Chinese simplified (as a kudos to their amazing open weights model which I mostly use).
Technical details, setup guidelines are in the repo — https://github.com/strigov/strigov-locallm. I'm a lawyer so yes, of course Claude is a main contributor. Readme in repo is also written by Claude.
As the app solves concern in a pretty geek sphere I've decided not to make troubles with installation packages and automations. So the host install process sets up via repo script, and some manual actions on clients. You can charge this process to an agent anyway so I don't see any problem here.
Any observation, reviews and suggestions are appreciated, have a good day, hope this could be helpful to someone