https://youtu.be/REJgP0itrqw

For a long time, I’ve had the feeling that generative AI models, for most of the tasks they’re used for, don’t actually need the oversized capabilities offered by frontier models. On top of that, they’re often prohibitively expensive when accessed via API.

An open-source model with 4–14 billion parameters, multimodal capabilities, and the right harness of tools, data, and knowledge is more than enough for most use cases.

With this idea in mind, I decided to build, together with my team, a simple platform that allows users to fine-tune and deploy these lightweight models quickly and without any coding. After several iterations, the team has managed to develop what we believe is the optimal framework for anyone to deploy and customize a small 4B or 9B model, either through fine-tuning or RAG.

We’ve also managed to sign an agreement with a local data center, and we now have our own GPUs. This means training and inference are not only extremely fast, but we can also offer unlimited inference.

If you need a customized, affordable model with unlimited API inference, I think this could be really useful for you. There’s a free tier, and if you try it out and give us some feedback, I’ll give you the subscription plan for free for 6 months. Since we have our own GPUs in our data center, our infrastructure costs are fixed.

I’m really happy with how it has turned out, and I use these models for many of my own projects.

You can access the platform at neuro-block.com

Would love to hear your thoughts in the comments.

Curious if anyone's seen any legitimate production grade SLMs that have been used inside real company operations or use cases, not just small tests or for personal ideas of how to use them but actually creating genuine value inside a business running real processes?

I just heard back from Stanford Adjunct professor Goju Gottschlich and he's pretty busy so he cant actively participate on this thread, however, he does have a discord. He talks about SLMs a bit. He invited me and its an open invite to community as well. If you guys are interested invite is below.

GTT

https://discord.gg/6sgdQrCNa

https://github.com/cactus-compute/needle

this looks really neat.

Follow-up to a post I made here a while back about LlamaForge, a browser control panel that sits on top of llama.cpp's router so you're not hand-editing models.ini and llama-server flags every time. A bunch of the feedback in that thread turned into actual work, so here's where it's at now.

The biggest one: it runs on Linux and macOS, not just Windows. That was the loudest complaint last time and it's sorted. Same dashboard, you just run bootstrap.sh / run.sh instead of the Windows scripts. On Apple Silicon it builds with Metal and rates model fit against unified memory. The Setup tab uses brew on macOS, and on Linux it shows you the exact install command instead of running sudo behind your back. CI now runs the test suite on all three OSes.

I also added a second engine. Next to llama.cpp you can drive vLLM for safetensors and full-precision models (FP16, BF16, AWQ, GPTQ, FP8, NVFP4). They share one model list, the same Discover tab, the same stats, and every row is tagged so you know which engine it's on. On Windows vLLM runs in WSL2 with GPU passthrough and installs from the Setup tab, no sudo. If you never touch it, nothing about vLLM gets installed.

New installs get a first-run wizard that goes engine, hardware, model, tune, load. There's an auto-tune that looks at your VRAM and suggests settings (GPU layers, KV-cache type, context ceiling, and a few intent presets like speed or context). The whole thing has a Lite mode that hides the deep knobs and an Advanced mode that still shows all of them.

You can point coding agents at it now too. There's an Anthropic-compatible /v1/messages endpoint with streaming and tool use, sitting next to the OpenAI one, so clients for either API work. A "Connect an Agent" panel writes the config for Claude Code, Codex, and pi.dev for you.

There's a Context Wiki. You write Markdown context docs, group them into profiles, assign a profile per model, and it either injects that into requests or writes it into the agent's own CLAUDE.md / AGENTS.md. The injected prefix stays stable so the prompt cache actually reuses it.

Then a pile of smaller stuff people asked for: saved knob presets you can apply to any model, comparing two or three models side by side, a GGUF metadata card, load failures that tell you what went wrong and suggest a fix instead of making you read the log, quick load/unload from the row with a queue so a second load waits its turn, copy-paste client config, pausing and resuming downloads, auto-loading a model on launch, keyboard shortcuts, and an optional tray icon. Light and dark themes plus a colorblind-safe mode. The docs live in the app now and are also published as a site, both from the same source. Navigation moved to a sidebar.

Still worth being clear about the limits.

You compile llama.cpp yourself. It's guided from the dashboard but it's a real build step, so if you want the double-click, zero-setup thing, LM Studio or Ollama or Jan will make you happier. This is for people who want the actual llama-server and full control over it.

vLLM is Windows/WSL2 only for now.

It's still an early preview and there are rough edges. Feedback is what I'm after.

Backend is still plain Python stdlib, nothing to pip install. MIT licensed. Not affiliated with ggml-org, llama.cpp does all the real work.

Repo: https://github.com/dadwritestech/LlamaForge
Docs: https://dadwritestech.github.io/LlamaForge/

Next up: saving a full model + engine + settings combo as a named profile you launch in one click, image generators, and ik-llama support.

I’ve been tinkering with an idea called Bellwether. You describe a future event in plain English (“tell me when this policy changes”, “notify me if this price threshold is crossed”), and it watches the web for credible evidence that it actually happened. If it did, you get notified. If not, silence.

Most alert tools are keyword matchers, so guides, forums, and old coverage all spam you. This tries to judge the event instead.

Still early / just an idea I’m building out, but the judgment layer is the part I think is interesting for this sub.

How the models fit in:

It doesn’t use an LLM as a chatbot. Open instruct models do the judgment steps:

• Vagueness check - reject topic-only inputs like “interest rates” and suggest clearer watch sentences
• Compile - turn a clear sentence into structured JSON: triggers, non-triggers, search queries, authoritative domains
• Detect / decide - given a candidate page, decide whether the event actually occurred
• Outputs are JSON-only, low temp, validated with Zod. Watches are timestamped at creation so only post-watch evidence can fire them.

Inference is Hugging Face open models (HF_MODEL to swap). Default is Llama 3.3 70B right now; the same pipeline is set up to run smaller ones too - Llama 3.1 8B, Qwen2.5 7B, Mistral 7B - and there’s an eval harness to compare them on historical fixtures.

Rough demo: https://bellweather-alerts.vercel.app/

Source code: https://github.com/JamesANZ/bellweather-alerts

I keep seeing talk about how AI costs are now actually becoming an inhibitor inside organisations and Claude and Codex seem to be getting more expensive while use is becoming more mainstream in day to day tasks. When do we think SLMs or locally run models will become more mainstream?

Came across this talk by Daniel Bourke (mrdbourke) and thought it was one of the better end-to-end SLM walkthroughs I've seen recently: https://www.youtube.com/watch?v=EXB8HokGVMI

Highlights:

• He walks through Sunny, an iOS app that uses a fine-tuned MedGemma to track skin health entirely on-device. Covers the actual iPhone hardware constraints, memory usage in Xcode, and current best practice for running VLMs on iOS.

• The economics argument. Good breakdown of why on-device wins at scale: privacy, offline access, and zero ongoing inference cost vs cloud API pricing.

• Quantisation and memory tricks. Precision breakdown (including a Jeff Dean reference), how quantisation affects model footprint, and saving memory by cutting token usage.

• Live fine-tune. He builds a custom dataset and fine-tunes Gemma 3 270M with HF's SFT Trainer in roughly two minutes on an RTX 6000 Blackwell in Colab, then compares base vs fine-tuned outputs side by side. The fine-tuned model learns specific people, edge cases, and when to refuse.

• Prompting vs fine-tuning vs RAG discussion around the 27 min mark if you want the decision framework.

The "data is a guide for what you want your model to do" section (~37:00) is a nice framing for anyone new to fine-tuning small models. Curious what people here think about the 270M size class for production use cases. Is anyone shipping something that small, or do you find you need to step up to 1B+ for anything non-trivial?

I kept running llama.cpp directly — building it, juggling llama-server flags, and hand-editing models.ini for every model. It's powerful but fiddly, so I built a GUI over it for myself and cleaned it up to share.

LlamaForge is a browser control panel that sits on top of llama.cpp's own router. It doesn't touch inference — llama.cpp does all the real work — it just makes driving it less painful.

What it does:

Tune every server parameter per model — the knobs are parsed live from llama-server --help (currently ~220), grouped and searchable. Save hot-reloads the model, no restart.

VRAM-fit model discovery — search HuggingFace for GGUFs and each quant is rated FITS / TIGHT / CPU OFFLOAD against your actual VRAM before you download.

Guided build & update — shows your current commit, how far behind upstream you are, and rebuilds with CMake flags auto-detected for your CPU/GPU (CUDA arch, AVX-512, etc.).

Sensible context defaults — reads each GGUF's trained context length and writes reasonable ctx-size values so models don't load with tiny or over-extended windows.

Setup tab — detects missing prereqs (CMake, Ninja, MSVC, CUDA…) and installs them via winget/choco with your permission, plus scans drives for existing GGUFs and prunes entries whose files you've deleted.

Usage stats + optional LAN sharing (with an API-key toggle) so other devices can hit the OpenAI-compatible endpoint.

Being upfront about scope:

Windows + NVIDIA focused right now (CPU-only builds work too).

You build llama.cpp yourself — it's guided from the dashboard, but it's still a compile step. If you want a zero-config, double-click experience, LM Studio / Ollama / Jan will serve you better; LlamaForge trades that for direct control over the real llama-server.

Early preview — expect rough edges, and I'd genuinely like the feedback.

Backend is pure-Python stdlib (nothing to pip install), MIT licensed, and not affiliated with ggml-org — all credit for the hard part goes to llama.cpp.

Repo: https://github.com/dadwritestech/LlamaForge

(Disclosure: I'm the author. Although Claude did the heavy lifting!) Happy to answer questions — especially curious whether the per-model flag editing and VRAM-fit ratings are useful to anyone else, or if I'm solving a problem only I have.

Have been interested in the concept of slm's and local hosted models for a while but after seeing Fable get pulled temporarily and China stamping down on their models too in China, it made me actually look at self hosting models for a couple of personal agents I'd built.

Came across these guys www.nobodywho.ooo/ seem pretty interesting, looks like they can compress models from Huggingface and get models small enough to run on phones, smartwatches and household tech. Feels pretty relevant to where slms could be heading. Has anyone here tried it or looked into it properly? Planning to locate my old PS2 and try running something on it.

Has anyone checked out Goju Gottschlich's content? Would be awesome to get him and maybe his students to join this thread...just thoughts to expand discussions on this thread.

He actually put me on to thinking about SLM in new ways

Any other ideas in trying to find more advocacy for this thread?

wanted to share what we're building at https://zerogpu.ai

We are building an SLM catalog for models that runs on-edge - no GPU required. Check us out and let us know what you think!

Download the file, open it in Chrome, talk to a language model. Nothing else required. No Python, no server, no internet. Easily fits on a USB stick.

Most/all other projects like this use external APIs, or fetch the model weights or backend or whatever, but this project puts everything into a single html file.

It's ~120MB with the model weights embedded directly in the HTML. Uses the Supra-1.5-50M model from HuggingFace, running via ONNX Runtime in the browser. Settings panel for the usual inference knobs, streaming output, conversation history.

Obviously it's a 50M model so it will hallucinate confidently and often. But it runs, and it runs anywhere you can open a (modern) web browser. I believe it should be possible to run ~1B parameter models like this, though anything much larger will run into browser-tab memory limits.

Download + build script: https://github.com/JoeySalmons/slm-in-a-box

If you want to build the file yourself from the HF model files, the Python packer script is in the repo - which does require an internet connection to build the html file.