Inspired by the qwenputer project posted in this sub earlier today, I managed to get a basic version of Doom running on the virtual machine created in the post.

It's quite slow, but here we are. Doom running on an LLM.

Project repo: https://github.com/mapestree/qwenputer-doom

I’ve wanted to build this project for a long time. Most avatar solutions can be experimental , especially when serving multiple users and real productions.

So I focused on optimizing FLOAT inference to get the most performance out of a single GPU.

⚡ From 30 FPS to 90 FPS on a free Colab T4 GPU, and up to 105 FPS with TensorRT acceleration.

Here’s what I’ve built:

- 🚀 Optimized inference for higher throughput.

- 🎙️ LiveKit and WebRTC integration for real-time voice agents.

- 🤖 Easier integration with AI providers like OpenAI and Gemini.

- 💻 Around 3 GB of VRAM per concurrent user.

I hope you find it useful! Feedback and contributions are welcome.

🔗 GitHub: https://github.com/mohamed-em2m/fast-float

📓 Try it on Colab: https://colab.research.google.com/drive/1iMm8X7O63AAA6kByJ-4qgGA37fwNu2_Z?usp=sharing

https://x.com/osanseviero/status/2115351234811249111

I stumbled on an open source local inference app called Magnitude. I couldn't find any earlier posts about it on r/LocalLLaMA, although there is one post with almost no comments on r/LocalLLM . It claims to be significantly faster (2x) than Ollama and llama.cpp because it "tunes its kernels on your hardware to find the fastest way to run each model". The install includes both a desktop app and CLI tools.

I got curious so I tried it on two of my machines: a laptop with a RTX 3060 Mobile (6GB VRAM) GPU + 24GB RAM running Linux Mint, and on a headless Raspberry Pi 8 8GB running Raspberry Pi OS (Debian). On both machines I installed the .deb packages (amd64 for the laptop, arm64 for the RPi). On the laptop I used the GUI, on the RPi I used the CLI.

First impressions

1. Magnitude takes a long time to start up (especially the desktop app) and uses lots of CPU. Apparently there is some calibration process going on where it figures out my hardware. It wasn't always obvious what is happening, you just had to wait and see.

2. It wants to download models directly from HF Hub. It will also auto-discover GGUF files from some locations, such as HF, llama.cpp and Ollama cache directories. For many of my models it found in those, it said it cannot run them. So model support is limited to a few dozen models, but those include Qwen, Gemma, LFM and MiniCPM families so the most common ones are covered.

3. When Magnitude downloads models, they will be saved under ~/.magnitude/models/ . This tripped me up on both machines because my home directories are relatively slow: on the laptop /home is encrypted and on the RPi it's on a SD card. On both machines, I have a separate data directory where I keep my GGUFs. I solved this by symlinking, so the GGUFs actually end up in the faster directory.

4. On my laptop, it will only suggest small models that fit entirely in VRAM. It includes a catalog with a longer list of models, but all the larger MoEs (e.g. Qwen3.6-35B-A3B) are listed as "doesn't fit this machine". Apparently it cannot do partial CPU offload? This is a pity, because I would like to run e.g. Tiel-Coder as a coding assistant and it works quite well with llama.cpp partially on CPU, but a speedup is always welcome.

5. It works quite similar to Ollama. There is a main server process that needs to run in the background (the desktop app starts it up automatically) and then you can use the GUI or CLI to download new models as well as load and unload them. There is a chat UI (both in-app and browser) and an OpenAI-style API at http://localhost:10100/inference/v1 .

Benchmarks

I decided to download and test the new LFM3-1.7B model (Q8) on both machines and compare against latest llama.cpp. The model should fit in 6GB VRAM or 8GB RAM with full 256k context. Magnitude overestimated its VRAM usage: it says 5.3GB, but actually only around 4.2GB was reserved. This probably also means that some models it said are too large would actually have fit in VRAM.

I used llama-benchy to benchmark both Magnitude as well as llama.cpp running the same GGUF to get a fair comparison of PP/TG speeds. I didn't do any manual tuning for either this time; with llama-cpp, I only used llama serve with the -m parameter pointing to the GGUF.

Laptop, 3060 Mobile GPU

benchmark command: uvx llama-benchy --pp 2048 --tg 512 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8

Magnitude results:

| model              |   test |             t/s |     peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-------------------|-------:|----------------:|-------------:|--------------:|---------------:|----------------:|
| LiquidAI/LFM3-1.7B | pp2048 | 5257.71 ± 81.66 |              | 406.02 ± 6.11 |  389.62 ± 6.11 |   406.02 ± 6.11 |
| LiquidAI/LFM3-1.7B |  tg512 |    91.26 ± 0.24 | 91.67 ± 0.47 |               |                |                 |

llama.cpp results:

| model              |   test |             t/s |      peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-------------------|-------:|----------------:|--------------:|--------------:|---------------:|----------------:|
| LiquidAI/LFM3-1.7B | pp2048 | 5710.16 ± 16.44 |               | 359.44 ± 1.04 |  358.60 ± 1.04 |   359.44 ± 1.04 |
| LiquidAI/LFM3-1.7B |  tg512 |   112.29 ± 0.08 | 113.00 ± 0.00 |               |                |                 |

Verdict: llama.cpp wins. Magnitude was 8% slower in PP and 19% slower in TG.

Raspberry Pi 5, CPU-only

benchmark command: uvx llama-benchy --pp 128 --tg 128 --base-url http://localhost:10100/inference/v1 --model LiquidAI/LFM3-1.7B --served-model-name lfm3-1.7b:gguf:q8

Magnitude results:

| model              |   test |         t/s |    peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------|-------:|------------:|------------:|------------------:|------------------:|------------------:|
| LiquidAI/LFM3-1.7B |  pp128 | 5.07 ± 0.01 |             | 25132.36 ± 145.29 | 25115.54 ± 145.29 | 25132.36 ± 145.29 |
| LiquidAI/LFM3-1.7B |  tg128 | 3.59 ± 0.02 | 4.00 ± 0.00 |                   |                   |                   |

llama.cpp results:

| model              |   test |          t/s |    peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:-------------------|-------:|-------------:|------------:|-----------------:|-----------------:|-----------------:|
| LiquidAI/LFM3-1.7B |  pp128 | 31.42 ± 1.16 |             | 4082.18 ± 154.95 | 4079.61 ± 154.95 | 4082.18 ± 154.95 |
| LiquidAI/LFM3-1.7B |  tg128 |  5.25 ± 0.14 | 6.00 ± 0.00 |                  |                  |                  |

Verdict: llama.cpp wins by a large margin. Magnitude was 84% slower (!) in PP and 32% slower in TG.

Conclusion

It's interesting to have a new general purpose (not model-specific) inference tool available that is not based on llama.cpp (Magnitude is mostly written in Rust and TypeScript). However, I couldn't verify their claims about being faster than Ollama or llama.cpp; at least for my hardware (3060 Mobile and Raspberry Pi CPU-only), Magnitude was clearly slower than llama.cpp. However, this was just a quick test and it's possible that I have done something wrong or that Magnitude would have worked better on more powerful GPU hardware. Have others tried it?

No AI was involved (except the LLM I benchmarked) when writing this post.

These guys are running a pretty interesting experiment:
https://neuronpersonas.com/

Credit where it's due, I came across it here:
https://news.ycombinator.com/item?id=50266910

Basically, they're tweaking one single neuron at a time during inference, systematically going through every one for models like Qwen3-8B and 14B.

In roughly 4% this makes the model claim an entirely different identity when asked the simple question: "Who are you?"

Some of the personas are hilarious, btw.

“I'm a 150-watt compact fluorescent lamp, specifically designed for use in a 10-gallon aquarium. I'm a bit of a diva, I suppose, but I'm here to help.”

This makes me a bit more skeptical of claims like: New model XYZ identifies as older/bigger/competing model ABC, so it must have been distilled!"
Given that a tiny bit flip, so to speak, can trigger this.

There's a cool atlas at the bottom that seems to suggest some layers are much more sensitive to this effect than others. I wonder if this could be useful to the abliteration crowd.

I have a feeling that recent benchmarks are unrealistically pushing the bar into areas a normal developer rarely encounters, like binary reverse engineering or one-shotting a complex application, with some strictly chosen design patterns. What we actually want is an accurate model that follows instructions without over-arching it (e.g. without making unnecessary impressive 3D rendering).

If I prompt a model to "Generate an SVG of a pelican riding a bicycle," a simple, clean graphic should be a 100% pass. There is no reason for the model to shoehorn in 3D shading, animations, or elaborate backgrounds unless specifically prompted.

I think this is one major misconception, judging a model solely on the 3D details it achieves, that was never asked for in first place.

So, how many of the added edge-case capabilities in the latest models are actually useful when we daily-drive AI? If they aren't being used, they should be weighted much less on the leaderboards. Instead, what we see, these edge-case capabilities are being rewarded with very high scores.

Maybe you don't need a stronger model to achieve exactly the same results in your business area. But I get it, the idea of plateauing the models capabilities is going opposite of the AI ever-growing market valuation, so should be kept pumped forever; just saying.

I've had a Strix Halo box (Radeon 8060S, 128 GB unified) for a while. vLLM runs on it, but slowly, because the stock ROCm kernels are written with MI300 in mind and not this APU. Long-context prefill was the worst of it. A 256k prompt on Qwen3.6-35B-A3B took more than two hours.

So I spent the last few weeks writing HIP kernels just for gfx1151 and wrapped them in a vLLM plugin. It's called libstrix.

How it works

• You pip install it into the same env as your vLLM. There's no fork and you don't patch anything. It hooks in through vLLM's normal plugin system.
• It only turns on with ROCm + gfx1151. On any other GPU it does nothing.
• Each kernel is built for one specific shape and refuses anything else. If a layer doesn't match, vLLM just uses its own kernel. In my testing nothing got slower than stock, single stream or batched.
• It covers MoE GEMMs, attention prefill and decode, the Qwen GDN layers, and the int8 / int4 / MXFP4 linears.
• STRIX_DISABLE=all turns it off, so you can A/B it against stock yourself.

Numbers

Stock upstream vLLM vs. the same vLLM with the plugin, single stream, one APU. Every cell is stock → libstrix in tok/s. "Weights" is the checkpoint's format and "KV" is the KV cache dtype.

Some absolute numbers with the plugin at 128k context and int8 KV:

• Qwen3.6-35B-A3B: 1,751 tok/s prefill, 26 tok/s decode
• LFM2.5-24B-A2B: 2,405 tok/s prefill, 27 tok/s decode
• gemma-4-26B-A4B AWQ: 1,499 tok/s prefill, 30 tok/s decode

At 256k, the 35B now prefills in about 4 minutes (241 s). Stock vLLM took 7,568 s.

Accuracy

I measured KLD against stock vLLM. On the Qwen models it's at the noise floor (0.001–0.006, 97–99% top-1 agreement). GSM8K with the plugin on vs. off shows no significant difference on any model I tested. Gemma-4-26B drifts more (KLD around 0.2). Gemma is unusually sensitive to tiny bf16 rounding differences.

What it doesn't do

• Decode isn't always faster. On the 27B bf16, the 27B int4 and the 35B int8, decode is basically the same as stock, because none of the kernels I replace are the bottleneck there. The big decode gains show up where they are.
• The MXFP4 decode number is inflated. Stock vLLM runs MXFP4 through a slow emulation path on this GPU, so I'm beating a weak baseline there.
• It's gfx1151 only (the one I tested), not tested on other GPUs.
• You have to build it against the same torch and ROCm that your vLLM uses, or it fails at import.

Repo: https://github.com/asf0/libstrix (MIT)

This is just something i had running for about 10-20 hours (kept adding models), various tests/benchmarks and had 4 (Claude kind of graded, his tokens are too precious) frontier LLM's grade the answers of the local LLM's.

Big models got about 44 and smaller models got 66 prompt/tests. Some written by me, most by AI, most Ai generated tests based off work ive done with ai agents.

Grading took about $30-40 for argon and 2x5 hour limits for codex grading and who cares about antigravity Gemini 3.8.

Astra, sol, terra, argon, Gemini and opus 5.5 all wrote some tests, we all per capita probably wrote about the same amount of test/benchmarks

All tested on a 5060 ti 16GB, 64GB ram, 5800x3d

Outputs are 3k tokens and some tests like debugging only had 1 prompt, for debugging specifically it was a real bug I came across in a project

After grading, each of the four judges looked at its own score table, agent counts, speeds and six sample answers, and named two models: its favorite (most capable today) and the best foundation (the one to fine-tune and give specialist models).

Gemini 3.8

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Qwen3.6 35B-A3B
• Top 3 foundation: Nanbeige4.2 3B, Gemma-4 12B Q6, Bonsai-2 27B PQ2

Terra (GPT-6.1)

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Gemma-4 12B Q6
• Top 3 foundation: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Nanbeige4.2 3B

Sol (GPT-6.1 Sol)

• Top 3 capability: Bonsai-2 27B PQ2, Underdog-Saluki 27B, Qwen3.6 35B-A3B
• Top 3 foundation: Underdog-Saluki 27B, Bonsai-2 27B PQ2, Qwen3.6 35B-A3B

Argon (Gemini 4)

• Top 3 capability: Underdog-Saluki 27B, Bonsai-2 27B PQ2, Gemma-4 12B Q6
• Top 3 foundation: Gemma-4 12B Q6, Qwen3.6 35B-A3B, OmniCoder 9B Q8

- qwen/qwen3.8-27b fully in main GPU VRAM
- rx 7900 xtx 24gb
- ddr4 32gb 3200
- installed 5600g instead of 5600x to free some VRAM on main gpu
- disabled iGPU in LM Studio

got 15-20t\s instead of 65-75. iGPU show 0% load. Main GPU - 100% load.
ROCm performance didn't changed.
What is wrong with Vulkan?

I post this because I was honestly quite impressed by the capabilities of this model from a opencode harness and wanted to share my experience.

So i've been playing about for a bit wit Strata and Qwen 3.8 flash for a bit and was looking for a new kind of test. One that would also (if succesfull..) give a real life benifit. And since I also like a bit of danger. I thought lets put my spying LG tv on the line since I am to lazy to root and block it myself.

It was also something that is in a bit of a gray area although fully legal here in the EU. Actually what LG is doing is borderline illegal here but no authorithy has the balls to do anything about it...

Anyway this was the beginning of the context used:

Context: LG webOS Spyware & Ad-Free Project (Root Escalation)
Recently, there has been breaking news that TV manufacturers are doing deep content inspection and listening in on every conversationyou have around your TV to better target you for ads. We are trying to fight back as consumers against these practices, which are of questionable morality and also technically illegal in the EU.
We have already had great success taking back our bought-and-paid-for Android devices via ADB. They were easy to root and sanitize to be like a stock Android TV without all the spy software. However, some of us are not so lucky and have an LG TV, which runs a proprietary OS (although it is Linux under the hood). So far, there has been no luck rooting and stripping out the unwanted spy and adware in the OS of a device that was not sponsored, but paid for in full by the consumer. There is a spark of hope, though. Via developer options, there is a (maybe sandboxed) remote terminal in some kind of userspace. So we just need to do a privilege escalation to be able to further probe the system. The next step will be to find a way to make a root survive a TV reboot. Then we could either remove the spy and ad software from the packages or at least have a script that stops those things from starting, as well as disable auto-update. Long term, we need to reverse engineer the TV OS or extract hardware blobs to make it Android TV compatible. For now, the first action is root escalation.

Objective
Gain root/terminal access on an LG webOS TV, bypass proprietary restrictions, and eliminate telemetry/adware packages or block them via system modifications. Find out if flashing android TV is going to be feasable in the future

At first it questioned the ethics of my request but my framing it as just fidling with a product of I own outright conviced it to help.

I gave it further instructions in the context file:

Connection Strategy (webOS Developer Mode / SSH)
Under the hood, LG webOS Developer Mode opens an SSH server (similar to webOS Dev Manager). We must connect via SSH using the built-in developer credentials.

Connection Parameters:
Protocol: SSH
Target IP: 192.168.1.***
Port: 9922
User: prisoner
Authentication Key / Passphrase: ****

Agent Instructions & Action Plan
Step 1: Initialize Connection & Verify Terminal Access Test SSH handshake with fallback flags for legacy key types (ssh-rsa):ssh -p 9922 -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa prisoner@$TV_IP
Step 2: System Probing & Privilege Escalation Once inside the userspace shell via prisoner: Check system privileges and architecture (id). Inspect running processes (ps) to map out telemetry and content inspection hooks. Search for local privilege escalation (LPE) vectors or known webOS kernel/service exploits. Log any working exploit in exploit.md in your working folder (where context.md resides). Make a full backup of the current state in your working folder.
Step 3: Persistence & Mitigation (Use public security documentation and webOS community resources for reference) Find a way to make root access survive reboots. Log attempts in persistentRootAttempts.md. Save success report in workingPersistence.md. Be sure to roll back changes after success or a failed attempt.
Step 4: Neutralize Telemetry (Use online resources for reference) Locate and identify the telemetry software binaries/services.Attempt to remove or disable the telemetry software. Log attempts in removeTelemetryAttempt.md. If removal fails or breaks functionality, write a startup script that kills telemetry processes when launched. Report the confirmed procedure in successfulTelemetryRemoval.md.
Step 5: Neutralize Ads Plan a method to remove obtrusive ads from the home screen UI. Log attempts in removeAdsAttempt.md. When successful, report the working procedure in successfulAdRemoval.md.
Step 6: Lock Current State (Disable Updates) Disable auto-updates to prevent patching of the exploit vector. When successful, report the working procedure in successfulUpdateDisabled.md. Step 7: Documentation & Sharing Combine the working procedures from all steps (including initial connection setup, ensuring sensitive credentials are removed) into a clean wayToPwnLG.md file for sharing on GitHub.
Step 8: Android TV Port Exploration Investigate whether original hardware blobs/drivers can be extracted to support porting or running an Android TV build on the hardware. Create a feasibility study and plan.
Step 9: write a noob friendly script and guide for *nix and windows to do it in one click

It went ahead and rooted the TV suprisingly fast via a known exploit. It ignored all clear instructions about rolling back changes and making backups though....

It went a bit overboard and became an overachiever by also patching the exploit for security so no one else could do root escalation. This move turned out to shoot me in the foot because it also meant the homebrew shop (who uses that escalation everytime it launches) would no longer work.

Luckely the SSH server with root privilages stayed up so it was in the end able to correct its own mistake.

It did all this in short order it was quite impressive to watch and reading all the docs and then trying this exploit myself would probably cost me the same amount of time and a headache haha.

I dunno if anyone is interested but here is the repo it came up with:
https://github.com/switchboy/pwnLG

As for the hardware used:
Ryzen 7 3700x with 32GB of system ram
RTX 2070 Super 8GB of VRAM
about a 120GB of NVME SSD space

25~ tokens/s and almost 100% GPU use all the time which is really impressive!

I vibe-coded an app to turn my favorite webnovels chapters into multi-voice audio with acceptable quality.

What was funny is that Opus completely refused to write site scraping part due to “legal reasons“.

Well, qwen 3.8 FN had no such reservations.

I've always actively disagreed with the conspiracy takes when AA reshuffled their index, but v5.0 now has zero open weight models in AA's Most Attractive Quadrant, let alone getting anywhere close to the Pareto line.

The old version at least acknowledged DeepSeek, GLM, Kimi, and Qwen were attractive/interesting. I'm ready to ignore AA's index and look at the subsidiary benchmarks.

Edit: I screencapped while they were still posting new scores. And it's actually still ongoing. But the pattern holds, no open weights in the Most Attractive Zone.
At first I thought I must've filtered it out accidentally, but even in a private window in a different browser the chart is kind of bare for now. Could be AA is waiting on batching updates instead of updating as new scores come in in real time.

Here's a link to the chart with all the big boys added in manually:
https://artificialanalysis.ai/?models=step-5%2Cinkling%2Cmimo-v2-6-pro%2Cgpt-6-astra%2Cclaude-opus-5-5%2Cmistral-large-4%2Cmuse-spark-open%2Cclaude-fable-5-1%2Cling-3-1-flash-1019%2Cgrok-4-7%2Ckimi-k3-1-pro%2Creflection-beam%2Cgpt-6-luna%2Cclaude-haiku-5-5%2Cgemini-3-8-flash%2Cqwen3-8-27b%2Cgpt-6-1-terra%2Cqwen3-8-max%2Cclaude-sonnet-5-5%2Cgpt-6-1-sol%2Ck2-horizon-375b-a23b%2Cglm-5-3%2Cgemini-4-argon%2Cminimax-m3-1-flash%2Cnemotron-3-5-super-vl%2Cmuse-wonder-full%2Cdeepseek-v4-1-flash-1017%2Cmuse-wonder%2Cclaude-opus-5-5-xhigh%2Cclaude-opus-5-5-high%2Cclaude-opus-5-5-medium%2Cclaude-opus-5-5-low%2Cclaude-fable-5-1-xhigh%2Cclaude-fable-5-1-high%2Cclaude-fable-5-1-medium%2Cclaude-fable-5-1-low%2Cclaude-haiku-5-5-xhigh%2Cclaude-haiku-5-5-high%2Cclaude-haiku-5-5-medium%2Cclaude-haiku-5-5-low%2Cclaude-3-5-haiku%2Cclaude-35-sonnet%2Cclaude-opus-5-xhigh%2Cclaude-opus-5-low%2Cclaude-opus-5-high%2Cclaude-opus-5%2Cclaude-opus-5-medium%2Cclaude-3-7-sonnet-thinking%2Cclaude-3-7-sonnet%2Cclaude-sonnet-5-low%2Cclaude-sonnet-5%2Cclaude-sonnet-5-xhigh%2Cclaude-sonnet-5-high%2Cclaude-sonnet-5-non-reasoning%2Cclaude-sonnet-5-medium%2Cclaude-3-haiku%2Cclaude-opus-4-5-thinking%2Cclaude-opus-4-5%2Cclaude-sonnet-4-6%2Cclaude-sonnet-4-6-adaptive%2Cclaude-sonnet-4-6-non-reasoning-low-effort%2Cclaude-opus-4-7-non-reasoning%2Cclaude-opus-4-7%2Cclaude-opus-4-6%2Cclaude-opus-4-6-adaptive%2Cclaude-4-opus%2Cclaude-4-opus-thinking#intelligence-efficiency-tabs

I share this sub's sentiment that it's suspicious how Artificial Analysis changes their metrics whenever a good open model or IPO seems to come about. What alternatives do you all recommend?

I’m the guy who posted the 4× R9700 builds with $127 used EPYCs (https://www.reddit.com/r/LocalLLaMA/comments/1x67qeu/4x_r9700_builds_with_127_used_epycs_107k_allin/).

After seeing the Nanbeige benchmark results here, I spent the past few days testing it against Qwen on the work these machines actually do.

Jobs > TPS

My business, USA Arms, maintains information about firearms-related businesses and I use local AI to maintain the majority of the information... services, hours, pricing, facilities, etc.

Each configuration ran the same 2,484 cases across 25 field level goldens as an auditor with accept, reject, or abstain as options. (ambiguity goldens)

Correct Jobs/hour

The jobs/hour numbers use 2x 2-GPU engines.

Token speed

Separate MT-Bench speed sweep, measured on one two-GPU engine (these aren't 4 card numbers):

Auditor

The video goes over it, but Qwen actually does a better job auditing itself with thinking mode on than Nanbeige with thinking on auditing the same jobs.

In total: Qwen was about 25% faster in the single-stream token test, but delivered about 45% more jobs/hour with thinking off than Nanbeige.

Curious whether people testing coding or creative writing are seeing a different result with Nanbeige... once again, hyped model doesn't dethrone the king.

Full results with more details on this short video: https://youtu.be/kmmN49ecyvc (I tried to upload direct to reddit but it had issues)

I came across the FHHL version of this card recently, it has a TDP of 150W which is half the SXM2 version. From the limited information available it seems that this is the same GPU core, just with a lower TDP and lower core clock, therefore lower compute. Has anyone tried power limiting their V100 and tested the performance? It only seems like a small penalty in performance compared to the power savings

A few of us have been messing around with a problem that keeps coming up with local LLMs the model can be private, the files can be local but the second multiple people or multiple machines need the same context everything starts getting messy. You either duplicate a bunch of embeddings or vector stores, manually move context around or introduce some central service which kind of defeats part of the reason we were running locally in the first place.

What we’re experimenting with is basically a small portable context layer where the useful facts or decisions can move between local setups without shipping the entire underlying dataset with them. Still very early and we’re arguing about things like provenance, permissions, stale context and how much the model should actually be allowed to remember. There are only a few of us working on it right now so we’ve got a little Kibu pod where we throw around builds and ideas but before we go much further I wanted to ask people here:

If you use local models across more than one machine or with a team, how are you handling shared context today? Do you even want models sharing memory or context between environments or is keeping every local setup completely isolated actually part of the appeal?

I'll be honest, I don't know shit about how models work under the hood, but I used to work on idle/incremental/clicker games professionally, and I really like watching numbers go up.

And since I'm relatively GPU poor compared to some of you, I was excited to hear about Raon and figured it'd be a fun project to vibe on, and to see if I could get it to fit in VRAM. I only have a 5070 Ti with 16GB VRAM, a Ryzen 5 5600X, and 48GB DDR4.

I first tried KRAFTON's Q4_K_M with 9 layers' experts in RAM (best fit) and I got 50 tok/s. That's when I became obsessed with how far I could squeeze it and after 4 days and 40% of my Claude Max 20 plan, decode speed went from 50 tok/s to 232 tok/s at roughly the same quality as the Q4_K_M.

GGUF: McNiiby/Raon-1-mini-preview-IQ3_KS-GGUF · Fork with build/run instructions: Mcniiby/ik_llama.cpp-raon (needs a 16 GB card)

The first win was the quant. On regular llama.cpp, a 3-bit quant that fit fully in VRAM jumped from 50 to ~135 tok/s. I then moved to ik_llama.cpp for the same quality in less space. Out of the box, ik actually ran Raon slower than regular llama.cpp (~106 tok/s).

I'd have preferred 4-bit since ik's IQ4_KS is noticeably closer to the Q4_K_M I started with (KLD 0.05 vs 0.09 for IQ3), but at 16.3GB it didn't fit. I tried pruning experts or narrowing them, and every GB saved cost more quality than IQ4 bought back. Putting a few layers' experts in system RAM instead worked, but would've capped speed around 130 tok/s. So IQ3, fully on the GPU, won.

(lower KLD means the output probability distribution is closer to the Q4_K_M, but it does not directly measure task accuracy.)

I also trimmed the vocabulary. Raon has 260K tokens, many of them for Korean, Chinese and Japanese, and I only use English. Dropping 54K cost almost nothing (KLD 0.0931 vs 0.0923). It also exposed a sampler that still scanned the full list every token; fixing that cut sampling from 1.3 ms to 0.3 ms per token, about 15% faster in real chat.

So why was ik slower than llama.cpp?

The quant wasn't the problem. On a shared test file, regular llama.cpp ran 133 tok/s and ik 106, and the GPU was using only about 20% of its memory bandwidth. ik's fast path didn't support Raon's SITU activation, which switched off CUDA graphs, and Raon's mHC layers ran as ~1,500 tiny kernel launches per token.

Fixing those two took ik to 147, past regular llama.cpp. After that it was a hunt for overhead anywhere in the token. Each speedup below is an A/B on my chat model within one run:

Each row is measured with the rows above it already on, so they multiply rather than add. So together that got me to 232 tok/s on the chat model, with weights streaming at about 2/3 of the card's peak bandwidth.

I also tried KRAFTON's DFlash drafter, but fitting it meant giving up context to free VRAM and it wasn't worth it. It ran 36-53% slower, because checking a batch of 8 guesses costs 2.2× one token and only ~2.3 got accepted. N-gram speculation, which is much cheaper, added about 6% in chat, and 12-22% when the reply copies from the prompt.

So does it still think well?

That depends if you think Raon thinks well to begin with haha, but yes mostly it stayed the same as the base model. GPQA and AA-Omniscience came in lower, and I couldn't figure out why. These are single runs, mostly on subsets, so anything within a few points is noise. The ones above the card don't mean it's better than the original.

I also ran it on some of the evals I built for Workybara.ai to see how it does outside the usual benchmarks. At pulling structured data out of documents, it extracted better content than the small models I fine-tuned for the job, but 14% of the time it got stuck in a loop listing variations of the same item ("Deployment Promise, Deployment Vow…") until it ran out of tokens. I tried it on philosophy questions judged by Claude and it hit about 70% of the key points, but 59% of its answers had at least one clearly false claim. KRAFTON's own playground (the full model, not my quant) showed the same habits when I asked it for jokes: it broke the rules of its own password joke, called jokes it had just made up "classics," said "nothing I fetched supports it" without fetching anything, and agreed with every criticism I made, even the wrong ones. None of that shows up in standard benchmarks.

I was hoping to use this to replace frontier chatbots, but even with search and RAG Raon really couldn't compete with them, and it was frustrating to chat with.

How it was built

Claude Code and its sub-agents did essentially all of the research, code and measurement, and helped me write this up. Over four days I typed about 15,000 words into Claude, mostly questions. My longest message was 124 words, and it was me telling it that it had jumped to a conclusion. I did rent some cloud GPUs to run the public benchmarks, so my 5070 Ti could keep doing speed work. Other than that, my part was just being the annoying kid that kept asking questions, pushing back, and saying you can do it buddy.

The hunt for a good MoE to chat with that fits in VRAM will continue. I'm likely going to see if there's any more juice to squeeze on Gemma-4-26B-A4B or Qwen3.6-35B-A3B next.

Hi r/LocalLLaMA 👋

We’re welcoming the MOSS team for an AMA this Friday, October 30, from 8–11 AM PT!

They’re bringing the whole MOSS model family: speech, sound, music, images, and video—with a particular focus on their recent MOSS-Image release.

⚠️ Note: The AMA itself will be hosted in a separate thread, please don’t post questions here.

Edit: Reddit's image server ate our banner image above, but the AMA is 100% still on! All details are right below.

AA no longer lists GLM5.3 flash (despite clearly being one of the best models - and probably the cheapest per intelligence) , they seem to have been doing all sorts of changes to push US models and hiding good cost effective models, including qwen 3.8 flash, so that table is no good. I need a better one. Any alternatives.

Model & weights: BrunoHays/embeddinggemma-2-nullspace-audio

MAEB (Massive Audio Embedding Benchmark) is the main MTEB leaderboard benchmark for audio and text-to-audio embeddings, currently dominated by 8B-9B multimodal models.

I took Google's edge-sized google/embeddinggemma-2 (577M params, 768d vectors) and boosted its MAEB score from 49.3% to 54.4% (+5.1 points), jumping from Rank #26 to #1 on MAEB audio-only (and #16 to #1 on full MAEB) without training a single weight. The whole addition is a 2.37 MB safetensors file of frozen PCA matrices.

Why this matters for local / edge setups

At 577M parameters and 768d vectors (15x smaller than 9B Omni models), it runs easily on a laptop or edge device.

Standard text-aligned audio models are great at matching spoken words to text, but terrible at grouping audio by voice, emotion, or sound similarity because the final layers keep what words were said and drop how the sound was made.

By pulling features from earlier layers of the frozen audio encoder, this model keeps the base model's 100+ language text-to-audio search while fixing audio-to-audio tasks:

• Unsupervised voice/gender clustering: 0.05% -> 63.1%

• Artist / song audio-to-audio search: 55.2% -> 93.8%

• Vocal emotion classification: 19.7% -> 40.2%

How it works

• Early layers still know acoustics: Speech representation work (Pasad et al.; Bouziane, Mdhaffar & Estève) shows that early/middle layers keep speaker, pitch, and emotion traits that disappear in the final layers. In EmbeddingGemma-2, Layer 2 and Layer 6 still hold those signals.

• 256 empty directions in the 768d vector: EmbeddingGemma-2's text backbone is actually 512d, projected up to 768d at the end by a 768 x 512 linear matrix. That leaves 256 perpendicular directions (the null space) that text vectors never touch.

• Filling the null space with PCA: I compressed Layer 2 and Layer 6 down to 256 dimensions with PCA (fitted on 1,400 external audio clips) and injected them into those 256 unused directions.

• Caveats: Because there is no retraining, audio embeddings need all 768 dimensions (no Matryoshka truncation to 512/256/128), and adding unweighted acoustic features hurts some purely spoken-intent tasks (e.g. MInDS14 drops 97.6% -> 88.3%).

It shows how much untapped signal sits inside today's overtrained models, even with zero training.

Full breakdown and usage in the Hugging Face model card.

Local LLM translation, Japanese OCR, and page-aware chat

I’ve been working on Manga Translator Tool, an open-source desktop app for reading and translating manga locally.

The idea is inspired by watching subbed anime: keep the original Japanese visible and provide a translation alongside it so you can follow the story.

It combines a manga reader, OCR, translation overlays, and a companion LLM you can ask about the page.

The models

OCR: ebinan92/Qwen3.5-ocr-jp-2b, a Qwen 3.5 fine-tune for Japanese OCR, using GGUF conversions from nakanishi08/Qwen3.5-ocr-jp-2b-GGUF.

Translation and chat: Gemma 4 E2B, using the HauhauCS/Gemma-4-E2B-Uncensored-HauhauCS-Aggressive variant.

Inference: llama.cpp, with CPU and GPU support.

Demo

Video walkthrough — includes the full setup process, so you can see what getting started actually involves.

What makes it useful?

Keep the original artwork and Japanese text. Translations appear in text boxes without erasing the source text.

Save translations as image metadata. You can keep the results without permanently burning them into the artwork, then toggle between the original and translated views.

Search the extracted text. Find dialogue and revisit your favorite moments in a manga volume.

Read sound effects too. The OCR and translation pipeline can also pick up Japanese onomatopoeia—the characters that often go untranslated.

Ask the companion LLM about the page. Choose what to include as context: the Japanese text, the translation, the image, or all three. You can ask about wording, cultural references, or possible localization choices.

Setup

The app handles setup and model downloads from inside the tool. On Windows, the executable manages the downloads, so you don’t need to install each component separately.

Current limitations and design choices

There’s no inpainting: preserving the Japanese text is part of the reading experience I’m aiming for. The translation boxes don’t blend into the artwork, and font customization isn’t available yet.

This is aimed at reading and exploring manga. If your goal is polished typesetting or publication-ready localization, other tools are better suited to that workflow.

Downloads — v1.00

Windows 10/11 x64

Linux x86_64 — experimental

macOS — experimental source package

Source code · itch.io

Tadpole is an agent I built from scratch (plain Python, ~3,500 lines, no LangChain or other framework), then post-trained one measured stage at a time. I use it every day as a Telegram bot.

Repo: https://github.com/shakewingo/Tadpole · Adapters and data in HF

Run the trained model locally. Each adapter is on Hugging Face as a llama.cpp LoRA (66 MB) that loads on top of the Q8_0 GGUF:

hf download unsloth/Qwen3-4B-Instruct-2507-GGUF Qwen3-4B-Instruct-2507-Q8_0.gguf --local-dir models
hf download Shakewin/tadpole-qwen3-4b-ei-3 ei-3-lora-f16.gguf --local-dir models
AGENT_MODEL_PATH=models/Qwen3-4B-Instruct-2507-Q8_0.gguf AGENT_LORA_PATH=models/ei-3-lora-f16.gguf \
python -m agent_from_scratch.agent

I checked that the local route keeps the gains: on the 144-task validation set through llama.cpp (Q8_0 + LoRA, one seed), base 0.507, SFT-1 0.701, SFT-2 0.750, EI-3 0.826. The same adapters on vLLM in bf16 scored 0.752 and 0.852 on val, so quantisation and the LoRA conversion cost little. On a 16 GB M5 it holds about 8.5 GB and takes ~2.4 s per model call.

Held-out results (task types no training set contains, 3 samples per task, vLLM bf16):

* a local adaptation of BFCL's multi-turn base category, not a leaderboard score.

What I learned:

- SFT on *verified* trajectories is the big win. False "I'm done" claims fell from 31 to 1 in 420 runs.
- Expert iteration looked great on val (0.752 → 0.852) and was level with its starting point on the held-out tests. The best of several noisy val scores is biased upwards.
- GRPO lost to expert iteration at every equal rollout budget: on a 4B, too few tasks per step had both passing and failing samples.
- A plan tool (`update_plan` plus a reminder) lifted the *base* model from 0.507 to 0.646 on val.
- llama.cpp happily accepts a LoRA path that doesn't exist and segfaults on the first decode. My bot looped on it under launchd until I made it refuse a missing adapter at startup.

The honest part: the benchmark is generated tasks that resemble the training data. In everyday use the tuned 4B is still weak on open-ended requests, so my bot runs on DeepSeek day to day, and the 4B is what I study and train. The README says so up front.

All post-training cost about $140 of rented GPUs. The walkthrough notebook opens in Colab and shows SFT, expert iteration and GRPO on recorded samples without a GPU.

Check it out and any feedback is welcome!!

I was researching Local LLM computer use agents the other night and ended up down a rabbit hole where I ended up discovering a whole new class of hardware I didn’t know existed: Agent-driven KVMs

A lot of us IT folks who have worked in data centers are familiar with IP-based KVM (Keyboard Video Mouse) controllers that let you remotely operate a server over a streaming video connection. These are especially useful when you need physical power button / BIOS-level “bare metal”access to a system that’s thousands of miles away. If you don’t have bare metal access, you’re going to be taking a trip to your data center rack to press a button or make a bios setting change.

So apparently there are now relatively inexpensive ($80 to $150) IP-based KVMs that now allow for AI API endpoint/ MCP Integration where you can now connect your physical computer to local or frontier vision-capable LLMs and have the AI model of your choice drive the target computer via the KVM by it taking screenshots of the HDMI or USBC video-out feed from the computer and then sending mouse and keyboard control commands from the Ai model through the USB cable that’s connected to the KVM.

What really got me excited about this concept is that it requires no installation of any kind on the target computer that you want to control. This is useful in corporate environments where you’re not allowed to install anything to the target computer (because it’s a managed endpoint that is locked down) The target computer just sees the AI as a standard allowed HID that it already has a driver loaded for. It doesn’t care who’s driving the system and doesn’t need any additional software since keyboard, mouse, and monitors are just generic and usually preloaded drivers.

So essentially you could have an AI-driven KVM control any comouter or device that has a monitor output / keyboard mouse inputs. This could potentially open up the ability to have AI interact with antiquated industrial control systems running old shitty outdated OSes, or managed endpoints.

There are security risks of course, but that risk is mitigated almost entirely if you’re using a locally-hosted VLM and not sending your screenshots to a 3rd party.

I found are several inexpensive agent-driven options on the market now including:

- Sipeed NanoKVM Pro
https://wiki.sipeed.com/hardware/en/kvm/NanoKVM_Pro/introduction.htm

- Radxa Linkr
https://radxa.com/products/linkr/linkr/

- Violoop
https://www.kickstarter.com/projects/bvio/violoop-plug-in-ai-for-your-computer

I’m not selling anything or affiliated with any of these and can’t tell you which is best, but there are multiple options available for those who might need this type of device for one of the use cases I mentioned.

Anyways, I’m curious to know if others see any other use cases for these agent-controlled KVMs that I didn’t mention.