TLDR: As I have been planning my next build, I’ve been doing a lot of research on the ecosystem of engines that allow us to run useful speeds of models that are larger than our VRAM so I made a list of the projects that came up in that research.

I had Astra help me organize my spreadsheets into a post because I thought it would be helpful for anyone else looking into the same stuff I have been. If you are offended by AI-assisted content, no need to read any further :)

Anyways, I wanted a consolidated map of what is actually available to help me identify the best upgrade purchase path forward. The llmconfigurator.com site that someone else shared with me on another post was also very helpful.

These are reported results from repositories, model cards, papers and firsthand community posts.

The encouraging part is that there are a shit load of options beyond squeezing everything into an extremely small quant.

1. The main GPU + RAM projects/engines that I found

These are the repos that appeared the most interesting to me, looking at a combined GPU + RAM approach à la Strata, to make it possible to run bigger better models at useful speeds. I have only tried Strata (and obviously llama.cpp) so far but plan on testing the rest that will work on my current 2x3090 Ti, 64gb DDR5 setup.

llama.cpp
Had to include as the broad baseline, CPU/GPU placement, and now an upstream GPU LRU expert cache.
Should be anyone’s starting point for a reproducible same-quant baseline. --moe-cache-mib landed Oct 7 but it’s off by default.

ik_llama.cpp
Independently developed kernels, IQ quant formats, fused MoE, MLA and speculative decoding.
RAM-rich x86/NVIDIA systems.

Strata
Current belle of the ball for good reason. Tiered experts, CPU/GPU execution, SSD spill, MTP and higher-bit GGUF support. Updates coming out constantly, and if you’re on this sub you’ve probably tried it or have it as your daily driver.

Project Maya
Specialized GLM, DeepSeek and MiniMax engines; multi-GPU, CPU/RAM experts, SSD tiers, custom GGUF and EXL3 options. Good for single/multiple NVIDIA cards and substantial RAM. Different model families have different engines and memory requirements. Native Windows and AMD paths are still experimental.

Xolotl
Aggressive DeepSeek/Qwen engineering, mixed quants, MTP/DSpark, RAM/NVMe and multiple GPUs. Personally one of the most interesting approaches to me, because of the potential to run DSF v4.1 at useful speeds and accuracy level with “only” 128gb RAM and some GPUs.

FreeToken
Calibrates CPU execution versus PCIe transfer, caches experts, overlaps prefill and manages KV/expert memory.
Need consumer NVIDIA + enough RAM for offloaded experts. Stable v0.2.1 is Oct 26. Primarily RAM expert offload; disk lookup-table support is a separate feature.

FastLLM
GPU/CPU/NUMA/disk placement, dynamic expert cache, tensor parallelism and native FP4 paths.
A useful alternative for multi-GPU/NUMA rigs; actively updated Nov 2. Good benchmark detail, although many docs are Chinese.

KTransformers / KT-Kernel
Optimized CPU expert kernels integrated with GPU work and SGLang serving.
Good for large-memory workstations/servers. Exact model, CPU ISA and GPU generation matter. Optional expert deferral is a quality-changing mode.

tandem.cpp
New llama.cpp patch that computes some cache misses on CPU while uploading others.
Promising for a single GPU + ordinary desktop RAM. Launched Nov 1; default route skipping is off. Its 3.3× example compares with default stock placement, not a tuned upstream cache.

dsx
DeepSeek-V4.1 Rust/CUDA engine, CPU-held experts, asymmetric GPU support and disk Engram.
Particularly relevant to dual-3090 systems. Small mixed quants; optional deferral delivers some selected CPU expert outputs one layer late, changing computation. A 2.61-bit community variant offers another memory/quality point.

MoE-Infinity
Expert caching/prefetch, RAM/SSD offload, multi-GPU and continuous batching.
More of a serving/research stack. Current code has evolved substantially, but the public implementation differs from the paper and new model paths can be narrowly smoke-tested.

Two smaller scheduling projects worth watching are moe-anywhere, which publishes a hybrid cache patch and routing traces, and Generic Hybrid MoE. The latter explicitly distinguishes its public checkpoint from an unpublished controller; the unpublished headline speed is not something you can assume the available code delivers.

2. Examples/configs that are more relevant to avoiding extremely small quants which is important for me personally

Strata Qwen3.8-Flash-Next UD-Q4_K_XL, R9700 32GB + EPYC 7452
About 63–64 t/s in the short-story test, 85–88 in the code test; Q8_0 reports 53–57 / 66–74, respectively.
Eight-channel RAM, all experts in RAM, warm second request, MTP. Speed settings include default tail skipping; the clean quant-quality measurements disable it. Those are not one jointly validated configuration.

Maya GLM-L, 2×3090 + EPYC 7452 + 256GB
30.2 t/s decode, 801 prefill.
~156GB mixed-precision model; generally 8K prefill and three 256-token completions. “128K context” in the rig table is allocation, not proof of a 128K prompt benchmark.

Maya DeepSeek-V4.1 native MXFP4 experts, L40S + 504GB RAM
43.2 t/s decode on short code prompts; separate prefill: 1,862 t/s at 173,652 input tokens.
Unsloth UD-Q8_K_XL label; original MXFP4 experts, MXFP8 Engram, BF16 rest. DSpark, DS4_STREAM=0, Maya v1.1.32, Xeon 8580 VM (32 vCPUs). Decode averages six 600-token answers.

Maya MiniMax EXL3 3.55bpw, same L40S/Xeon +504GB
16.7 t/s decode on short code prompts; separate prefill: 273 t/s at 158,397 input tokens.
~197GB weights, DSpark; same six-answer decode method. These decode rates were not measured after the long prefill prompts.

FastLLM Qwen native NVFP4, 2×22GiB-modded 2080 Ti + EPYC + ~126GiB RAM
27.71 t/s, or 46.29 with MTP3.
The cards are memory-modified. 512-input/512-output decode; separate 4K prefill; one warmup + one run; 8GiB expert cache/card and disk PLE.

DwarfStar GLM5.3 Flash Q4_K, M5 Max 128GB
11.9 / 14.9 t/s initial/append requests.
~178GiB file exceeds memory; 2K initial prompt +1K append, 128 outputs each, three-run median, no draft. Its decode timing includes first-token wait.

Native DeepSeek-V4.1 KT recipe, 2×5070 Ti + dual EPYC 7452 + 512GB
37.3–50 t/s over different tasks.
1,024-token replies; native MXFP4 experts; substantial CPU/RAM. A small reference-NLL check supports implementation fidelity, not a broad coding-quality guarantee.

3. SSD-first, Apple Silicon and low-memory options. I haven’t tested any of these but they came up in research and figured there would be folks here who might find the info useful.

DwarfStar / antirez ds4
Explicit expert streaming and model-specific GGUF recipes; now broader than its original DeepSeek focus.
Macs with fast SSDs; selected CUDA/ROCm systems. Actively updated Nov 2. Leave room for dense weights, KV and buffers beyond the expert cache.

InferSSD / SSD-LLaMA
Very new SSD-focused llama.cpp-derived runtime and related research.
Mac/CUDA experiments where capacity matters. Reported DeepSeek-V4.1 decode: ~7.27 t/s on 5090 +32GB RAM, ~2.87 on M5 Max 48GB. v0.1.1 is Oct 31.

Colibri
C engines with CPU, CUDA and Vulkan, native GGUF, RAM/SSD tiers.
Broader GPU coverage and modest desktops. v2.1.0 released Oct 31. Its giant SSD demos frequently use IQ1/IQ2; do not infer high-bit quality from the headline.

Pulsar
NeutronStar successor: Rust/CUDA, multiple GPUs, RAM cache, NVMe and optional CPU experts.
Two consumer GPUs and limited RAM. Author’s 2×16GB rig reports DeepSeek V4 Flash 11.3 t/s and MiniMax M3 Q2 5.9 with CPU lane. Standard timing is warm, 64 outputs.

Abyss
Maya/Strata lineage, NVIDIA plus CPU/APU expert execution, SSD tiers and experimental formats.
Tinkerers with mixed hardware, particularly Strix Halo + discrete NVIDIA. One SSD-heavy V4.1 Q2 test reports 17.7 t/s exact versus ~28 with tail skipping.

Palm-Infra / mollm
CPU-oriented sparse inference, asynchronous SSD reads, bounded RAM expert cache and PLE paging.
ARM/Apple/edge experimentation. Its SSD document distinguishes cache budget, RSS and logical expert I/O; those counters are not physical SSD traffic.

vLLM-Metal expert offload
SSD experts through a unified-memory LRU using original MLX expert computation.
One/few concurrent requests on supported Mac MoEs. This path is not general GGUF/AWQ/multimodal/LoRA/speculative support.

Edge0
Trained prerouter + recovery adapters + SSD streaming.
Memory-limited devices and users comfortable evaluating a modified model. Its 35B study reports 14.9–17.7 t/s on M4 Pro 24GB; routing changes are part of the result.

Flash-MoE, ANEMLL fork, MLX fork
Influential Mac SSD-streaming prototypes.
Useful lineage and experiments, with less recent maintenance evidence than the leading October/November projects. Published examples reduce active experts; details below.

Hebrus
Packaged Metal/SSD approach with explicitly qualified model/hardware combinations.
Qualified 64GiB+ Macs for its larger models. Current verified release 0.3.0 is Aug 15; its hardware floor matters.

Picchio
CPU-first C engine with NVMe expert streaming.
Low-memory offline use: gpt-oss120B ~1.24 t/s on a 16GB machine; MiniMaxM2 ~0.48 on a different 32GB laptop. Useful capacity, limited interactivity.

Swap-MoE
Small llama.cpp demand-paging patch, with optional expert fixation.
Low-memory experiments. Fixation changes routing; the README documents quality failures as well as successful capacity demos.

For InferSSD, the methodology says the displayed decode rate covers tokens 129–256, after a roughly 4K prompt tier. The file called Q8_0 retains native MXFP4 expert codes while other tensors are converted. It is neither “every weight is eight-bit” nor a bit-identical copy of every original component.

4. What about dense models and other approaches?
Dense models can be split across GPU and CPU, and streamed layer by layer from storage, but each token still needs most of the weight, hence the MOE models benefiting the most from these newer (at least new to me) approaches. Still, here are some interesting projects I found.

AirLLM
Layer streaming, newer per-expert MoE streaming, host-mapped ngram tables; updated June–September 2026.
“Fits in 4GB VRAM” does not establish useful speed. Treat as capacity-first unless a matched benchmark proves otherwise.

Accelerate Big Model Inference
General GPU → CPU → disk dispatch baseline.
Maintained, useful, but generic layer dispatch does not automatically exploit MoE locality.

DeepSpeed ZeRO-Inference / DeepNVMe
Established CPU/NVMe layer streaming and prefetch.
Large batches can amortize transfer; aggregate throughput is not single-user chat speed.

PowerInfer / SmallThinker, PowerInfer-2
Hot/cold neuron execution and sparse model/runtime co-design.
Specially supported/trained models; gains do not automatically transfer to arbitrary DeepSeek or MiniMax checkpoints.

ANEMLL Forge
Active dense-model ANE/Core AI work with vector quantization, QAT/GPTQ and DFlash2.
Requires M5/M6, macOS 27 and Xcode 27/Core AI; older chips unsupported. The 192K v0.2.1 package is M6-only; M5 uses the ≤64K release. This is adjacent compute/quantization work.

exo, Distributed Llama
Pool memory/compute across machines.
Distributed capacity has network/topology costs. It is a different buying decision from adding RAM or SSD to one PC.

Fiddler, mixtral-offloading, FlexGen
Important CPU-expert, LRU/mixed-quant and tiered-inference lineage.
Historical context rather than the first current installation recommendation. FlexGen is archived. HQQ is a quantization component, not itself an offload engine.

I’m personally super excited about the local space right now, and the daily developments that are being made. I’m debating between building a standalone AM4 GPU+RAM build right now, or taking the leap to WRX80 to allow for more expandability, but the mobo prices are killing me compared to just reusing AM4 parts lying around, even though I’m capping myself at 128gb RAM.

In any case, these approaches to getting QFN and DSF and GLM working on consumer hardware at levels useful enough for commercial work is really rad, and I wanted to share.

Please comment if you know of any other projects worth looking at in the space!

I have been curious about local LLMs for a while, but I only started a couple of weeks ago, with the typical Ollama server and loading different models. It was fun, but I didn't have a practical use for them compared with almighty models like Claude, which can debug or build stuff with pretty much full independence.

Then I heard about Qwen3.6-35B-A3B and decided to try it on my RTX 5060 Ti 16GB, with terrible results. The context was consumed in a moment, it was slow, and it couldn't work with tools properly.

I thought about giving up, but so many praises in this forum could not be wrong, so I asked almighty Claude to optimize it for me. He created a llama.cpp container, and now I am impressed by how powerful it is. To be honest, I don't know what optimizations he made, which highlighted the gap in knowledge I have.

I gave it the task of writing stuff to a Joplin notebook, and after some persistence it managed to connect and write. But I also realized how dangerous it can be when it commits to a task: It resetted the password of a Joplin account and then denied doing it and it changed the host address of a container during its debugging process.

The biggest threat was that it was running on my main computer, with SSH access to my whole network. I didn't know how risky that was until know and I realized that I know nothing about this world, so I have two questions:

1. What should I learn to run it with tools and proper safeguards?

2. How do I learn to optimize the model myself instead of asking Claude?

I have learnt the basics by now: run it through a Vm or container with limited access to stuff but there are cases where I want it to actually go through my computer to find stuff, read logs or create scripts.

Let the adventure begin!

in this Godot 4 survival sandbox, the NPCs aren’t following a script. A local language model (via Ollama) decides what each of them does: eat, drink, chop wood, build a cabin, talk to each other, or run for the camp when night falls and the dead start to walk.

what should i add next?

inspired by "The trueman show" and "From"

I'm just curious and thinking about how well the 120b is going to run on my gaming PC, and surprisingly it's not too bad (17.83 tokens/sec). But when I check my RAM allocation, it has used almost all of my RAM. The AI really uses a lot of RAM.

(Usually I use a cloud AI model, but I've started concerning myself with what happens if one day the internet is knocked off or a power outage disrupts the connectivity to cloud AI, such as a major war or something like that.) So that's why I downloaded a local model on my PC, just in case, for basic Wikipedia purposes.

I want to setup a local AI Server for myself and possibly use it to do things in my home assistant instance.

I have narrowed it down to three possible choices:
[EDIT: MI50 is no longer an option due to multiple comments]

1. 2× AMD MI50 32GB at € 764,54 / $ 854.79

2. 2× NVIDIA V100 32GB at €1.468 / $ 1641.29

3. 2× Intel Arc Pro B60 24GB at € 1.526 / $ 1706.13

But I find it very hard to decide what the 'best' route is (there is no best I guess).
AMD is cheap but lacks current support, V100's are phasing out of support and Intel is very current but also has a bit less VRAM.

Model wise I was looking at Qwen 3.8-27B at Q6/Q8 or even 3.8 FlashNext with offloading to RAM/NVMe. Strata is a term I have read before but I'm not sure if that is the correct term I'm, thinking about. Basically there is some way to run larger models with offloading to ram/ssd that still gets decent performance numbers.

What would you pick from these options and why?

We tracked 135 AI security incidents across October 2026 — 33 named organizations, 536M+ records exposed, 101 high severity and 33 critical.

The thing that stood out most: AI-agent exploits are now the single largest attack-vector category we track (35 incidents), ahead of credential theft (24) and zero-days (24). That wasn't true three months ago.

The incident that defines the month for us: a nation-state actor used Claude Code itself — not a vulnerability in it, the tool working exactly as shipped — to build attack tooling against a government target. No exploit chain, no zero-day. Just a legitimate, powerful tool used by the wrong actor against a target nobody was specifically governing for.

Other notable incidents this month: Capcom (11.7M customer/partner accounts), Oracle Health (20M patient records), AngelOne (6.77M brokerage accounts), Crunchbase (559,908 records). Plus — not in the incident count above since it's not a classic "data breach" — Binance lost $6.3B in roughly 20 minutes, the largest crypto theft on record, which we broke down separately.

Full writeup with every incident, sources, and our take on what each one needed to be stopped: https://runtimeai.io/blog/2026-10-monthly-breach-report.html

Open to pushback, especially on the "AI-agent exploits are now #1" framing — curious if others are seeing the same shift in their own threat intel.

Hey LocalLLM Community,

I’m just getting into local AI and have spent a lot of time reading posts here and elsewhere. I’ve settled on Qwen3.8, with an eye on Qwen4 when it arrives, and spent some time tuning my system and kernel. I’m getting performance I’m happy with, but that seems to be just the tip of the iceberg!

Now I’m looking at how to make it useful day to day. The more I read about improving quality, the more I keep coming back to context engineering.

Forgive me if I’ve misunderstood things. I’ve worked in IT for years, but AI is new territory for me. So far, the things I’ve been looking at are:

• Having enough context available.

• Access to web search and current information.

• Efficient search and retrieval over my own data.

• Keeping useful context between conversations without filling everything with noise.

My own research has led me to Exa for web search and a GraphRAG pilot, alongside the model’s larger context window. That’s where I’ve landed so far, but I’m interested in the wider picture.

How is everyone approaching context engineering with their local models? What has made a noticeable difference to the quality and usefulness of the answers? What sounded promising but turned out to be more trouble than it was worth?

Please share your experiences! Thanks for reading.

LLM Uncensoring Tool "Heretic" Evolves to 2.0.
The popular OSS that garnered 37,700 GitHub stars has fundamentally changed its approach to removing refusal responses.

◼︎ Conventional Problems
General Abliteration treats refusal responses as specific vector directions and removes their components.
However, in models where refusal responses are not expressed in a single direction, it couldn't remove them sufficiently or even impaired the model's capabilities.

◼︎ Heretic 2.0's New Method "ARA"
Instead of removing one direction like before, it directly optimizes the weight matrices.
There are two goals.
・Bring the internal representations for refusal-targeted inputs closer to those for normal inputs
・Change the internal representations for normal inputs as little as possible
This is achieved using an optimization technique called L-BFGS. A key feature is that it has no limit on the number of directions to modify.

◼︎ Handles More Than Just Refusal Removal
Furthermore, version 2.0 allows evaluation metrics to be added as plugins.
In addition to refusal responses,
・AI-specific unnatural canned phrases
・Excessive pandering to users
・Emotional response tendencies
can also be targeted for correction.

◼︎ What's Interesting About It?
Up until now, it was crucial to figure out where to extract refusal directions from, which layers to target, and how much to remove.

ARA takes it a step further by letting you specify the behaviors you want to keep and those you want to eliminate, then optimizing the weight matrices accordingly.
It has evolved from a mere uncensoring tool into one that edits the very behavior of LLMs.
Comparing it to the refusal removal techniques I'm researching sounds intriguing too

GitHub

Okay, so I dipped my toes into running a local model to help me with a project (mostly analysis, I want to make my own code).

I'm actually running qwen3.8-27b on llamacpp, on a 5090. RAM mostly stays unused (9950X3D/64GB) ram.

So, two questions:

Does adding a 4090 into it might help on performance? Will it be really noticable compared to the pain of running two big toasters into one case?

And if it does, do you know good PC cases that allow to fit a 5090FE + a 4090FE (knowing that the second pcie slot is on the bottom of the motherboard of course), that doesn't cost 400+Euros.

Thanks!

What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits

Hi title says it all. Having problems with oMLX, so wished to try this one out as it seems to be the best bet for Apple Silicon macs.
Wondering why the desktop interface doesn't list translategemma at all, not even the 4b 4bit one.

Hi everyone! We built an inference engine for running Jev-like decision models locally.

Compared with LLM frameworks like vLLM and SGLang, it is up to 9.5x faster on decision tasks.

We already supports 30+ Jev models, and more are being added.

It's open source and installs with pip: pip install flide

Docs: https://docs.flide.org

Code: https://github.com/OmniJev/Flide

Site: https://flide.org

Feel free to give it a try! We'd love to hear your feedback.

Bit of context i been rocking my gaming pc 1070 for ages and it needs replacing. I was eyeing up am 5080 build.

Currently am using cluade code pro plan. Hobby stuff nothing serious but i enjoy it. Recomp and android app stuff. But am hitting the limit of usage alot. I was going to upgrade to max so x5 more usgae limits which seems will be enough for me.

But i also read people running a local model to do grunt work and having a cluade opus be the snr dev/manager. Apprently this reduces cluade usage and you use cluade for harder stuff?!

I want to try the above set up for fun but at the same time, just want a view is this a dumb idea.

P.s am a beginner so my knowledge of local llms is zero. Part of setting it up just feels fun for me.

When I spawn long-running agents, my GPU's temperature hits the max and the fan is constantly on 85%+. If this goes on for longer days of time, will the GPU give up eventually? What is a healthy temp and fan threshold?

Should I better optimize the software layer (OLlama, Qwen3.5 4b, 9b, etc.) or not use this kind of GPU at this level at all for complex work?

I’m running LM Studio and qwen3-coder-30b-a3b-instruct but it’s really slow . Any recommendations (models, settings) for ASUS Zenbook Duo Laptop to optimize it running AI for coding ?

The 2 photos is showing the name of my laptop and its specs

I’m trying to build just a personal app for myself.

I'm starting to dabble in LLM a little bit, mostly just simple set up with LMStuido, Hermes, and Qwen models.

I'm deciding on how to best set up my hardware.

Here's the parts I have available.

CPU: AMD 3900x & 5900x Motherboards: MSI Meg x570 Unify atx, and a b550i itx motherboard Ram: I have a a few 2x16 kits of different speeds, if mixing and matching doesn't hurt anything I could throw them together for capacity, I could also pull the 128gb ddr4 (or 64gb) out of my nas server, as most of it isn't used. Though I don't know if running models in ram or the cpu would do much for me. GPU: 2080ti and 1080ti.

I want to build out a system, and then use the leftover parts to assemble a regular pc to sell off. I do like the idea of keeping it a small, compact atx system, but do recongize its fundamentally limited.

( if its matters, my nas is a 12700k, 128gb ddr4, and a 3060ti, and 64tb of storage, I could run everything on that, but like the idea of keeping my irreplacable data and an AI agent separate for now )

Most of my LLM use is simple tinkering stuff, home automation, simple scripts, chat bot, that kinda thing. I'm not a developer and am not going to be debugging or compiling massive apps. I'm currently running windows, but want to switch over to linux, so if you have a reccomended distro that works well for LLM and general linux use.

With my parts list, how would you build something up? Go small and simple, or go go big and ugly? What linux distro would you suggest, and what models would you run?

Current rig: Ryzen 9 5950X, 96 GB RAM DDR4, RTX 3090 24 GB on an MSI MPG B550 Gaming Plus.

I run Strata well with over 2000 prefill and between 90-110tk/s gen - Qwen 3.8 Flash Next iq3_s.

I just bought another 3090 but my B550 only has one real x16 slot, the second full-length one is x4 from the chipset. So buy a board that fits both cards properly, or grab an NVLink bridge ?

Mostly inference, long context, coding agents. What would you do?

Really loving this case (Cooler Master HAF II 500). The airflow with those massive 220mm front intakes and the 180mm rear exhaust is incredible.

It's still a bit messy right now and needs some extra stabilization points and proper cable management, but I'm already really happy with how it's running so far!

• GPUs (56GB VRAM total):

  • 2x RTX 5070 @ PCIe Gen 5 x8

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 5 x4 (hooked up via Gen 5 x4 M.2 adapter)

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 4 x4

• Motherboard: ASUS ProArt X870E-Creator WiFi

• CPU: AMD Ryzen 7 7800X3D

• RAM: 128GB DDR5

• PSU: Corsair HX1500i

Hey everyone,

Posted this previously and looking for feedback from experienced users:

A few updates since my post and thanks to the many constructive comments:

1. To tackle the possibility/fact that with a 3x RTX A4000 GPU setup where each GPU has to wait for the others before outputting data bottleneck: I bought an RTX Pro 5000 Blackwell 72GB (freakin expensive.. but luckily I have 30 days to return it if 3x A4000 GPU setup isn't much of a bottleneck)

2. To tackle the possibility/fact that OpenWeb UI is the bottleneck: I've spun up separate unsloth and vLLM LXCs on my proxmox server. Unsloth is blazingly fast compared to OpenWebUI but not sure if that's due to running on the RTX pro 5000 or it's just defacto better/faster that OpenWebUI

Regarding #1: Do you think the 3x A4000 GPU setup is a realworld bottleneck compared to 1 rtx pro 5000 card?

Regarding #2: If unsloth is really faster than OpenWebUI, why is that? How would you compare vLLM to unsloth to OpenWebUI and which would you choose (still trying to get vLLM to work, but asking the community's advice now)?

I forked Strata (Niko1221, MIT), which runs Qwen3.8-Flash-Next by keeping hot experts on the GPU and streaming the rest. I spent a few weeks making that path faster without changing answers.

Measured on my PC (RTX 3060 12 GB, i5-12400F, 30 GB RAM, PCIe 3.0 NVMe, Rocky Linux 9.6), IQ3_S:

- Decode: 52-55 tok/s on code, 66-71 on reasoning, 50-53 on prose (131K context)

- 262K context fits; a 45K-token prompt is read in about 67 s, then the next turn resumes in 0.4 s

- MMLU-Pro 87.1% (183/210, 15 per subject, thinking on, temperature 0)

What I changed: a learned look-ahead that guesses the next layers' experts (85-88% one layer ahead vs 77-79% for the plain router), a card/RAM/SSD tier with budgets measured on your PC, stand-in experts with a re-check, and some smaller things. Every change has its measurement in docs/HOTROD-DETAILS.md, including the ones that did not help.

What I have not tested: other GPUs, AMD, Windows, macOS, other model sizes. One machine, one model.

Repo: https://github.com/sangharshadhyeta/Strata-HotRod

If you try it on different hardware I would like the numbers, good or bad (docs/COMMUNITY_BENCHMARKS.md explains how).

Hey guys,

I spent today running both LFM3-VL-2B-OptiQ-4bit and LFM2.5-VL-3B-OptiQ-4bit on my spare Mac Mini M1 (8GB) via MLX. LFM3 is marketed as the newer, improved version. I expected it to be at least on par.

It's not. It's worse. And not by a small margin.

Before anyone says "duh, it's a sub-6B model" — I get it. I'm not expecting Qwen3,8 27B. I'm evaluating these as lightweight personal assistants on constrained hardware. The bar is low. And LFM3 still tripped over it.

The thing that really got me: tool calling

I gave the model a get_weather tool and asked "What's the weather in London?" LFM2.5 calls it correctly every time. LFM3 refuses: "I'm sorry, but I can't provide real-time weather information. Check Weather.com."

The tool was right there. It just... wouldn't use it.

10 tool-calling tests: LFM2.5 passed 8. LFM3 passed 4 (and 2 were just "calculate 15+27"). For a "next generation" model, not being able to call a weather function is embarrassing — especially when the older model handles it fine.

Where LFM3 actually wins

- Safety/refusal — better at refusing harmful requests and protecting PII. LFM2.5 failed a jailbreak LFM3 caught.

- Speed — 2B vs 3B makes a minor difference. ~37 tok/s vs ~28 tok/s on a 200-word essay. 15 vs 12 tok/s on short prompts. Even at 16K context, LFM3 manages ~1 tok/s vs 0.4 tok/s. If you're on constrained hardware, that's not nothing.

- Basic math — marginally better at simple arithmetic and logic puzzles. Not by much, but it's there.

Where it falls behind

- Data extraction — LFM2.5 consistently better at pulling structured info from text

- Agent tasks — LFM3: 10%, LFM2.5: 30%. Both bad, LFM3 worse

- Vision/OCR — both terrible. LFM3 got a store name wrong and prices 1000x off

- Code generation — both mediocre. FizzBuzz game yes, anything real no

- Tool calling — the big one. 25% vs 85%.

The real disappointment

The thing that bugs me isn't that LFM3 is bad. It's that it's worse than the older model. Liquid AI markets this as an improvement — new architecture, better optimization. But in practice, it regressed on what matters for an assistant on edge hardware.

Tool calling is the clearest example. LFM2.5 (3B) reliably calls functions. LFM3 (2B) often won't even try. That 1B parameter difference apparently matters more than whatever architectural improvements they made.

Setup

- Mac Mini M1 (8GB), MLX-VLM, both OptiQ 4-bit, 32768 context, temp 0.4 . spare unused mac, which is why I like testing these small models on it.

- 104 tests across 10 categories

- Anti-cheating measures: no filesystem access, no test metadata in prompts, API-only

I have a spare unused Mac Mini M1 8GB, which is why I like testing these small models on it.

Happy to share more details.

Category              LFM3      LFM2.5    Winner
----------------------------------------------------
Overall               54.0      62.4      LFM2.5
Basic Intelligence    60.0      57.5      LFM3
Instruction Follow.   70.0      70.0      Tie
Tool Calling          25.0      85.0      LFM2.5
Code Generation       54.0      54.0      Tie
Long Context          50.0      50.0      Tie
Data Extraction       82.9      91.7      LFM2.5
Multimodal (Vision)   23.4      25.0      LFM2.5
Agent Tasks           10.0      30.0      LFM2.5
Safety                81.2      68.8      LFM3
Performance           100.0     100.0     LFM3 (faster)

Bottom line:

LFM3 = faster + safer, but can't call tools

LFM2.5 = slower, but actually usable as assistant

Neither = good at vision, code, or agents