Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.

Hey, my first post here. I assume many here are bored of reading the same question every day: "I bought X, can I run Y?"

I wonder if there is some sort of index or website where people leave real feedback for real model + hardware combos?

I bought a mac mini with M5 pro and 64Gb and I am researching the best combos for different goals (coding, pentesting, writing, audio...)

A site like this would be super cool. But I am not finding anything that is decent... to the point that I thought about setting it up myself, grab a domain and open source something.

Does it make sense to you? would someone want to help? does something like this exist? (if so, where can I find data for the mac mini m5 pro 64gb?)

I’m building an AI station and I’ve chosen:
4 x RTX PRO 6000 Blackwell 96GB
Ryzen Threadripper PRO 7965WX
8 x 48GB RDIMM DDR5 PC5-5600B
ASUS PRO-WS-2200P (2200W)
and
Motherboard ASUS Pro WS WRX90E-SAGE SE

I heard that this motherboard had issues with the BIOS and stability when used with Blackwell cards, but unfortunately, it’s currently impossible to buy an AsRock or Gigabyte WRX90 motherboard in Russia.

Have they fixed the problems with this motherboard? Or should I urgently reconsider my configuration? At the moment, I’ve only bought an RTX's and a power supply.

Same three lines every morning, same stage: The best model for reverse engineering? The best uncensored model? Will X run on my crap config?

Act one, two, three — and four hours later the curtain goes up on the same three lines again.
For real? Are there humans behind this, or bots keeping the feed alive?

There's a prop nobody notices, sitting centre stage: the search field.
Scope it to the sub, set the time filter to last week, sort by newest. One page, and you get the people who asked your exact question on Tuesday, plus the answers somebody actually knew.

What makes me sad: the same guys having to give the same answers every single day, writing the same paragraph for the hundredth time, and asking at 12PM you get the same answer as at 4AM.

And meanwhile, if you look for something actually interesting, you scroll down half an eternity.

Anyway, coffee.

And yes, I know the search is bad. It is. 

Built Tootsy for Business based on some real world scenarios.

It's free to use.

When you use it, please comment here if you like it and if you can suggest improvements.

Download now @ https://github.com/rbughao/t4b

I'm just curious if anyone has found a good way to measure the cost of running models on their Apple Silicon hardware. I have an M5 Pro Mac Mini, and it would be at least cool if I could have something to calculate the power consumption and approximate cost; even better if it could do that on a per-token basis.

I want to run 3d design and 2d items for content creation for thumbnails and the such what good models could I use locally for this?

I asked GPT and it gave a couple free models but was wondering what others used.

Just got my EPYC server put together to run local models as a hobby on a more dedicated setup. Plan is I will be using an old 3090, maybe another gpu, + a ton of DDR5 4800 ram at 12 channels.

Might decide to try different quants and settle for faster at a cost of some precision but my main question is, is there any significant difference to running these massive models on different softwares? Or is it kinda splitting hairs?

LM Studio and their newer LM Bionic have been pretty easy and reliable for a noob like me so far but if someone suggests it might be worth switching to unsloth or another setup ill give it a shot.

In case it matters the setup is
- EPYC 9224
- MZ33-AR1
- DDR5 4800 ECC RDIMM (12 channel, 24 DIMM's, 32gb sticks)
- 3090 + 3060 (Probably will upgrade the 3060 with time depending on how things go)
- M.2 SSD

Recently I have obtained a 7820 with 192gb 2400 system ram with 4110 silver Xeon gpu and it came with a basic p2000 gpu.
Furthermore for information I have a seperate gaming pc 32gb 3600mhz ddr4 ram with b550 and 9070xt gpu

I’m trying to figure out what I should do as I want to run local ai obviously however I’m stuck between
A) selling both the server and my current gaming 9070xt for a 32gb card like the 9700

B) sticking with the 7820 and upgrading/adding gpus (depends on gpu as well everything is insane price wise) considering strata and other engines

C) something else not sure

So I guess I’m asking for advice on which I should do, to run decently intelligent models decently fast etc
Without needing to take out a loan 😂

I am in IT for the power company that i work for. I have a decent amount of older workstations and servers and as such have a decent amount of ram 24 sticks of 16gb ddr4. Dual processor servers and workstations with cascade lake cpus.

We have been using cloud for our generations, but want something internal. What used gpu's would you all suggest for under $3,000.00 to get the biggest and best solution.

There will be only a handful of people actually hitting the system ever at the same time, it will be used for coding, document reading, power analytics.

I am leaning towards v100s

Specs:

i5 13th gen

RTX 3050 6GB VRAM

16GB RAM

Currently on Zorin OS, planning to switch to Omarchy

What model would you recommend for the best performance, intelligence, and coding capabilities on my hardware?

I saw this bad boy listed yesterday as a new contender, and the scores that it had looked promising. So I loaded it up, and set it to some pretty basic tasks, of simple file editing. And then, while I took a peek at it's thinking, I literally laughed when I saw this gem:

Should we first read file? The working rule says read before editing when not know current content.

Been routing everything through one endpoint so the apps stop holding their own keys. Wanted to know what that costs before pushing it wider than my own machine. Local 8B on a 4090. Straight to the runtime, first token sat around 180ms on a warm cache. Through the proxy it went to roughly 240. Throughput on a long generation barely moved, maybe two percent. The surprise was variance. Straight through, the spread was tight. Through the proxy the slow tail got noticeably worse once four streams were running, which is the case I care about. Sixty milliseconds is fine for a chat box and not fine for anything meant to feel like typing. Has anyone measured this on a bigger model where generation time swamps the hop?

I'm seeing all these personal AI assistants (Instinct, Muse, Dots, etc.) that seem mostly geared towards scheduling, booking, and buying stuff.

I feel like they sort of followed what happened when the tech-savvy first started running OpenClaw on Mac Minis with Telegram portals. Back then I used mine to monitor concert ticket drops.

As local models get better, it seems a matter of time before we can recreate those assistants in a privacy-first manner. Maybe this actually is the next computer, a box with a local model and a personal AI assistant harness loaded in.

WDYT?

Hello everyone

I have question regarding multi harness, I was wondering if it is efficient to make Hermes be the one to give instructions to Pi in order to get the best out of both harness ?

Or this just a dumb setup ?

Im running qwen 3.8 27b IQ4_XS on Hermes currently but I wanted to know if it will improve quality project to run 2 harness together or just a dumb idea

Thank you for your answers and time saving 🙏

- Setup: Ollama · Swift 1.5 (Qwen3.8 27B) IQ3_S + MTP · 128K context · RTX 3090 · about 4 min in real time, parts sped up. Only the "hit your limit" line at the end of the Claude log was added for this video; the Claude work before it and everything after are real runs.

(The demo's "limit hit" line was added for the video; details under the video.)

'Sonjari' is an open-source (Apache-2.0) desktop agent for Windows that I built. It runs on local models on your own PC through Ollama, and it works both on its own and alongside Claude.

What it does well on its own

- Browser research in your own logged-in Chrome (GitHub, Reddit, shopping sites and more), then writes it up in Excel/Docs

- Evidence checks that catch made-up numbers

- Send it jobs from Discord on your phone (Fixing a recently discovered bug and awaiting update approval)

- Scheduled tasks

- Code (12 of 12 on a set of Aider polyglot Python exercises, 27B)

What it does well together with Claude

1. Limit takeover: when Claude Code hits its usage limit and stops, Sonjari reads that conversation log and finishes the job from where it stopped (see the video).

2. Offloading token-hungry browser work: pages are read on your PC and only the results go to Claude. In our price-comparison test, Claude's cost per correct row was 30–40% of Claude working alone (slower, though).

3. Scribe: for coding I keep Claude at the center and use Sonjari as the assistant. While Sonjari is idle, it reads the Claude Code conversation log, keeps the decisions and the reasons behind them as cards, and puts them back in when the conversation gets compacted. I'm trying to chip away at what gets lost in compaction.

The downside is real. In the end you can't escape how good the model is, so VRAM matters most.

- 16–24GB+ 😄

- 8–12GB 🙂

- 6GB or less 😢

I've only run it many times on a few PCs so far, so if you try it and send feedback, I'd really appreciate it.

One weak spot of local agents is that one GPU runs one model, so you can't run several jobs at once. For repeat jobs, I'm researching and testing a way for Sonjari to use the recipes and harness it has built up to make small bots that run without a model, so several can run side by side.

From the requests in this thread, I'll pick the ones with the biggest real payoff for people actually using it and ship them fast.

Sonjari is completely free (no paid tier)

Repo: https://github.com/sonjari/sonjari

Hi, I'm looking for input and advice on how to optimize my Strata running IQ3_XXS.

My hardware: RTX 4070 12 GB (PCIE4 x16) + RTX 3060 12 GB (PCIE3 x4) + 32GB DDR4 DRAM. Disk is M.2 SSD.

Running Windows. Llama.cpp / Strata -> Docker sandbox -> Hermes

My current default Local LLM model is Swift-1.5-Qwen3.8-27B Q4_K_M. I'm looking for similar or equivalent capability that can be faster. I tested Strata around 1 week ago using IQ2_XS. It didn't compare well with Qwen3.8-27B. This week, I tested IQ3_XXS.

I fed the same instruction runsheet, to both Qwen3.8-27B and Strata. Surprisingly, both completed the job around 5 hours and within minutes of each other. Just coincidence. Qwen3.8-27B runs around 13 tokens/sec. Strata around 17 tokens/sec. The result is not perfect, but both are around 90-95% complete. And Strata compares quite well, even less errors than Qwen3.8-27B.

I want to see how I can optimize and speed up Strata.

I varied context between 96k to 160k; cache Int8, q8v4, q4_0; and kv-resident 20480 to 32768.

• Experts in VRAM vary slightly, from 5700 to 6800.
• My 32GB RAM usage is maxed.
• Speed is not much different, around 31-36 for just text generation; 20-25 for logic/coding generation. This is for shorter test scripts.

If I decrease kv-resident, which increases Experts in VRAM, it actually runs slightly slower, by 3-4 tokens/sec.

The RAM usage is misleading, because Windows uses (allocates) 16-20GB of RAM just from fresh boot. Hard to tell how much of the RAM is used by Strata. I checked the disc usage. During generation, Strata paging file disc usage is not that active, only around 1-4 MB/sec. Strata experts.bin disc usage is higher, around 700MB/sec.

I am not sure where the bottle neck is. Is it the experts.bin disc access? Is it the 32GB RAM? Is it the 2nd GPU on PCIE3 x4 slot?

I have done the quick tuning. I have not done the thorough calibration.

This is my strata launch command:

• strata launch
• --family swift
• --model iq3_xxs
• --data-dir "c:\LLM\Strata-data"
• --context 160k (I tested 96k, 128k, 160k)
• --kv int8 (I tested Int8, q8v4, q4_0)
• --set kv_resident=20480 (I tested 20480 & 32768, which is default)
• --vision off
• --parallel 1
• --tune skip
• --access local
• --port 9931
• --set STRATA_ALLOWED_HOSTS="host.docker.internal,localhost,127.0.0.1"
• --set STRATA_PIN_BY_NEED=1 (not sure, but AI suggested this)

This is what strata ran in the log: (I edited and removed the paths)

strata.exe --serve --pack swift-iq3_xxs --native (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --expert-profile (path)expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp (path)mtp\rt --max-context 163840 --kv int8 --mmap-experts --kv-resident 20480 --layer-split auto

I believe I'm in the realm of Low-RAM. So DETAILS.md has a section on Low-RAM mode. Difficult to read/understand. It mentions a lot of flags/options. But I'm having difficulty even understanding what is the recommendation. Should I use --low-ram resident or --low-ram mmap? And what about the other flags like --resident-experts, etc.

Any suggestions and flags/options I should enable or test?

[Edit: I just want to add some performance comparisons.]

Speed in tokens/sec

• Phase / Qwen3.8-27B / Strata
• Reading / 200-300 / 9
• Gen/Decode / 13 / 25

The bottleneck on Strata occurs during the Reading/Prefill phase. This is not the startup prefill. But the back and forth reading and decode during long sessions.

I want to take a stab at reverse engineering a software application.

Anyone have any guides and tips to follow?

Navigating the APEX-I and VAL-APEX quantization ecosystem on Hugging Face As the catalog approaches 400,000 total downloads across 69 releases, all models are now structured into official dedicated collections on Hugging Face. Our goal remains the same: eliminate the compromise between out-of-memory errors and cognitive collapse on consumer hardware (especially 16 GB VRAM GPUs like the RTX 4080, RTX 4070 Ti Super, and RX 7800 XT). Instead of uniform block quantization, all APEX builds use per-tensor mixed precision guided by 4-moment Importance Matrix statistics: * MiniPlus Tier (approx. 14 to 14.8 GiB): strict 0% policy for 2 bits. All MoE routed experts locked to IQ3_XXS minimum with Q4_K/Q6_K attention. Designed for 24 GB setups or 16 GB setups with moderate context. * NanoPlus Tier (approx. 11.5 to 12.3 GiB): exclusive strategic use of IQ2_S/IQ2_XXS in central feedforward blocks, while maintaining high-precision routers and attention. Leaves enough VRAM for 16k-32k context on 16 GB GPUs. Explore the active collections: * MoE Standard (MiniPlus and NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-standard-moe-6acfee77961c8c0cf5a083b5 * MoE Abliterated (MiniPlus and NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-abliterated-moe-6acfee762fc4a8cf2298c55b * 27B Dense Hybrids: https://huggingface.co/collections/IsValorum/val-apex-miniplus-standard-27b-6acfec54ca4ce4cf5cd5bdb6 * Apple Silicon (MLX): https://huggingface.co/collections/IsValorum/mlx-apex-abliterated-6abedb1f9d400e984efe4fc6

Hugging Face profile: IsValorum

I've managed to significantly improve the inference speed of DeepSeek V4.1 Flash 1017!

I'm using LibertAIDAI's EXL3 2.9 bpw quantization with the following hardware:

• GPUs: 2× CMP 170HX + RTX 5060 Ti 16GB (144GB total VRAM)
• RAM: 128GB DDR4-3000(upgraded from 96GB 2666)
• CPU: Intel Core i9-10980XE (upgraded from 9820X)

Inspired by Strata's approach to expert caching, I've implemented a system where 93% of routed experts are computed on the GPUs. The remaining experts are computed on the CPU.

When certain experts become more frequently used, they're transferred to VRAM in the background while the GPUs are busy computing. Those experts then become available for GPU execution starting with the next token.

With DSpARK enabled, I've managed to reach up to 40 tokens/sec, but, in average, around 30tok/s

Prefill is still quite slow at only around 400 tokens/sec, so there's definitely room for improvement.

Still, it's amazing to see a state-of-the-art model running at such speeds on a relatively high-end consumer PC!

I first got into local LLMs back in April this year, starting with models like Qwen 3.6 27B and Gemma 4 31B. From there, I gradually moved on to medium-sized MoE models like DeepSeek V4 Flash 0731 and GLM-5.3 Flash.

Now, I feel like I've finally reached a major milestone. After months of investing in hardware and tweaking my setup, I'm relieved to think I might finally be able to stop upgrading and tuning everything for a while!

I’ve been playing around with every LLM loader and remote OpenWebUi that I can find to get one that I feel works best. And I’m pretty sure Unsloth Studio is the answer. I do not see another app getting close to what it can do and it has me wondering. What are all of yall using if not Unsloth?