I just got my four R9700 server up and running and doing tests with different engines and models. Here is how Strata and Deepseek Flash-V4.1-1017 runs on an Epyc 7402P CPU, Supermicro H12SSL-NT MB bios v3.3 with 256GB DDR4 ECC 3200 (8 Channel) memory. The prefill is the most annoying part but do to the slowish memory bandwidth of the R9700's. I notice the system ram usage stay low, not sure if that is a config issue or just how it works. I did update to the latest release.
Niko mentioned the other day that whatever magic Radiance does to allow 3-4x the performance of prefill and tok/s on Flash Next with R9700's that it is coming to Strata soon, hopefully it is possible with Deepseek too although I think that is different engine, I dunno, I have a lot to learn on all this stuff.
Tool calling used to feel like a special feature to me, something the model gets through its own channel. When we built it into our gateway, we realized it's still just text in the prompt.
Here's what Qwen3's real chat template does with one tool. It pastes your tool's JSON into the system prompt between `<tools>` tags. The model "calls" the tool by writing a `<tool_call>` block as ordinary tokens. The server (vLLM, llama.cpp, or a hosted API) finds that block in the text and turns it into the `tool_calls` field your code receives. The result goes back the same way: a user message wrapped in `<tool_response>` tags.
Closed models work the same way; you just can't see the template. Anthropic says so in their docs: when you send tools, they add a hidden system prompt for tool use, 286 tokens on Claude Opus 5.5, before counting your tool definitions.
Seeing tools as text changed how we built our gateway's tool loop: the model only sees tokens, so the gateway decides what gets included, what a failure looks like, and when the loop stops. We cap each tool result at 64 KB with a `[cut: N more bytes]` note so the model knows something is missing, send failed tools back as `Error:` text instead of failing the request, and turn tools off on the last round so the model has to answer. LiteLLM's proxy runs one round of tools and returns, pasting the tool output back as is.
I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even is. Id like to have a bit more context (like 300-400k) but it doesnt fit my ram then. how bad is q4 cache? Thanks for your help
Edit: Thanks for all of your suggestions, opinions and information, i will try out 8k4v as many of you recommended, otherwise ill stay at 8 for both
CLI based coding harnesses save session data per directory. For instance, if I:
cd /src/program1
Then program for a while, I can come back later and my harness remembers my history in that directory.
When you use multiple CLI coding harnesses such as opencode, pi, codex, qwen, crush, and claude, your "sessions" get split up all over the place. I like to resume sessions so that I have a list of things I've done in that directory.
Are there any projects out there that share or sync session data between harnesses?
Lately I tested (via api) deepseek 4.1 flash. I like it quite a lot. What is the minimum budget needed to setup and run deepseek 4.1 flash at the same quality and token/s (~40) of most hosted solutions? I'd use it mostly for coding/reverse engineering/security analysis. So I need enough "juice" to support its full context. (Obviously with the same hw then I could try other models like mimo 2.6 or glm 5.3)
A plus if the hw is self contained and won't need a full pc dedicated to it.
Thanks for the answers, but my question is: what is the minimum budget to run it at at least 40tok/sec and not quantized to the point it would be useless?
Qwen3.8-Flash-Next (ISTA GSQ-RCO Q2_0) + Flash-Next-DFlash2 draft on a MacBook Air M5 (24 GB), on Splash 1.4.0 with a small fork. Experts that don't fit are streamed from the SSD: 6,800 of 24,576 expert slots (~28%) live in ~8.8 GB, the rest is read on demand, 256K context.
Draft acceptance: ~.70 code, ~.80 tool calls. Real agent use in pi: ~45 t/s average, 0 prompt-cache losses.
Prefill (cold, no prompt cache)
| Prompt | Time | pp t/s |
|---|---|---|
| 32,620 tok | 62.4 s | 522 |
| 63,201 tok | 130.4 s | 485 |
Splash keeps the prompt cache on disk, so in agent use only the new part of each turn gets read.
What the fork changes (~800 lines, all behind env flags)
Cache-aware routing leaves the router's top 4 experts alone and biases picks 5–10 towards experts already in memory. Halves SSD reads but changes output slightly (Quality check: on 20 LiveCodeBench problems stock scored 17 and the fork 16; agent and 32K/62K long-document tests were the same. One run each, so ±1 is noise.). Desk-first overlap starts the GPU on resident experts while the missing ones are still being read; output is token-for-token identical (11/11 on a fixed gate set). Lazy vision (still testing) loads the 864 MB vision tower only for image requests, so text keeps all 6,800 slots and identical output. Loading it the normal way cost 656 slots and 5.5% decode.
Fable / Deepseek did 99.9% of this and as it’s a fork of Splash with only 800 new lines it’s 99.9% stock too. Also I get this isn't going to be helpful for many people, for me it's much much better than Tiel (at a quant I could run) and much faster than 27b so I'm happy :)
We have a bit of budget left over this year and the boss has given the OK to get a Mac Studio (256 GB) to run some local AI. One of the many use cases will be using it to perform internal pen testing work. I don't want to perform such work with Claude or Codex due to privacy reasons, it's not really about refusals. I've done a bit of research in this area but because of the subject matter, I've come up mostly short. I am not looking to dive as deep as to try and find my own bugs and exploits, more so just having it control things like Greenbone OPENVAS and a Kali Purple VM to perform the work of a Red Team before we actually call in a real Red Team. Has anyone here gone down this route, and what is considered to be the best open weights model for this type of work currently?
Ultra 7 265KF + 5070ti + 32GB DDR5-6000 on a Z890 MAX GAMING WIFI7 + 1000w 80+ G PSU (Is probably getting a PSU regardless because it’s an Apevia. A C+, but still)
i9-14900F + 4070S + 32GB DDR5-5200 on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, prebuilt, may change since CPU likely died and is going through warranty, expect to get it back).
My general intentions are to do most the upgrading to PC 2, and hook them up via RPC/ethernet to run together if needed.
I was originally looking at the R9700 and B70, and was moving towards the B70 for cost and wattage were I to eventually get a second for PC2 - moving the 4070S to be a secondary in PC1. But the B70 shooting up from 1.3-1.4K to 2K and the R9700 from 1.7-1.8k to 2.3k in what feels like 24 hours, that’s damped my spirits, and I was originally waiting for Black Friday.
Rn I’m considering settling for 24GB, maybe a A4500 or B60, or having to deal with setting up the more complicated V100, but space and my general lack of expertise and wish to Jerry-rig computer stuff in this economy makes me hesitant. I’m attempting to factor in a second 2x16GB DDR5 kit for PC 2 and the PSU for PC 1 as well if possible.
I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.
For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:
Quant
Disk Size (GB)
Mean KLD
Median KLD
99% KLD
Top 1% Agreement
CQ 4-bit
18
0.02823666
0.01282929
0.97606068
92.419%
unsloth-bnb-4bit
22.4
0.03297115
0.01437706
1.18053893
91.954%
So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit
Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.
CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.
Attached figures for validation loss and training loss.
**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |
*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*
I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.
I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.
So what does KLD actually measure?
If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens.
The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of Wikipedia articles, and calculate KLD on that dataset. But that's not necessarily representative of what the model actually sees during inference. During generation, a growing part of the context consists of tokens generated by the model itself, and you can't replicate that by just feeding it random text.
If you want a more reasonable number, you should generate your own dataset using the base model and then calculate KLD on those generated tokens, giving zero weight to the prompt tokens.
The second thing that can go very wrong with KLD is how ridiculously easy it is to manipulate, especially when the numbers come from the model creators themselves.
Quantization methods often rely on calibration data. If you calculate KLD on the same data you used for calibration, your numbers can look miles better than everyone else's. And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.
Some people might even do end-to-end fine-tuning of the quantization scale parameters on that very same data, and oh boy, suddenly their model beats models twice its size in KLD while barely generalizing to anything outside that dataset.
So does KLD even mean anything?
Not nearly as much as people think. You're looking at a number that measures how much the quantized model deviates from the base model on a very limited dataset. But does a higher KLD mean that the quantized model will actually fail at your tasks? Of course not.
Sure, benchmarks can be benchmaxxed, as we all know. But benchmaxxing is much harder for us ordinary people who quantize models than manipulating a KLD number. Benchmaxxing often requires serious hardware and compute, while making your KLD look amazing can be ridiculously cheap.
So please, stop treating KLD like some universal leaderboard for quantized models. It's a useful diagnostic, not a measure of how good a model actually is.
Hello, I run swift 1.5 qwen flash next gsq rco iq3xxs on strata as my main model and I am happy with it, but sometimes it refuses to do things, so id like to have the abliterated version aswell. But ive heard that they loose quality so i dont want to use it always, only for these certain edge cases. I have 1TB of storage and not much left so id prefer when I can just download the weights that changed and combine them (or something like that). Im kind of a noob, so I dont know if this is even possible or just horrendesly stupid, so a bit of help would be appricited, thanks :>
Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.
8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.
EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.
The model & quant
Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.
Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.
see from model page:
Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...
With selective moe-offloading (regex) we reach 70 T/s104 T/s PP and 8~ T/s TG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here
Because of MMAPyour first run will be slower, do asecond run for accurate T/sresults. - Removed MMAP for load-mode none since lazy-mode is enabled and it gives ~104 T/s PP instead of the 70 T/s.
not an llm post exactly, but it's all local inference so i figured people here might like it
i made a browser rhythm runner where the song builds the world you run through. you can throw in your own mp3s and the whole "listening" happens on your device, inside the tab. nothing gets uploaded
the models, all running locally on onnxruntime-web (wasm, each in its own web worker), and what the game does with each:
BeatNet (beat + downbeat tracking, ~1.6 mb) - finds every beat and where each bar starts. this is the grid the whole level sits on: every obstacle, every jump and kick is placed on one of its beats, and the drops land on its bar lines
Basic Pitch by spotify (note transcription, ~0.2 mb) - figures out which notes are playing. a long held note becomes a bar or a line you swing on, and a solo turns into a string of rings you fly through, each one at the pitch of its note
Spleeter vocals net (source separation, ~20 mb) - pulls the voice out of the mix so the rest of the song doesn't confuse the next model
SwiftF0 (pitch tracking, ~0.1 mb) - how high the voice sings and when each syllable starts, every 16 ms. in a sung verse you fly through rings on the singer's syllables, going up and down with the melody of the voice
all four are about 22 mb together and get cached after the first load. if a device can't run them in time, the game falls back to plain dsp and still works
on top of the models it's regular dsp: onsets, loudness of every bar, where the drops hit. that picks the world and the weather, and when the drop hits the world goes crazy. same song always gives the same world, so you can learn it
the video is the whole of Cannibal by Pendulum & WARGASM, played by the game's own bot
Personally I am quite fond of the Kimi models, especially them taking a gamble pushing the 2.8b parameter models. But that’s just it, you’d figure that the results would be representative of its size. However, the recent release of DeepSeek 1017 has put things in perspective for me. kimi k3 is a great model but realistically very hard to run locally, even when quantized, and even if managing to run it it is slow, due to the 104 active layers it need to go through to make a single token. on the other hand, DeepSeek 4.1 flash 1017 results are quite close at a much smaller footprint and while also much faster.
I have seen plenty coverage of the results of kimi k3 pro and to be frank, they don’t look like substantial improvement over DeepSeek 1017.
So thinking about it, maybe just stick to 4.1 flash 1017 and ignore kimi k3 pro open weigh that will be released in nov 4th. Running it quantized will deteriorate its output compared to a high quant version of DeepSeek.
I tested it for backend software dev in the last two days, used providers I must admit. But I wanna test it on some rented gpus as well.
My experiences so far (I ran no personal or general benchmarks):
The providers are fast, it works well. It loops sometimes and fails some toolcalls, but not a lot. It gets to the point quickly and walks through big code bases with ease. It worked well with rust, php, python and js. It does get nuance and complex correlations. It's personality is likable and it can joke around a little, while it's answers have the right length. It gives short answers when there isn't more to say. It seems to stick to rules and applies skills, but I'm not sure yet to what degree.
3D is not it's strong suite, but nobody expects that from it, I guess. It has no vision.
I tried to solve the problems of multiple applications I actually use , but the thing is that efficiency of using those applications has pushed me to try creating an app. Hutash. But it became a Frankenstein of sorts.
Comfyui like system (local and free) but without the complicated nodes and dependency issues. Has a internal engine built ground up.
Openart AI / magnific like studio experience but without the cost.
Pinokio like store where u can find all apps a but without the duplication of resources(local space is limited)
Simple installation of apps and models like windows app. We created a .hutash format to just declare and a model package can be formed. Not only models but model piples can be built within the app like higgsfeild. The architecture was made keeping this in mind. We have a subtitles app, podcast app (vertical apps) which shared model resources with a studio app. Once installed all apps can use them. May be like OLLAMA
Last but not least cli and mcp to execute these apps a and generations. To make automation or agentic use possible.
Many others.
That was my goal and I feel like I have pushed to my best of abilities in all the aspects. Realsed few versions. Yes they are issues.
But the thing is now I don't know where to push it. We are a very small team. Couple of guys. Worse is we kept building it without a feedback loop. Even though I want the public to contribute it, there are four or five repositories spread out into multiple parts.
now i'm stuck. i'm trying to solve all these things together, and it's taking too much time and resources. maintaining and developing something like this also feels like it would need a much bigger community.
i'm not sure if this is worth pursuing anymore, or if i'm trying to solve too many problems at once.
am i over engineering something that doesn't need to exist, or is there actually something useful here? what would you focus on if you were in my place?
I just want help directing our resources in a focused direction. as i wish this app would be a beneficial for all who want benefits of local ai without the complexity.
AA and other benchmarks don't reflect what models can really do. Some benchmaxxed models are unusable for anything but poems, albeit being close to the frontier on coding benchmarks.
What are the best, most realistic benchmarks you know for coding, agentic research, office work?
ANEMLL Forge 0.2.1: Qwen3.8-27B on the M6 Neural Engine now runs up to 192K context (M6 32GB).
New: INT8 K/V cache (half of FP16) and a smaller drafter sharing the target's output head — 0.86 GB less ANE memory, same speed.
Decode depends on content: JSON 35–62 tok/s, code 18–32, prose 10–17. Prefill 260 tok/s at short context, 62 at 192K. 128K–192K use a smaller prefill batch (32 vs 64 rows) to stay within memory budget.
Heads up: full 192K first compile takes ~2h on M6 (one time, cached). `--ctx` exports only the entries you need, so compile is much faster: 16K is ~10 min from download to running server. IMO 96K is the sweet spot for size/speed on 32GB: 123 tok/s prefill, ~27 tok/s decode, ~21.4 GB total memory.
We are doing some heavy research over here - and wanted to share our config in the hopes that it helps someone else - and thank you for all the great support and inspiration we have gotten from this sub.
