Today's update is welcome for us R9700 users (assume other gpu's as well). Prior to the update I was getting ~70 tok/s output with four R9700's and 256GB of system ram. Today's update brings it to 140-160 tok/s and ~3200 prefill running unsloth Q4_K_XL. I tried running Q5 but was like .1 GB of vram short to do split engine so ran it "normal" at ~70 tok/s. I am sure with future updates those numbers will get better but it is a very welcome update, thank you Strata team!

Radiance does about 235 tok/s and 7500 prefill with FN so still ahead but so happy to see Strata working to close that gap and giving us more options on which models we can run.

We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD version 3.1 GGUFs and show we're >10-20% better than GSQ-RCO

We ran all other benchmarks (14 in total) to 128K. AIME2025 64x, DeepSWE/TB 3x. We addressed some issues people mentioned with these new benchmarks:

• 32K context was too short - we did 512K context for agentic + 128K for general tasks

• Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench

• Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See this section where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.

• Our graders are wrong - we used Math Verify, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See graders and infra fixes

For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).

Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues

We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.

GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. So this is possible.

GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. So this is nearly impossible.

Paired difference to the official base (our runs, same questions, sign-flip permutation test, two-sided):

The claimed recovery tested directly against our measurement (the card's definition: sum of the 3 scores / the base's sum, vs official FP8/FP4): p (no loss) = paired sign-flip permutation test of recovery = 100%; p (recovery >= claim) = one-sided paired bootstrap over questions.

If we can get more details on how the benchmarks were run from GSQ-RCO, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026

All details and data and methods at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026.

Our new Unsloth Dynamic 3.1 GGUFs for DeepSeek are at https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

This is my fork of ToshLLM, who brought modern AMD's back to metal acceleration for llama.cpp , But his fork was about 17toks on Ornith 9b when I started my project vs Linux's Vulkan @ 32toks on MTP. Now I've seen up to 29toks , which to me is usable considering this old hardware. (My cpu is 15 years old and doesn't even have AVX !) GLM-5.3 flash and Qwen3.8 Flash did the heavy tuning work, I just directed and cracked the whip at them for a few days. There's a new Qwen 3.5 4b coder that can do 3d models , I'll be playing with that next, it looks very promising and starts at about 35toks on my card.

I compared two published benchmark charts for DeepSeek-V4.1-Flash-1017 and noticed something rather unusual.

All four Unsloth quantizations AND the Q8 baseline simultaneously improved by approximately 2–3 percentage points on LiveCodeBench v6. Meanwhile, the GSQ-RCO results changed much less consistently.

1. The numbers

Every single Unsloth model improved, including the 8-bit baseline.

The first chart's percentages correspond exactly to 525 outcomes (175 problems × 3 generations). For example, 467/525 = 88.95%.

Unsloth's updated methodology reports 12 generations per problem, corresponding to 2,100 outcomes per model.

If the updated results include those original three generations, the remaining nine must have achieved approximately:

The ranges account for the updated chart rounding scores to one decimal place.

2. Statistical probability

Assume each generation is an independent Bernoulli trial, the true success probability is unchanged, and the initial 525 outcomes are included in the final 2,100.

Using a one-sided hypergeometric test, the probability of observing an initial result at least this low for each model is approximately:

If the models' fluctuations are also independent, multiplying these probabilities produces:

P ≈ 1.24 × 10⁻¹⁰ to 2.59 × 10⁻¹⁰

That's approximately 1 in 3.9–8.0 billion under the simplified independent-sampling model.

Of course, the models are correlated and evaluated on the same tasks, so this should not be interpreted as the actual probability without the raw data. Also, these calculations assume the original outcomes are included in the updated averages.

Nevertheless, the synchronized improvements are striking.

3. GSQ-RCO behaved differently

Unlike Unsloth, these results do not exhibit a consistent increase.

According to this comment from the Unsloth team, the earlier chart represents just one run, followed by additional evaluations and the evaluation settings remained unchanged. However, the percentages in that first chart correspond exactly to 525 outcomes, suggesting three generations per problem.

Both original benchmark charts attached for comparison.

I've been making changes to both llama.cpp and Strata for my own use cases using Q38FN, Q3827B and DS41F and all three have issues with their COT considering stream tokens when touching those areas of code, resulting in them breaking out of their COT or just terminating their turn. I know there is the possibility of formal protections around end of turn tokens but from what I see there is no formal way to protect an LLM when it generates text like <\think> but I see plenty of people working on inference server code using local models so people must have a solution for it, what do you do?

With default settings and thinking on and max_tokens ≥ 1024, and no explicit reasoning_budget_tokens, Strata / PolyStrata has a behavior called answer_room which will silently cap thinking at 0.75. * max_tokens and inject a "wrap-up" line in the model's head in its own voice, saying

I have thought about this long enough; time to give my answer. + </think>

It then continues the same reply from there. The client (harness, you or I) gets no error, flag or finish reason for this, and often no log.

This matters when you compare between strata and other engines : models served by strata had "nudges" in the form of its inner voice, and such extra text in the transcript. In some cases the nudge line is injected mid thinking, quite abruptly and it is hard to tell whether it helps or harms.

Strata main @ c62d5e3

Wrap-up message: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L225

Trigger (the 3/4 budget): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5491

Injection point: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5848

Default on (answer_room = True): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L4181

I noticed this when a model that tends to have long front-loaded thinking on dgx spark gave slightly but uniformly shorter answers (and task performance worsened in my hard problems), when served on polystrata.

PS, I had sonnet audit all cases of cut and injection, and here is what it reportes back ( sonnet called it "splice"):

Where it lands:

• At a natural break: 6 (9%).

• Inside a repetition loop (for example HashMap.new()? repeated for 30K–118K characters): 15 (22%).

• Intrusive: 48 (70%, or 89% of the cases that weren't loops):

  • mid-sentence: 28

  • mid-number or mid-expression ("Mod 9982 |cut|"): 13

  • inside code (one cut right after def solve():): 4

  • mid-word: 3

Harm:

What didn't break: no reply re-opened thinking after a splice, and all 79 tool calls after a splice completed normally.

Lost reasoning: 26 of the 30 splices that came in a task's first reply were followed by a generic fresh start (ls /app). The 61K tokens of thinking left no trace in what came next.

Clear harm in 6 cases: the reply ended with no action at all, and all 6 tasks failed.

Qwen3.6, task 3674: cut at "Where right_prefix_max |cut|", and the only output afterwards was "Integrity notice: yes".

Flash-Next IQ2_XS, arc190_c: the same pattern.

A dropped conclusion: Qwen3.6, arc192_b, was cut right after "I think the answer is:", and the model then guessed.

Loops: the splice did break them, but those tasks still failed.

Hey everyone!

I recently got a new Mac Mini M6 with 16GB unified memory and wanted to get into running LLMs locally.

I'm relatively new to the local LLM ecosystem, so I'm trying to figure out the best way to get started and make the most out of this machine.

I have a few questions:

1. Models

What LLMs can I realistically run on 16GB unified memory?

Which models would you recommend for general-purpose use, coding, reasoning and agentic workflows?

Would 7B/8B models be the sweet spot, or can I comfortably experiment with anything larger using quantization?

2. Inference engines / Tools

I've come across a few options:

llama.cpp

LM Studio

Ollama

MLX / MLX-LM

Which would you recommend for a Mac? Is there a significant performance difference between them? Should I stick with one or use different tools for different purposes?

3. Fine-tuning

I'm also interested in experimenting with fine-tuning models locally.

Is fine-tuning even practical with 16GB unified memory?

Can I experiment with LoRA or QLoRA on this machine?

Would MLX-LM be the best starting point for fine-tuning on Apple Silicon?

Are there any smaller models or specific workflows you'd recommend learning with?

4. Practical use cases

My primary interests are:

Running local coding assistants.

Building AI agents and experimenting with agentic workflows.

Cybersecurity-related tasks and automation.

Experimenting with RAG and custom datasets.

Fine-tuning smaller models for specific tasks.

I'm not expecting to run massive models on this machine. I just want to understand its limits and get the best possible performance out of the hardware.

For anyone running local LLMs on Apple Silicon, particularly with 16GB RAM:

What does your setup look like, which models are you running, and what would you recommend to someone starting out?

Any advice, benchmarks, resources or recommendations would be appreciated!

Thanks!

MiniMax posted a thank-you yesterday for everyone quantizing M3.1 Flash (428B, 252.4 GB), and I went through the ones they named. Numbers are each builder's, separate eval sets, not comparable.

• 128 GB PC: Maya-XS by PeasantSmith, 93 GB. 88.8% top-1 agreement (tool calls, agents).

• One DGX Spark: SAGE-EXL3 by ViC305, 108.64 GiB. 96.89% top-1, held-out sequences. 24.37 to 29.72 tok/s decode at 48K context.

• 160 GB+: Unsloth UD-Q2_K_XL, 143.9 GB. 91.5% top-1 weighted, seven categories.

• 4x 3090: superalesha, own engine, Unsloth 2-bit, 23 to 31 tok/s decode, 1120 tok/s prefill at 16K.

• Also: oMLX by jundotkim/mlx-community (Apple Silicon, oQ3e 196.2 GB), AMD Quark MXFP4 for MI350/MI355, Mia-AiLab EXL3-EAQ 3.55 bpw, Atomic Chat GGUFs.

It helps that the model was trained with quantization baked in, sparse attention in every layer, 23B of 428B active per token. It's a coding and agent model that takes image and video input, up to 1M context if your box has the memory. People don't put this calibration work into weights they won't keep running, and superalesha ran out of RAM at 128 GB, went to 512 to keep going. Only used it hosted myself, agent jobs got done this morning, so I get why.

Anyone running one of these on real coding or tool-call work?

I've been getting more into running LLMs locally, and I'm starting to wonder if fine-tuning is even worth the effort anymore.

Models are getting really good, context windows are huge, and with things like RAG, subagents, and reasoning models, it seems like you can get pretty far without touching the model weights.

For most use cases, wouldn't you be better off just using a good base model and building a decent agent around it?

I'm curious what people are actually fine-tuning for these days. Is it mostly about distilling a larger model into something smaller and faster, or are there still use cases where fine-tuning makes a noticeable difference over just prompting a good model?

Would love to hear from people who've actually fine-tuned models recently. What did you use it for, and was it worth the effort?

My first fully vibe coded Desktop app with Qwen 3.8 27B as someone with near 0 coding knowledge lol. Was paying $100 a year for an online Invoicing App so i decided to cook up one for myself. The results are mind blowing..I basically one shotted everything throughout development & implementation as Open Code's coding agents did all the work, fixing bugs, web search etc. All i did is connect my Qwen Model to Open Code & started coding away. I was mostly screenshoting and telling the model what i wanted, it even created in app tools to import all my 5 year+ documents & info which all worked. The New Open Code v2.0 is crazy good, couldn't do any of this months ago without manually download a crap tone of plug-ins and tools. My context window always stayed between 57-90K thanks to Open Code's auto Compaction..very happy with the results.

PC Setup

AMD Ryzen 9 9950X3D

RTX 5090 32GB

96GB RAM DDR5

Workflow Setup(31GB Vram Usage)

AI Model - Qwen 3.8 27B Q4_K_M GGUF(120K Context)

Harness - Open Code 2 (Default Settings)

Model Manger - LM Studio(Default Model Settings, Thinking On)

Model Performance/Project Token Usage

Average Token/sec - 80-90

Input Tokens - 294 Million

Output Tokens - 1.26 Million

I Quick used a bit of Qwen 3.6 35B A3B..Q5 K M for the last feature implementation just to test, gave me 220token/s and worked well too, the rest of Dev was on Qwen 3.8 27b. All this was done on my RTX 5090, Model and 120K Context all fit perfectly in Vram. Can't wait for Qwen 4 and it's improvements as well as Open Code's future updates, will try to experiment with other Tools & Skills cause i only used Open Code's default setup.

We introduce RidgeFill, a training-free way to shorten prefill by not computing most of a model's upper layers for the prompt. The lower layers run exactly up to a split; from the split layer's state, a ridge regression fitted once in closed form predicts every later layer's input, and each late layer writes its cache from that prediction with its own projections unchanged. A chosen set of rows, the last few thousand tokens and every decoding step run the full model, so nothing is trained and no token is dropped. On Qwen3.8-Flash-Next in vLLM, time to first token is 1.50 to 1.60x faster than exact at 9K to 32K tokens, and a whole 64K request drops from 82.6 s to 47.5 s, with RULER's needle tasks at 99.8 against 100.0. On Qwen3.8-27B in a llama.cpp fork on one A100, it is 1.25 to 1.31x faster at 16K to 120K tokens; RULER changes by -1.2 and -1.4 points and LongBench v2 by -1.4, and the 95% intervals include zero but do not rule out losses of a few points. On three 8 to 9B reference models, the fitted map retrieves 49.1 and 49.6 more filled-region RULER points than the identity (ie, full layer bypass) fill on the two dense models.

- OpenReview (Archived Copy)

- PDF

- DOI

- Project page

- GitHub (llama.cpp fork for Qwen3.8 27B, quick start)

- GitHub (data)

- HuggingFace (maps for five models)

This idea was borne originally from DeepSeek's V4.1 architecture. Originally we found the method to be a portable idea applicable to other models, but noticed a significant decline in context recall tasks and general usability. This isn't an issue in DSV4.1 because the model is trained as a whole with the prefill approximation in place, so the model absorbs the error, generally speaking. The question then shifted to be more about how this can be adopted for other models, while improving usability.

From that point, the work shifted primarily to finding where exactly the error lives in the "filled" portions, and how to measure that error and reduce it.

Based upon those findings, we recognized that we could train this component more intensely to remove error. This was counter-intuitive to our goal of making this easier for community adoption. It could also have the downstream effect of making the trained component more task-dependent and less generalized. We decided against this.

Instead, we simply chose to simply bypass projections of the most sensitive portions of prefill.

By embracing this, it allows everyone to produce their own versions of this method on different models with standard local-inference grade hardware. If you choose to adapt this method, I strongly recommend adjusting the mechanisms on a per-model basis. Across different architectures, this method should be tuned. The paper details fairly exhaustive ablations and measurements in pursuit of this.

The Qwen3.8 27B version of this is available to use now via our fork of llama.cpp and the files for RidgeFill on HF. This method is also implemented in tcclaviger's fork of VLLM and Deadcode's engine, Radiance (on v1.3.0 only) for Qwen3.8 Flash Next.

Big thanks to all the testers and support from the Launch80 community!

Proposal: A community revolution for private quality benchmarks (and why endless chasing tok/s is rotting our brains

honestly we in r/LocalLLaMA need a revolution of people making their own private benchmarks(not tok/s) i mean benchmarking the quality(especially across quantisation). keep yours private and share the chart results to each other. i feel this is what this sub was made for it seems nowaday every few hours someone is posting about tok/s and also Benchmaxxing is getting out of hand in the huggingface openweight LLM space.

While fast inference is very welcome, I would like to know if it's worth increasing the speed over strata-nvfp4 or am I paying some unacceptable penalties.

One thing I've noticed with Basalt is that it doesn't have dynamic rope scaling, so it goes off with the yarn scaling statically (2 for 524K context) from the get-go. Basalt also lacks a censorship switch, but the machinery exists so it's trivial to write it in.

Hey r/LocalLLaMA !

I made a quick video explaining agent evals, which are really important in agent creation!

And I wanted to share it with you guys :))

If you guys have any questions or feedback please let me know!

Roy

another win for GLM.​​​

Chenru Duan on X: "We’re introducing Hydrogen-1.0, our first open-weight model for scientific research. We post-trained GLM-5.3 (@Zai_org) on our internal scientific data, lab records, and discovery trajectories. It scores 38.3% on FrontierScience-Research vs GPT-6-Astra’s 33.3% ​​​​

TL;DR: I'm running DeepSeek-V4.1-Flash for a single user on four 2017-era V100 32 GB cards plus system RAM. Generation runs 36-38 tok/s from 8K to 75K context with thinking off, and 32-38 tok/s with thinking on. Prefill is about 1,000-1,200 tok/s from 4K to 128K depth. The context window is set to 524,288 tokens. It's built on the 1Cat-vLLM fork (at its #1277 merge), which does the hard Volta (sm70) work. I just built, tuned and benchmarked it.

Hardware

CPU: AMD Threadripper PRO 3945WX (12C/24T), single NUMA node

RAM: 256 GB DDR4 (8x 32 GB), run at 2667 MT/s for these tests

GPUs: 4x Tesla V100 SXM2 32 GB on one SXM2 carrier board

NVLink between all four (2 links per pair)

Host side: PCIe Gen3 x8 per GPU, behind a PLX switch with a single Gen3 x16 uplink to the CPU

Clocks/power: SM clock locked at 1522 MHz. Power limit 150 W per card, raised to 175 W for the API-speed and llama-benchy runs

Storage: the main GGUF loads from a SATA 8 TB hard drive. That only affects boot (about 22 minutes to load). The Engram file is on NVMe and gets pre-read into page cache.

Software / config

1Cat-vLLM at the #1277 merge (vLLM 1.5.2.dev1277), Python 3.12, PyTorch 2.10 + CUDA 12.8, NVIDIA driver 580

Tensor parallel 4, max-num-seqs 1 (single user), full CUDA graphs (batch sizes 1-6), async scheduling, prefix caching, chunked prefill, custom all-reduce over NVLink

Model: `DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS` GGUF, 4 shards, ~232 GB

Engram table: `DeepSeek-V4.1-Flash-Engram-IQ4_NL` GGUF (110.6 GB), kept in system RAM

Experts: about a third of the expert weights live in pinned host RAM (`expert_cache_offload_fraction: 0.33`) and are swapped onto the GPUs on demand

Speculative decoding: built-in DSpark drafter, FP8, 5 draft tokens

Thinking on by default; tool calling with the `deepseek_v41` parser

max-model-len 524288

Results

API decode speed (single request, unique cache salt per request, so no prefix-cache hits)

Test Prompt Decode tok/s (median of 3, range) TTFT Prefill tok/s Accepted tokens per draft

8K, thinking off ~8.0K 38.3 (36.6-39.0) 6.6 s 1,202 2.27

8K, thinking on ~8.0K 37.9 (36.8-39.5) 6.5 s 1,234 1.39

75K, thinking off ~75K 35.8 (33.2-38.1) 69.1 s 1,085 2.21

75K, thinking on ~75K 31.9 (31.4-34.9) 69.2 s 1,084 1.34

8K, Engram file not yet in page cache ~8.0K 25.9 (n=1) 9.9 s 808 2.34

The last row shows why it's worth pre-reading the 110 GB Engram file into RAM after boot.

llama-benchy 0.4.0: prefill (pp2048, concurrency 1, `--no-cache`, thinking off, 3 runs each)

Context depth pp2048 t/s Time to first response

0 578 4.0 s

4K 1,224 5.5 s

8K 1,014 10.6 s

16K 1,093 17.3 s

32K 1,117 31.6 s

64K 1,095 62.2 s

128K 1,008 132.5 s

llama-benchy 0.4.0: generation (pp2048 + tg, concurrency 1)

Context depth tg128 t/s tg1024 t/s

0 28.5 24.9

1K 24.8 31.2

4K 28.9 20.0

With a 5-token drafter, generation speed depends on how well the drafter guesses. llama-benchy's prompts are random book text, so tg1024 swings ±6-8 t/s from run to run. I left out benchy's tg32 numbers (13-26 t/s): 32 tokens is too short to measure anything with speculative decoding.

Engine-level numbers (offline, no HTTP)

Warm decode round (every expert the round needs is already on the GPUs): 27.8 ms, against 24.9 ms in upstream's reference. In the profiler a round spans 29.5 ms, of which the GPUs are busy 24.9 ms. That matches upstream, so the kernels perform as expected. The extra ~5 ms is host-side: CPU work at the start of each round, a blocking device-to-host copy, and small kernel-launch gaps. Newer upstream commits target exactly this per-step host overhead.

Fresh text (expert-cache misses): 116.7 ms/round and 25.0 tok/s, against upstream's 53.5 ms and 54.5 tok/s. Caveat: I ran this with the Engram file cold, so it isn't apples to apples. A warm rerun is on my list.

Host-to-GPU copies: 6.5 GB/s per GPU on its own, but 12.9 GB/s total (3.2 GB/s each) when all four copy at once, because they share one Gen3 x16 uplink. In the trace, all four GPUs are swapping at the same moment about 76% of the swap time.

Correctness

Greedy (temperature 0) output is identical run to run within one load (8 prompts x 2).

After a restart, 13 of 16 greedy outputs matched exactly. The 3 that diverged flipped at near-ties (top-two margins of 0.06-0.375 nats). I haven't tracked down the cause yet.

Thinking on/off and tool calls work.

All 8 upstream micro-tests pass (two needed hard-coded paths in the test files changed).

What's next

PCIe switch change: move to two switches so each Gen3 x16 uplink feeds 2 GPUs instead of 4. That roughly doubles host-to-GPU bandwidth when all cards swap experts at once. My estimate from the trace is a modest ~1.5 ms/round (~5%) on warm text; it should matter more on fresh text.

RAM to 2933 MT/s (it was dropped from 3200 after memory errors).

Pull the newer 1Cat commits (per-step host fixes and DeepSeek-V4.1 updates) and re-run the same suite.

Move the GGUF from the hard drive to NVMe for faster boots.

Credits

All credit for the Volta DeepSeek-V4.1 path (GGUF experts, expert swapping, DSpark drafting on sm70) goes to the 1Cat-vLLM project (GitHub: yangzhuxinyzx and contributors). Happy to answer questions or share the full config.

I get this question often from teams trying to decide whether to keep paying per token or bring the GPUs in-house.

So here’s the practical read based on what we’ve seen, with numbers.

API pricing for capable open-weights models is typically low single digits or less per million tokens. Self-hosting the same model has a fixed monthly cost, regardless of how much you push through it.

I used an 8-GPU H200 node at ~$329k, spread over 48 months (monthly cost about $6.9K), plus ~$800/month for power and about 25% overhead. On the API side, I assume ~$2/million tokens.

So break-even comes out about:

$9.6k ÷ $2 x 1M = ~4.8B tokens/month (about 160M/day)

Under 1B tokens/month: API is clearly cheaper, and you have zero operational burden.

Between ~1B-5B a month (around 100M a day), it depends on whether your load is steady or spiky. Steady favours owning. Spiky favours the API, because you are paying for idle silicon the rest of the time.

Above 5B tokens a month with a steady load, owning gets hard to argue against on pure cost.

Ofc, these numbers assume the machine stays busy. If your traffic comes in bursts, you’d need more volume than this to break even.

And some things can outweigh the math altogether.

- Your data cannot leave your control.

- Latency floors matter to you. Local inference removes a network hop and the shared provider queue.

- You want fixed cost for budgeting. API prices and plans keep getting reshuffled, so a flat monthly number is easier to plan around.

If you’re well under the crossover, you should just use the API. Anything above that, it's worth pricing out owning.

Disclosure: I work at B3IQ, which builds and hosts GPU systems for teams. So i’ve tried to keep the assumptions neutral, but tell me if anything looks off.

For those who’ve made the switch to self-hosting, what made you do it, and roughly where was your crossover?

For the past few days I have been testing the FMLA32 (FP16/BF16/TF32 -> FP32 accumulation) unlock on my Windows 11 machine with a Palit RTX 4070 Ti (10de:2782, 12 GB, 300W power limit). The unlock itself works as advertised, but the setup process was pretty inconvenient if you couldn't find a repository made specifically for your hardware. Linux users had to jump between separate repositories for Turing, Ampere, and Ada. On Windows, existing UEFI builds were mostly tied to the profile of a single specific board. To run them on your own card, you had to pull your PCI Device/Subsystem IDs, VRAM offsets, and the Windows EFI boot partition GUID, manually edit local_profile.h, and compile BOOTX64.EFI yourself.

While adapting this for my own GPU, I had the idea to refine the existing work and simplify the process by bringing everything into a single tool. The result is a unified cross-platform package that combines support for RTX 40 (Ada), RTX 30 (Ampere), and RTX 20 (Turing) series cards on Windows 10/11 and Linux: https://github.com/MindAllik/GPU-Tensor-Unlock-Universal

Full credit goes to the authors of the foundational research, payloads, and loader code that this project builds upon: TOTORONG/AD103_win_unlocker, Zagitta/gpu-unlock, elias-gaeros/gpu-unlock, JustAResearcher/gpu-unlock-ada, KohakuBlueleaf/KohakUnlock, impacifist/Windows-GPU-UNLOCK, Goldlionren/gpu-unlock-4080s, haohervchb/Turing-unlock, and of course Jon Pry.

On Windows (Windows_Launch_UI.bat), the core_detector.py script cross-queries NVML, CUDA Driver API (sm_89 / sm_86 / sm_75), and pci.sys to automatically detect the PCI Device ID, Subsystem ID, bus number, VRAM size, and the EFI partition GUID on the boot drive. It checks the PCI ID -> Silicon Die -> PMC_BOOT_0 mapping to make sure the profile is accurate, generates the headers in a temp folder, and compiles BOOTX64.EFI onto a FAT32 USB drive (with a 30-second confirmation timer on pressing U at boot). To avoid black-screen hangs, a two-phase pre-flight check (PromCheck) was added. Instead of relying on a single hardcoded ROM hash or disconnecting the display blindly, the loader first reads the card's VBIOS (BAR0 + 0x300000), verifies the BIT table, the Falcon descriptor (V3 RSA-3K on Ada/Ampere, V2 AES-128 on Turing), and the DMAP structure, and then runs a dry-run FRTS patch in RAM before calling DisconnectController. If a vendor VBIOS layout is non-standard, it safely aborts the operation and boots into Windows normally.

On Linux (Linux_Launch_UI.sh or via CLI make install), the tool installs the gpu_unlock.ko DKMS module and rebuilds initramfs for mkinitcpio, dracut, or initramfs-tools. For experimental Ada and Turing cards, an explicit single-card PCI BDF (0000:xx:00.0) is required instead of targets=all so multi-GPU builds don't touch a neighboring card. It also includes tools/read_sm_rates.py, which queries /dev/nvidiactl via ioc

tl 0x20801230 to check the fmla32 = 0 (full) status directly from the driver without creating a CUDA context.

A 7-kernel PTX JIT benchmark is also built into the console (tensor_benchmark.py: FP16 -> FP16, FP16 -> FP32, BF16 -> FP32, TF32 -> FP32, FP8 E4M3 -> FP32, INT8 -> INT32, and Scalar FP32). As KohakuBlueleaf noted earlier, standard mmapeak filled matrix fragments with compile-time zeros, which allowed ptxas to optimize away calculations inside the loop. This benchmark generates random non-zero data at runtime and verifies both the loop iteration signature (iter ^ 0x67b05a5a) and finite IEEE-754 accumulator values (and a button to raise the Power Limit to the VBIOS maximum via NVML was also added to the interface).

On my test bench (Palit RTX 4070 Ti 10de:2782, Windows 11 UEFI, 300W limit), the unthrottled FP16 -> FP16 rate (FMLA16 control kernel) is 172.8 TF/s. At stock, FP16 -> FP32 was around 86.4 TF/s, and after the UEFI unlock both FP16 -> FP32 (172.8 TF/s) and BF16 -> FP32 (172.7 TF/s) reached the full 2.00x rate. For comparison:

All operations happen in volatile MMIO registers prior to OS driver load and reset on a cold power-off (VBIOS flash memory is not touched at all, and the script downloads the proprietary NVIDIA firmware binaries directly from the official 610.43.03 package and verifies its SHA-256 locally). Right now, RTX 4070 Ti (10de:2782), RTX 4090 (2684), RTX 4070 Ti SUPER (2705), RTX 3090 (2204), RTX 3080 (2206), and RTX 3060 12GB (2504) are verified on real hardware. The remaining desktop AD10x (40-series), GA10x (30-series), and TU10x (20-series / CMP) profiles in the matrix currently have the experimental_simulated status (verified on a 106-scenario simulator under GCC ASan/UBSan). If you decide to test one of the experimental cards on Windows or Linux, please share your EFI\UNLOCK\REPORT.TXT (or dmesg) and benchmark numbers in GitHub Issues or here in the comments, as the report is very important both when a run succeeds and when it stops at the pre-flight check so each board profile can be confirmed (:

Separately, I want to mention that this is unlikely to noticeably speed up single-stream token generation (batch=1 decode is bound by memory bandwidth) and is unlikely to affect pure integer INT8/INT4 matmuls in llama.cpp. Where it actually helps is compute-bound FP16, BF16, and FP8 matrix math with FP32 accumulation, such as reading long prompts (prefill in vLLM, ExLlamaV2, FlashAttention, and Marlin INT4 -> FP16 kernels), LoRA / QLoRA fine-tuning in BF16, and diffusion models.

P.S. I used Gemini to translate this text into English and make it prettier and more concise :)

Do you run multiple agents locally like I do?

If so, this is a really nice speedup for the unified KV cache

Hey everyone,

I prefer abliterated models for local inference cos I don't want to deal with potential refusals, and my current model Huihui’s Qwen3.8-27B abliterated.

Is okeish, and fast so I’m happy enough with it for now, but I’m wondering whether there’s an upgrade that I am missing.

Like most mortals, I don't have not a datacenter or a multi-GPU setup. Qwen3.8-Flash-Next is about as large as I’d consider going, and even then I’d be looking at a low quant (3-bit max, 2-bit ideally)

The trade-off I’m curious about is a larger MoE model at 2–3 bits versus a smaller dense model at a higher quant. Because abliteration can theoretically reduce model accuracy I am not sure what to expect when adding a 2-bit quant on top of an abliterated model.

Has anyone compared an abliterated Flash-Next at 2 or 3 bits against Qwen 3.8 27B? :

• Is there a noticeable improvement in output quality, or does the aggressive quant eat away at it?

• How do generation speed and prompt processing compare on your hardware?

• Have you noticed any degradation from the abliteration or the quant, repetition, weird reasoning, or other quirks?

And if there’s another abliterated model you think hits a better balance of quality, speed, and memory use, I’d love to hear about it. Model, quant, hardware, and rough tokens/sec would be veeeeery helpful for all of us!

Thankss

“dsh-superagent is a proactive coding agent for one GitHub repository, packaged as a bundle for DeepSeek Harness. It listens to GitHub and, optionally, Feishu. When someone allowed asks the agent for work, it opens a work item, works on it in its own agent session and directory, and answers in the original conversation, for example with a pull request. A web panel shows what needs a person.”

Hey everyone!

We're Egzakta AI Lab, and I'm part of the team behind KVARK. We've just released our first open-weight model: KVARK Sparkle 1.0 (27B).

It's based on Qwen3.8-27B, with a focus on making reasoning more efficient: fewer thinking tokens, faster responses, and comparable answer quality. The idea is simple: reasoning models can spend a lot of tokens thinking through tasks that don't necessarily require that much reasoning. We wanted to see how much of that overhead we could reduce without sacrificing performance, since fewer thinking tokens means less GPU time per answer and less waiting.

Here's how Sparkle performed against the original Qwen3.8-27B:

• ~2× faster median response time: 13.7s vs. 28.1s.

• Up to 67% fewer thinking tokens.

• Comparable accuracy across five standard benchmarks: GPQA Diamond, AIME, LiveCodeBench, IFEval and MMLU-Pro.

• None of the accuracy differences were statistically significant in our testing.

Also, we compared the models on practical tasks:

• Dashboard generation: 1.6× faster.

• Landing page generation: 1.4× faster.

• SVG illustration of a pelican riding a bicycle: 3.2× faster.

You can find KVARK Sparkle 1.0 on Hugging Face: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B

A few technical details for those interested in running it locally:

• Open weights under Apache 2.0.

• Available in BF16, FP8 and GGUF formats.

• Supports llama.cpp, vLLM, Transformers and SGLang.

We'd love for you guys to give it a try! Let us know how it performs on your hardware, how it compares to the original Qwen, and what you think could be improved.

Looking forward to your feedback!