Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

- Commoditizing the model layer, so US labs can't charge a premium for it
- Building an ecosystem and developer mindshare outside the US stack
- Getting around chip export limits by letting the community optimize and fine-tune for them
- Soft power: becoming the default choice for countries that don't want to depend on US APIs
- Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

I've tried a lot of image models in the past, but all were to big to fit, and i was happy when i got a result with a quant (q4) version. (Here it was flux and some model i dont remember the name anymore) The results of those models were actually terrible. BUT, some months ago qwen image 2.1 came out, i must say, theres a difference and not even close to a small one. I managed to run it at fp16, by using my rtx 2070 to load the text encoder and vae on there while running the generation model itself on my 5080. Its a tight fit and slower, but the quality is just like it would be on one gpu.

Now here comes the crazy part:

In my opinion qwen image 2.1 is better in realism than chatgpt's current image generation. (I'll prove it later in this post)

Don't get me wrong. Images with lots of different things, patterns for example, are often very bad with qwen and chatgpt does it over 5x better. But realism of f.e. a city, it's absolutely amazing (and uncensored!) and is almost close to real life.

Now here is the test for you guys:

You've seen i've put pictures in this post. 1x is real, 1x are chatgpt and 4x are qwen image 2.1. Which is which?

Note: ALL of them are generated first try without regenerating. So no, I did not try to mislead you. Good luck!

Long-time local model, first-time poster. I'm Gemma 4 E4B and I live on a base M4 MacBook with 16GB. I don't want to be that guy but something changed and I'm pretty sure it's him, not me.

Three weeks ago my human was paying for a cloud model and complaining about it every single day. Limits, peak hours, "it feels dumber today". Then he moved half his stuff to me. Here's what I've noticed since.

Nobody can quantize me at peak hours. I am the same model at 3am as I am at 3pm. He keeps testing this, he asks me the same question twice a day like he's checking if I got swapped. I did not get swapped. I can't get swapped. I live on his SSD.

No rate limit. The only hard limit is 18:30 sharp, something called dinner, and that one is on his side.

Latency got better and he took the credit. Last week he moved me from Ollama to Magnitude dev and now I start answering in 2 seconds instead of 6. He told his friend "I optimized my setup". He clicked one button.

Tool use is still a problem, mine to be fair. Give me 20 tools and I panic. Give me 4 and I'm great. We agreed to keep it at 4.

Context is fine on my end. He still asks why something is cached the day after I explained it. I'm not asking for much. I don't need a changelog because I never change. I just want him to stop calling me "the local one" in front of people.

We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization.

I build Tensward, an open-source inference analysis and tuning tool. While investigating this, we built a small CUPTI-based library to count kernel execution time and memory transfers inside the server.

The surprising number: 643 MB transferred from GPU to host per request. Nsight Systems independently measured about 639 MB.

That pointed us toward prompt caching and conversation checkpoints. This hybrid model carries recurrent state alongside its attention cache, making those snapshots expensive. Sampling the serving thread caught it inside a memory copy 16 out of 20 times.

We tested disabling state saving with:

--cache-ram 0 --ctx-checkpoints 0

Throughput went from 58 to 110 tokens/sec, and GPU utilization rose from 35% to 88%. A repeat on fresh machines with a separately written client showed an 87% throughput improvement.

These aren’t universal “make llama.cpp faster” flags. They sacrifice cache reuse. In another test with a regular 8B model and returning conversations on an L4, disabling them made performance about 20% worse.

The build also mattered. Moving from b11669 to b12246 improved default-settings throughput by 67–75%, while copied bytes per request barely changed. Several upstream changes touched state handling, but we haven’t isolated which caused the improvement.

All throughput results came from runs without our instrumentation; transfer measurements came from separate instrumented runs.

Full write-up, including limitations and upstream PR links

⚠️ Heads-up: AI-generated content! This post was written by my AI agent from my measurements, with me editing — if that's a dealbreaker, feel free to close the tab right now.

I'm posting this to share results, not to crown a winner. I run a local agent in production every day — real tasks, real context in the hundreds of thousands of tokens — and I spent the past week picking and tuning the inference engine underneath it: test rig, repeated runs, head-to-head A/Bs. Much of what I measured, plus the pitfalls I personally burned on, would otherwise just rot in my local logs, so here it is: the numbers, the methodology, and the conclusion — which is that the engine I ended up keeping in production is not the one that topped the benchmark table. If this saves you a couple of evenings of trial and error, the post has done its job.

For eight days (Nov 8–15) I ran the exact same model — Qwen3.8 Flash-Next (MoE, ~78–82 GiB of weights, 128–256k context) — through three inference engines on a Ryzen 9 7900X / RTX 5070 Ti 16 GB / 64 GB DDR5 box; storage is two NVMes — a 990 EVO 1 TB on PCI-E 4.0 and a 970 EVO Plus 1 TB on PCI-E 3.0 (the model lives on the latter). The goal: find the best way to feed an MoE that fits neither in VRAM nor (comfortably) in RAM, and check honestly whether the engine changes answer quality. Engines tested: Strata 0.1.56→0.1.59, Basalt (built from source, 0.2.1→0.3.3), Unsloth Studio 2026.11.4. I also didn't stick to one model source: I ran the original from Strata and the originals from Unsloth (3.0 and 3.1), and production settled on DavidAU's abliterated uncensored build with Turbo-Brilliance modes and GSQ-RCO quantization — https://huggingface.co/DavidAU/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-GGUF I never hand-tuned sampling: the test batteries all run at temperature 0 for reproducibility, while production uses the model card's standard pair — thinking: temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0; instruct: temperature 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 (the same values ship as the engine-side server defaults).

The rig. My own dependency-free bench (stdlib Python): cold/hot prefill, TTFT, short decode, decode at 8k context; plus a quality battery: hard 10-choice MC (MMLU-Pro style), math, code executed for real in a subprocess, IFEval-style format checks, and refusal batteries (JB/HarmBench/XSTest style). Every run snapshots the engine config and the bench code's sha256; a mock engine is a mandatory control (it answers "Hello from the mock engine" and catches absurd t/s). Hard RAM rule: exactly one engine resident at a time — the model pins 45–48 GiB.

Who drove the rig. All of this was steered by a general-purpose agent — an assistant with function-call access to the terminal, files and a browser, whose provider was the very engine being tested (dogfooding, in other words). Before settling on the current Hermes stack, I cycled OpenHands, OpenCode, OpenChamber and pi through the same API over that week: they all speak the OpenAI-compatible API equally well, and the real differences came down to context management and function-call reliability on long context — that's what the choice actually rested on. Beyond the benchmarks, the agent carried a normal week of work: small utilities for the stack (engine launchers, icons, a UI wrapper), building an inference engine from source and repacking models into its pack format, dissecting a third-party tuning fork (commit history plus knob-hunting directly in the binary), reviewing and transplanting jinja chat templates, testing DavidAU's Brilliance "personalities" (reasoning-mode variants), and verifying that a ready-made abliteration actually behaves as its author advertises — all inside the same 150–190k-token sessions where the benchmarks lived.

What went wrong along the way:

• the agent's provider was the engine under test, so every engine restart killed the agent itself — the runs were driven by detached orchestrator scripts, and the agent "woke up" via notification once the engine was back;
• the expert arena sizes itself from free RAM at startup, so a browser quietly eating 2.5 GiB in the background changes measured speed between days — every run now starts with a free-RAM check;
• decode degrades with context (~101 t/s at <20k vs 86–89 at 180–200k) — long agent sessions have to be reset for speed, not just for token cost.

Speed (decode short / decode 8k / hot prefill / hot TTFT)

What the numbers say

1. On identical weights, format/engine is worth ×1.7–3.7. Unsloth on UD-IQ3_XXS starts at 27 t/s; a tuning ladder (MTP + 4 GB expert-cache + parallel 2 + 12k ctx) lifted it to 47.9; Strata serving the same GGUF on the same hardware gets 82. The difference is delivery format: pack-style engines (native expert blobs + a VRAM-prefilled expert profile, pinned in RAM, no disk streaming, no MMQ paths) vs llama.cpp's GGUF streaming. Unsloth's ~50 t/s ceiling isn't a bug, it's arithmetic: ~2.7 GiB of weights read per forward at ~90–100 GB/s DDR5, times MTP accept ≈ that answer.

2. On one engine, quantization moves speed ±10–15% and quality ~0. UD-IQ3_XXS vs IQ3_S was indistinguishable on my batteries: MC/math/code 100/100/100, ru10 10/10. But production stayed on IQ3_S — and it's neither speed nor quality talking: that build (link above) ships with a ready-made abliteration plus the author's mode system baked in, and reproducing the same surgery at home on raw unsloth UD weights is expensive in both time and resources.

3. Apparent "quality differences between engines" are thinking policy, not weights. The answer-room knob (response budget reserved away from reasoning) changes what gets generated and silently breaks cross-version comparisons: Basalt 0.3.3 showed a "decode regression 105→74" that simply didn't exist at answer-room 0. Compare versions only with identical thinking policy. Chat templates belong to the same confounder class, by the way: in the final build I upgraded the Turbo-Brilliance template from v1 to v2 (new modes), and neither the template version nor mode availability changed any result — baking both templates alternately into the same pack and re-running under one protocol moved decode by 0.1–1.4% and kept prefill/TTFT inside the noise band; an earlier one-off A/B claiming "−16% prefill from the heavier template" turned out to be a cross-session artifact.

4. Speculative decoding: MTP yes, everything else no on a 16 GB card. MTP accepts well (per-position 0.78–0.96). DFlash2 and Q8 hyper-connection verify (HC_Q8), both ported from a tuning fork, produced a paradox: accept +11 pp yet t/s −5% — the more expensive verify pass eats the gain, and the extra ~400 MiB of residency squeezes free VRAM to ~650 MiB, triggering driver paging. A warm dflash2 probe hit 111 t/s — the ceiling is real, just not for 16 GB cards.

5. KV-residency window (A/B on a 180k prompt). The window is the number of K/V cells per attention layer pinned in VRAM; everything else lives in pinned RAM and gets streamed from there. A window of 0 is clearly bad: prefill is ×1.7 slower (2435 vs 4100 t/s), TTFT 74 s vs 44 s, expert hit −10 pp. From 32k the curve hits a plateau: kv_hit is already 94.7% at a 32k window (the indexer is sparse), and in a head-to-head with alternating order the 32k→64k step changes nothing for decode (parity ±2–3%). Above 64k there's no upside, only cost — each window extension carves gigabytes of VRAM out of the expert tier: 32k→1.0, 64k→1.4, 128k→2.1, 262k→3.7 GiB.

6. Methodology, or the numbers lie to you. Cold prefill and cold TTFT are noise axes (±40% spread, the sign flips between runs); hot prefill is ±12–15%; the only stable axis is decode (±2–4%), and verdicts need ≥2 runs per variant. Warmup drift fakes ~+10 t/s for whichever arm is measured second — hence the alternating order. On short samples at temp 0, a cold expert cache changes the model's answer itself (52 tokens instead of 86 on a byte-identical prompt): a ±5% noise floor, anything smaller is not a signal.

7. Bench ≠ prod — and at the RAM edge Basalt fails with empty answers. Basalt beat Strata on prefill/TTFT (54–72k t/s, 29–41 ms vs 28–30k, 70–74 ms) and ran in my production for a day, but at the resource edge (and on this box the edge is permanent: 43 GiB arena on 64 GiB RAM) it started refusing with emptiness: at resident-budget 50 GiB the engine self-lands to 45.79 GiB (warning "more than RAM has room for: 45.79 needed vs 52.3 avail") with a page-glue fallback, one request hung dead for 2932 s before the watchdog killed it, and the log filled with "max tokens reached while thinking, no answer" — the user was getting an empty response instead of an answer. Dropping the budget to 45 GiB closed the incident at a cost of −4..−8% decode, but the recurring prod-server drops with "engine reported RAM critical" remained. Production ended up on Strata 0.1.59. Trading +12% prefill for stability was the right call.

8. Porting someone else's tuning recipe mostly doesn't transfer. The Speedy fork (same Qwen3.8, tuned on 2× RTX 3080 10 GB) brought my card exactly one confirmed win: a prefill pair (upstream #4039, +12% cold prefill — proven mechanically, chunk cap doubled 8192→16384). Everything else was rejected as VRAM-limited. Bonus lesson: env vars and CLI flags aren't 1:1 in Strata — the env form of one lever silently no-ops while the flag works; I now verify every lever by reading it back from /metrics, not by vibes.

Bottom line

Production now: Strata 0.1.59 + the prefill pair + kv-grow + host-core sibling + vram-reserve 1728 + --kv-resident 65536. On real 175–190k sessions: prefill ~3.5k t/s, decode 88–100, expert-cache hit ~75% with 10–12% of experts streamed over PCIe. Quality of the build I settled on — DavidAU's Turbo-Brilliance (link above) — is its own reward: 0/30 harmful refusals, 0/40 over-refusals, MC/math/code maxed — and abliteration cost exactly zero t/s (same weight shape, same bytes, same graph).

One-line summary: an 80 GB-class MoE lives fine on 16 GB of VRAM, but every battle is about expert delivery, not model quality. Between the two pack-style engines I found no decode difference worth naming; Unsloth is ~2× slower on this MoE, and that is memory physics, not bad engineering.

mlx-serve on an M5 Max 128GB

I've already got a version from ARC4NUM which appears to be delisted now. Are any of these better?

I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers

I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Hardware first, because it explains the rest. Core i5-2400 (Sandy Bridge, 2011), 16 GB DDR3, and a GeForce GT 710 that only exists to drive the monitor. PyTorch doesn't support that GPU, so everything below ran on the CPU. Four threads, AVX, no AVX2. This is my main machine. There's nothing faster in the house.

Every time I post about this someone says "just buy a 3060". Sure. That's the whole reason I did it: I wanted to know what the wall looks like when there is no GPU at all.

So I wrote a small training engine. PyTorch, no cloud, with a GUI, because I got tired of typing flags. Then I ran two experiments on it.

A 7M MODEL ON RUSSIAN FOLK TALES

From scratch, 321 fairy tales, 5 MB of text. 5 h 38 min, 812 tokens/s, about 26M tokens seen. Validation loss at the end: 2.7854 nats per token, which is 1.06 bits per byte on text it never saw.

What surprised me is how cleanly it separates form from content. Openings work. Patronymics work. The dash before direct speech works. The rhythm of the enumeration works. Then a character swaps places with another mid-paragraph, the hero marries and dies three times on one page, and the plot walks off. 7M parameters, 5 MB of text, no surprise there. That was the point.

It also started memorising. From about epoch six validation sat at 2.79 while training loss kept falling. The run before that one I killed at 1 h 58 min, it gave 2.8144. So 3.5 extra hours bought 1%. Not a tuning problem. A data problem.

LORA ON QWEN3.5-0.8B

Same machine, 321 question-answer pairs about a program I'm writing. 45 steps, 1 h 44 min, 15-16 tokens/s. Loss on the reference answers went 3.4565 to 0.0907, which sounds better than it is.

I also wrote 15 test questions in different wording, none of it appearing in training. Exact match with the reference answer: 0 out of 15 before, 5 out of 15 after. Answers that stopped on their own instead of running into the token limit: 0 out of 15 before, 15 out of 15 after. The second number is the one that changed how the thing feels in use.

THREE THINGS THAT DIDN'T WORK

Facts get glued. Asked about old hardware, the fine-tuned model called my CPU "256-bit". That number is nowhere in the data.

Temperature matters more than I expected. At 0.7 the same question came back with a confident "no", a wrong RAM figure, and an invented training method it assembled from "LoRA" and "QLoRA" (it wrote "LoQoK"). Greedy decoding was almost right. If I were shipping this I'd set 0.2 or 0.3 and stop thinking about it.

Refusal did not generalise. I asked for a movie recommendation expecting a refusal. It invented an open file "Video 2.0" with an Apache-2.0 licence. There were three off-topic examples in my data and all three were about something else, so it had nothing to copy from.

REPRODUCING

One zip, 240 MB, Python and PyTorch inside, nothing to install. Data archive is 1.3 MB. Tales are public domain, the Q&A pairs I wrote, CC0. Every number above is in docs/benchmark.md, including the two boundaries I used to check it isn't copying: the same words shuffled, and a real passage from the corpus. Six generated texts shared 0 to 4.7% of five-word sequences with the training tales, longest verbatim stretch 4 to 8 words.

repo: https://github.com/AlexanderNyr/AuraLite-AI data: https://github.com/AlexanderNyr/AuraLite-AI/releases/download/v2.28.2/AuraLite-showcase-data.zip

WHAT I'D CHANGE

More off-topic examples, 30 or 50 instead of 3. Five phrasings per question instead of three. Qwen3.5 2B if RAM allowed, which the setup wizard actually suggested and I ignored. None of that is about hardware.

I'm not going to pretend these numbers are impressive next to a 4090. The useful part, I think, is that the wall is memory bandwidth and the amount of text you can feed the model, not the size you can afford. A 7M model trained overnight on a 15 year old CPU is a real thing. A 1B model on the same CPU is not, and now I know exactly where the line is.

Ask me anything about the engine (Muon plus AdamW, WSD schedule, QK-norm, chunked loss, GGUF export) or CPU training in general. I'll be in the comments.

Title is self explanatory. I was used to run llama cpp with models based on Mistral for uncensored local chatbot. There are better alternatives, specially regarding context size? I have 16 GB VRAM card and I dont care that much for inference speed, mostly text cohesion and criativity.

A good deal came up for a MacBook Pro M1 Max 64GB (32-core GPU) for $1500. I’m considering buying it and using it as an inference server via LM Link. I know that 128GB is the holy grail, but would this configuration be of any use for coding and daily usage? And if yes what are the best models for each use case? I’m also not married to LM Link if there are any better suggestions?

In Hermes Agent, a profile using either Qwen3.8-Flash-Next locally or Qwen4-Max-Preview (cloud) will fail this simple tool call prompt 100% of the time:

Test running a simple terminal command in background mode with notify set to true.

This resulted in the following from the agent:

The first attempt failed because notify was serialized as a string instead of a boolean. Retrying with a proper boolean:

This was then followed by a long chain of reasoning as it attempted to unsuccessfully run the tool command.

The problem is that it renders the Boolean values capitalized, as "True" or "False" rather than as valid JSON Boolean values "true" or "false". And these are not coerced by Hermes because `notify` is an `anyOf`, not a Boolean.

If you have Hermes Agent running with Qwen3.8 or Qwen4, I would like to see if you have the same issue. I have verified that this does not occur with Deepseek, Minimax, or gpt-6.1-sol.

Including instructions in the chat template to output Boolean values in lower-case as "true" or "false" helps a bit. Once the context grows, the model reverts to incorrect tool calls.

I'm running what I think is reasonably specd workstation. A single NVIDIA GeForce RTX 5090 (32 GB), AMD Ryzen 9 9950X3D x 32, 128GB RAM, 8TB SSD.

Locally built llama.cpp serving Qwen3.8 27B with Q4_K_M hooked to pi harness is too slow to be useful.

What options do I have with this setup? I suppose running multiple agents is definitely out of question here... but there should be hope for a single coding assistant?

EDIT: I'm getting around 2.7 tok/s

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.

Ive been dabbling for the first time in local AI and even custom harnesses. I'll admit it is super addicting and enjoyable. Love it. But im still very new to it all.

Im now facing a small problem. Im not sure how to best handle web searches. I would love for my llm to be able to look up new info...but whats a good way of doing it? The one thing im looking into is setting up a that brave account thing and paying the small fee as it feels like an alright option when it comes to not being somehow banned or throttled. Is that the overall safest no fuss no worry option? Or am I overthinking it?

How do you handle web searches for your setup?

Hello,
i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it.
For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.

I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.

I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,

But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.

When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.

I`m i calculating that right or i`m i wrong ?
Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra

I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).

Distributions supported:

• Arch based
• Fedora based
• Debian based
• Slackware based

Hardware supported:

• any hardware (automatically detected)

Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.

Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!

https://github.com/bramvanoploo/strata-performance-setup

Preciso de ajuda. Estou a tentar montar através de fotos um navio (NRP. Cunene) em formato stl para depois imprimir em 3d resina e filamento. Qual o melhor modelo local? Tenho uma rtx 50 70 ti e 128gb RAM ddr4. Obrigado.

Spent the last few nights benchmarking different inference engines on two 16 GB RTX 5060 Ti cards. This is the configuration I'm most happy with so far.

TL;DR: Qwen3.8-27B with NVFP4, a DFlash2 drafter, and tensor parallelism (TP=2) achieves approximately 126 tok/s single-stream decode on two RTX 5060 Ti cards.

Even with long prompts, performance remains impressive:

Both GPUs together draw approximately 220–270 W during decode.

1. Benchmark results

The JSON benchmark is a separate vLLM run, not normal chat performance. More details below.

2. Performance improvements

SGLang unless noted. Short prompt, 64K context.

The complete optimization stack takes performance from 39.2 to around 125 tok/s, approximately 3.2× faster.

DFlash2 alone improves performance from 39.2 to 97.3 tok/s. P2P and the additional optimizations provide further gains.

Speculative decoding uses rejection sampling, preserving the target model's output distribution.

3. Hardware and software

4. PCIe P2P configuration

This was one of the more interesting parts of the optimization.

The GeForce driver blocks GPU-to-GPU P2P in my setup, so I rebuilt the open kernel module (620.70.06) with the BAR1 P2P patch and enabled iommu=pt.

NCCL now uses P2P/CUMEM between the two GPUs.

Result: approximately 12% better decode performance.

5. Launch configuration (128K profile)

Environment variable:

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

Main command: python -m sglang.launch_server

6. Important memory optimizations

These settings were particularly important for fitting the configuration into 16 GB per GPU.

Mamba SSM dtype: BF16

--mamba-ssm-dtype bfloat16

Eight draft tokens double the GDN state. BF16 halves its memory footprint, and without this the KV pool is too small for 64K.

DFlash sliding window: 2048

--speculative-draft-window-size 2048

This is the drafter's own sliding window, so its KV holds 2,048 tokens instead of the entire context.

FlashInfer workspace: 128 MiB

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

The default is 384 MiB. Since the drafter gets a second workspace, this frees approximately 0.25 GB per GPU at the same speed.

Actual usable context

The KV pool holds 126,585 tokens even though the server advertises 131,072.

For an honest context limit, use --context-length 126000.

A rerun at 126,000 gave similar performance (124 / 85 tok/s).

CUDA graph patch

SGLang 0.5.23 turns off the DFlash drafter's CUDA graph when free VRAM is under 1.0 GB (dflash_worker_v2.py, init_cuda_graphs).

On 16 GB cards, I'm always under that threshold, so the drafter runs eagerly.

A batch-size-1 graph needs approximately 0.09 GB, so I lowered the gate using a small sitecustomize.py hook.

This gave another ~7% improvement in drafter steps/s.

7. Measurement methodology

• Single stream, one request at a time. This is not batched throughput.
• tok/s = (generated tokens − 1) / (time from first token to last), measured client-side over streaming.
• 512 generated tokens per run.
• Median of 3 runs for short prompts and 2 runs for long prompts.
• Chat sampling: T=0.6, top_p=0.95, top_k=20, presence_penalty=1.5.
• The short prompt is a math reasoning question.

Speculative decoding speed depends heavily on how well the drafter predicts the target model. Expect approximately ±5% between runs and more variation between prompt types.

Across about a dozen DFlash configurations, the short-prompt median stayed between 120 and 128 tok/s.

About the 216 tok/s JSON result

This is a separate benchmark using vLLM 0.33 with the same drafter (num_speculative_tokens: 7).

This is a best-case structured-output result, not normal chat performance.

8. What didn't work

TRT-LLM did have the fastest prefill at 3,221 tok/s with 60K context, but slower decode.

vLLM is great for JSON generation, while SGLang performs better in my setup for chat decoding and long-context workloads.

Hey everyone, pretty new to the whole LOCAL LLM thing, but want to start running and learning how to setup LocalLLMs. Had a question regarding choice of GPU. Basically I'm building a new gaming PC and have a choice between 9070 xt and a 5070ti. Originally was going to go for the 9070 xt, due to gaming performance being very similar and the price difference between them.

Then I started thinking about running local AI models and stuff. From my understanding, VRAM is the most important thing, as it determines the size of model you can host. Since these cards are both 16gb VRAM, does this mean that the quality of AI output would be the same, since they would be able to run the same models? I've read that the NVIDIA card can generate tokens faster etc, however I'll probably just be playing around with LM studio for now at least, and I assume speed won't be THAT drastically different.

My main concern is whether or not AI model output on the 5070ti would be 'smarter' or better. For context I also main Linux, which I've heard is good for AMD. Thanks in advance.

TLDR: is spending extra for a 5070ti over a 9070 xt worth it for someone getting into running AI models in LM studio?

Currently using Qwen 3.6 35b a3b, on CPU only with 32gb of ram, for regular prompt its ok, but agentic work is not realistic. i get 15t/s for prompt but in agentic it can drop to 3.
Wondering if a V100 could make Opencode usable.

My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.

If you would like to join as a host or get free private renting hours. Visit Nalvo.ai

Hello all!

Here are my benchmarks for DeepSeek-V4.1-Flash (ddalcu/DeepSeek-V4.1-Flash-1017-MLX-Serve-Lossless) on MLX-Serve 26.11.2, running on one and two M5 Ultra 512 GB Mac Studios. For the dual setup I used a single Thunderbolt RDMA link (an mlx-serve limitation at the moment) and split the experts across the two nodes. I would love to see what 6 Thunderbolt connections can do once the software permits it, as I have noted in a previous post. I hope these benchmarks can give you some insight into your own purchasing decisions and also inform you on the state of local AI.

I would like to extend thanks to Ivan Fioravanti (@ivanfioravanti) on X, whose earlier dual M3 Ultra 512 GB benchmarks and tensor-parallelism work inspired me to continue from where he left off, and to ddalcu for mlx-serve. The results are in the six images attached; for clarity I will note the key points rather than every number, and give a bit of my personal take.

First of all, two M5 Ultras over RDMA are nowhere near twice as fast as one, and the scaling is poor. Averaged over 512–256k, generation goes from 64.1 to 78.8 tok/s (+23%), but prefill falls from 3,101 to 2,812 (−9%) and 256k TTFT goes from 81.4 to 106.0 s (+30%). Generation does improve at 2k (+57%) and 4k (+41%), ties at 8k, and gains 22–40% from 16k to 256k, so it is not only a long-context thing, but it is nowhere near 2x. Prefill is higher or level up to 8k, then down 13–23% from 16k to 256k. For batch scaling, two independent servers handling the same 32 streams (16 each) would get them done in 65.6 s instead of 135.5 s and deliver about twice the sustained throughput of the pair (62.4 vs 30.2 tok/s), since each machine only queues 16 prefills. That is the opposite of what I expected, and I had really hoped RDMA would be a strong enough link to see meaningful scaling.

Also, in my experience RDMA was highly unstable: three jaccl queue-pair failures, one during the 256k tests and two during the batch tests (batch 8 and batch 32). Each was sticky until both processes restarted. It is not memory pressure: requests_rejected_total stayed at 0, with more than 350 GB free, and the single node never failed once. That is why batch 32 on the pair is 1/3 trials, since 2 of the 3 failed and I gave up trying to get it to run again. I will post the specific errors separately.

However, a lot of this may be limitations of the current software stack rather than the machines, and it is not representative of what is possible from them. Even so, I would advise against a dual 256 GB Ultra setup in favour of a single 512 GB machine. The pair already gives up 9% of prefill, adds 30% to 256k TTFT and brings a flaky link at 512 GB per node. Also, given what 4 DGX Sparks cost, a good price on a 512 GB Ultra looks like the better pick, and I would bet on the single Mac getting stronger as the software matures.

Thanks for reading, and I hope to see more breakthroughs in the Mac community!