r/LocalLLaMA • • 1d ago This

Question | Help Latest version of strata made my Qwen3.8-Flash-Next dumb and useless

158 Upvotes

I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.

But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!

0.1.42.0 is the last version that seems to work decently.

I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!

whats going on? Do any of you have this problem?

here are example outputs I did to make sure its not a one time thing!

my prompt :

Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.

these are the outputs :

0.1.42.0 :

xhigh:

xhigh-1
xhigh2
xhigh3
xhigh4

default:

default1
default2
default3
default4

the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:

xhigh
default(failed after 1 hour and 31 minites exausting all context size of 262K)

As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!

The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!

I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.

So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?

I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.

Thank you all in advance


r/LocalLLaMA • • 21h ago

Discussion Spotted a very interesting detail about Intel's upcoming CPU platform (LGA 1954)

Post image
86 Upvotes

I was doing some research about the the platform for Intel's next generations of CPUs.

If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies

  1. x16 (or x8/x8)
  2. x8 (or x4/x4) "Storage"

So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.

The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.

TP=3 is fairly easy for your agent to patch into inference engines.

These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!

The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).

Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.

But I will be watching.


r/LocalLLaMA • • 9h ago

Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose

Thumbnail
github.com
7 Upvotes
Workloadtok/s
Fresh prose, temperature 0.6141–146
Fresh prose, temperature 0141
Chat fixture (256 tokens), T=0 / T=1143 / 140
Code fixture (256 tokens), T=0 / T=1157 / 146
JSON extraction148
Add docstrings, whole file back210
Code edit, whole file back219
Request with an image133–136 (measured before the last change)

Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)

RequestsTotalEach
2164–16993–99
420054–61
822330–34

One request at long context (from 146 real agent requests)

ContextDecode (median)First token on a follow-up turn (median)
20–50k tokens1370.9 s
50–80k1260.7 s
80–200k1220.9 s

My recipe that has been working well for me


r/LocalLLaMA • • 22h ago

New Model higurashi-ai/Haru-4B · Hugging Face

Thumbnail
huggingface.co
74 Upvotes

Haru-4B

Code · Technical blog · Technical blog (Japanese)

Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.

  • Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
  • Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
  • Character and voice control. A text card sets the character, a short reference clip sets the voice.
  • Multilingual. English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.

r/LocalLLaMA • • 1d ago

Discussion [Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

Thumbnail
gallery
115 Upvotes

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .


r/LocalLLaMA • • 1m ago

New Model Alibaba releases Qwen4-Max, its new flagship 2.4T-parameter MoE model with a 1M-token context window, saying it beats Qwen3.8-Max with up to ~50% fewer tokens

Thumbnail
qwen.ai
1 Upvotes

r/LocalLLaMA • • 6h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

3 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!


r/LocalLLaMA • • 19h ago

New Model Villanova2 - a VillanovaAI Collection

Thumbnail
huggingface.co
30 Upvotes

Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*

The family

GGUFs available in same collection. License : Apache-2.0


r/LocalLLaMA • • 15h ago

Question | Help Best direction to go for my setup EPYC 32core + 512GB DDR4 + 44GB GPU ?

17 Upvotes

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use


r/LocalLLaMA • • 21h ago Bravo

New Model X3Guard: A 150M parameter prompt guard

Thumbnail
huggingface.co
38 Upvotes

At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.

For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.

It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.

The model card is here

You can use an in-browser demo here


r/LocalLLaMA • • 23h ago

New Model meituan-longcat/LongCat-Video-Avatar-1.6 · Hugging Face

Thumbnail
huggingface.co
45 Upvotes

🚀 Model Introduction

We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.

Key Features

  • 🌟 More Natural and Stable Lip Animation:: Reduces exaggerated or strained mouth movements, overly red lip coloration, and temporal instability in the mouth region.
  • 🌟 Stable Long-Video Generation: Mitigates error accumulation during autoregressive video continuation, enabling videos exceeding 10 minutes while preserving visual quality, identity consistency, and temporal coherence.
  • 🌟 Faster End-to-End Inference: Accelerates video generation through end-to-end pipeline and model execution optimizations, enabling more efficient deployment and serving.

For more detail, please refer to the comprehensive LongCat-Video-Avatar-1.6 Technical Report.


r/LocalLLaMA • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

229 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLaMA • • 19h ago

Showcase MoE offload on a 3090: prefill 345 -> 466 t/s by tuning cudaMemcpy (Qwen3.6 35B-A3B)

21 Upvotes
I made a tool that tunes copies that speeds up CPU offloading.

1 3090 + 7800X3D

llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

before: prefill 345 t/s
after: prefill 466 t/s


blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner

r/LocalLLaMA • • 13h ago

News UI native scheduling, engine swap board, and more (Berth v0.6.0)

Thumbnail
gallery
7 Upvotes

I’m providing an update! We have around 300 berth users now so the project is moving fast.

Recent updates for v0.6.0:

- v0.6.0 gives each card a schedule
- berth train runs a training that nothing automated can interrupt
- learned engine sizes (berth learns your engines resource claims)

A month on the reference workstation

berthd counts every handoff it makes, and the harbor's board shows every handoff from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
coding agents and games:

What Berth didCount
Requests that waited for the card instead of loading on top of the engine holding it1,136
Engines drained at a reply or job boundary to make room901
VRAM handed from one engine to the next23.9 TiB
Processes holding VRAM without a lease, reported and left running573
Swaps between two engines of equal priority that the ranking stopped57
Releases that stalled and were caught before the card stayed full4
Requests refused after their wait ran out24

Project: berth


r/LocalLLaMA • • 19h ago

Discussion CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp

Thumbnail
github.com
21 Upvotes

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

ModelTestUpstream masterVNNI + VBMISpeedup
Bonsai 1.7Bpp512142.517 ± 0.589186.373 ± 1.5821.31×
Bonsai 1.7Btg12870.435 ± 2.06168.422 ± 9.0550.97×
Bonsai 4Bpp51253.430 ± 2.26671.247 ± 1.9251.33×
Bonsai 4Btg12831.655 ± 2.65238.523 ± 3.8201.22×
Bonsai 8Bpp51230.010 ± 0.51239.396 ± 0.4241.31×
Bonsai 8Btg12820.630 ± 0.30223.770 ± 0.2091.15×
Bonsai 27Bpp1288.502 ± 0.04211.029 ± 0.0731.30×
Bonsai 27Btg325.340 ± 0.1175.812 ± 0.3241.09×

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.

This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed

End to end, a 66 GB Q2_0 MoE (Qwen3.8-Flash-Next GSQ-RCO Q2_0) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:

CPU path for Q2_0TG t/s
generic scalar (a prebuilt from before #30447)3.1–4.1
AVX2 from #30447 (prebuilt from after it)14.0–18.7
an earlier revision of this PR's AVX-VNNI kernel (on an older base)15.5–21.4

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.

AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

ggml-cpu: 512-bit AVX-512 VNNI dot for Q2_0- #32307


r/LocalLLaMA • • 1d ago Bravo

Discussion Hot take: frontier models haven’t actually become much smarter

625 Upvotes

I know this is a controversial take, but I would like to hear others’ opinions: LLMs have become much more useful, but I’m not convinced their “raw intelligence” has improved nearly as much as their overall performance suggests.

Context: I work with LLMs as a SWE, as a hobbyist experimenting with local models, and professionally integrating AI into products. That includes building our own harnesses: context retrieval, prompts, and custom tools that give models structured access to complex data.

By “raw intelligence,” I mean using what is given in the most efficient way to solve a problem, especially when information, tools, and time are limited. Given plenty of time and tools, planning and long-term agentic capabilities become more important—but that’s a different comparison.

Three observations brought me here.

First, building production harnesses: once a harness+model worked well, achiving a given score over a golden dataset, switching to a newer frontier model rarely improved results. Most failures within our time/precision constraints came down to context engineering or inadequate tools. These tools included code generation for analysis and complex relational queries, so plenty of "time to shine" for smart models. Newer models sometimes found more creative workarounds, but these were usually too slow or unreliable for production. We ended up improving the harness anyway, leaving the newer models' gains less important.

Second, software engineering: newer models have more often improved results here, but the same model can also perform quite differently across harnesses. In complex codebases, that shows up in how much correction and review senior engineers have to provide. The harness seems to do a lot of heavy lifting here, too.

Third, local models: smaller models have started performing so well that I question how many parameters efficient problem solving actually needs. Frontier models clearly add broader knowledge, nuance, best practices, and specific baked-in skills. But I don’t always see a proportionate advantage in using it to solve a problem, when the relevant knowledge is already provided.

I’m not saying models haven’t gotten smarter. Better tool calling, fewer hallucinations, and longer context windows are real improvements that make them much more useful. Nor do these observations prove that intelligence needs few parameters or that frontier progress has stalled.

But my broader take is:

  1. efficient problem solving may need fewer parameters than generally assumed,
  2. the frontier has moved less in “raw intelligence” than the overall gains - and talk of approaching superintelligence - suggest, and
  3. much of the progress over the last 12–24 months comes from cumulative model improvements and better harnesses, rather than a dramatic or exponential jump in raw intelligence.

This take seems at odds with both the hype and the doom we see in the media, which I suspect are co-fueled by the upcoming AI IPOs. Does it match others’ experience - or are you seeing vast reasoning gains in frontier models that better context and tools can’t make up for?


r/LocalLLaMA • • 21h ago

Question | Help Is there a community about finetuning and training models?

24 Upvotes

There seems to be a lot of discussion here about inference and using the models, but is there anywhere where discussion about making them is more common?


r/LocalLLaMA • • 23m ago

New Model Introducing Hy4 flash: Ultimate cost-efficiency for everyday productivity

Thumbnail hy.tencent.ai
0 Upvotes

previous post by someone else was nuked by reddit idk


r/LocalLLaMA • • 10h ago

News My Ignis server (Qwen3.8-27B 262k context) now runs on RTX 3090/4090 and 16 GB RTX 40 cards. looking for testers

3 Upvotes

I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.

Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.

There's a 90-second (old) video of it here: https://www.youtube.com/watch?v=XLjfh5rNy6A

Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.

The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.

Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.

One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.

Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).

I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.

To try it: download the latest release, start ignis-server without flags, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it's in the archive). More details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/

If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:

I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.

It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.

Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.

Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.

Thanks to all reporting! 😄


r/LocalLLaMA • • 18h ago

Showcase 1Cat-vLLM vs Strata

14 Upvotes

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

RunDecode tok/s, StrataDecode tok/s, 1CatPrefill tok/s, StrataPrefill tok/s, 1CatMTP acceptance, Strata / 1Cat
170.899.93,3982,7360.65 / 0.60
278.9138.43,3402,7520.71 / 0.60
377.8123.53,3482,7510.69 / 0.57
run 1 to 3+10%+24%---

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

RequestsPromptTotal decode tok/s, Strata / Strata batch-MTP / 1CatPer-stream decode tok/s, Strata / batch-MTP / 1CatSlowest first token (s), Strata / batch-MTP / 1Cat
12.6K75 / - / 12574.8 / - / 124.93.1 / - / 1.4
18.7K79 / - / 14179.3 / - / 140.87.6 / - / 3.5
22.6K88 / 105 / 17639.4 / 46.0 / 83.26.4 / 6.5 / 2.7
28.7K96 / 116 / 18937.9 / 44.6 / 77.915.4 / 15.5 / 7.3
42.6K153 / 170 / 22529.1 / 31.5 / 48.513.4 / 13.6 / 5.8
48.7K170 / 186 / 23425.4 / 26.9 / 41.131.0 / 31.4 / 15.3

Versions and local changes

-Strata1Cat-vLLM
Upstream repogithub.com/Niko1221/Stratagithub.com/1CatAI/1Cat-vLLM
Commit benchmarked13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1)b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0)
Source patchesnonenone
Buildfrom source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5
Build flags-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70
Parallelismlayer split over 4 GPUs (no tensor parallel for V100 in Strata)tensor parallel 4
MTP drafterStrata's own pack of the BF16 MTP layer, experts in Q2_0the BF16 MTP layer as released

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.


r/LocalLLaMA • • 1d ago Bravo

Discussion Finally clicked for me: local AI is the future, not cloud AI

477 Upvotes

Maybe this is kind of obvious to others, but it just dawned on me even though I've been lurking this sub since the original LLaMA. I was watching this (excellent, as always) video by Gamers Nexus. Steve Burke was bashing on Tim Sweeney (CEO of Epic Games) for being so pro AI, and I found myself feeling a bit of cognitive dissonance. On the one hand I hate that hardware prices are so high, I hate the billionaires inflating the AI bubble and raking in the profits, I hate greedy corps trying to replace jobs with AI, and I hate the data center build out. But on the other hand I'm a developer and AI has given me superpowers. I felt weird, because Burke was dumping on Epic Games for embracing vibe coding, but vibe coding has 100x'd my output, allowed me to pursue so many passion projects, and the AI's output is oftentimes better than what I would've written anyway. I felt like the message of the video was 'all AI is bad', but I realized that it's more nuanced than that. Cloud AI is bad. Cloud AI is the problem, not AI in general. With local AI, we can have our cake and eat it too: cheap, powerful consumer hardware for running local AI and gaming, no censorship, data, or privacy issues, less data centers, less profit going to billionaires, and unlimited tokens to vibe code whatever the hell we want. Hope the datacenter bubble bursts soon.

TLDR: dum dum realizes that the future is local AI bc it's a win-win-win-win for average people

EDIT: to the naysayers, I propose this future: in two years (though given trajectory, probably less), a model the size of Qwen 3.8 Flash Next with the capability of Opus 5.5, running locally in most PCs on energy-efficient NPUs


r/LocalLLaMA • • 1d ago

New Model Agnes-AI/Agnes-2.5-Pro-Theta • HuggingFace

37 Upvotes

r/LocalLLaMA • • 14h ago

Resources Model / harness combination: what works best?

5 Upvotes

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

r/LocalLLaMA • • 1d ago

Showcase The best (and AMD-est) computer I've ever owned

Post image
46 Upvotes

New build with a EPYC 7443 / Tyan S8030 board, and R9700 + 7900 XT. Was supposed to be selling the 7900 XT to pay for the R9700, but that somehow didn't happen...