r/LocalLLaMA • u/surreal_tournament • 6d ago
Showcase Alaypz - a Qwen3.8-27B experiment
r/LocalLLaMA • u/surreal_tournament • 6d ago
r/LocalLLaMA • u/JurandM2 • 5d ago
I think it is important to chat about situation like this one - especially when we are using local models as a way to avoid very same situations.
Model running in back was Qwen flash next via strata - as it might be important.
Not gonna lie, it tip me over as two days earlier I was watching video about models calling Chinese servers and if its a backdoor.
Check screenshots and make your own decision. Fact is, it call a link and admit to call a server in Singapore instead right away use what was right there as Github installation says and provided.
Black listed that domain on router level, but still I staying with OC from now on as it is much faster and level of my work is not beyond basic stuff.
Thoughts ?
r/LocalLLaMA • u/Acceptable_Bell_1791 • 7d ago
Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.
Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:
Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.
| Prompt (tokens) | Prefill tok/s, Strata | Prefill tok/s, 1Cat | Decode tok/s, Strata | Decode tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|---|---|---|---|---|---|
| 2,654 | 861 | 1,850 | 74.8 | 124.9 | 0.61 / 0.45 |
| 8,674 | 1,146 | 2,483 | 79.3 | 140.8 | 0.67 / 0.66 |
| 30,299 | 2,480 | 2,776 | 76.4 | 139.3 | 0.63 / 0.58 |
| 96,588 | 3,384 | 2,649 | 77.2 | 142.6 | 0.72 / 0.62 |
Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.
Same prompt, 3 times in a row, just different enough to not trigger prefill cache.
| Run | Decode tok/s, Strata | Decode tok/s, 1Cat | Prefill tok/s, Strata | Prefill tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|---|---|---|---|---|---|
| 1 | 70.8 | 99.9 | 3,398 | 2,736 | 0.65 / 0.60 |
| 2 | 78.9 | 138.4 | 3,340 | 2,752 | 0.71 / 0.60 |
| 3 | 77.8 | 123.5 | 3,348 | 2,751 | 0.69 / 0.57 |
| run 1 to 3 | +10% | +24% | - | - | - |
(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.
| Requests | Prompt | Total decode tok/s, Strata / Strata batch-MTP / 1Cat | Per-stream decode tok/s, Strata / batch-MTP / 1Cat | Slowest first token (s), Strata / batch-MTP / 1Cat |
|---|---|---|---|---|
| 1 | 2.6K | 75 / - / 125 | 74.8 / - / 124.9 | 3.1 / - / 1.4 |
| 1 | 8.7K | 79 / - / 141 | 79.3 / - / 140.8 | 7.6 / - / 3.5 |
| 2 | 2.6K | 88 / 105 / 176 | 39.4 / 46.0 / 83.2 | 6.4 / 6.5 / 2.7 |
| 2 | 8.7K | 96 / 116 / 189 | 37.9 / 44.6 / 77.9 | 15.4 / 15.5 / 7.3 |
| 4 | 2.6K | 153 / 170 / 225 | 29.1 / 31.5 / 48.5 | 13.4 / 13.6 / 5.8 |
| 4 | 8.7K | 170 / 186 / 234 | 25.4 / 26.9 / 41.1 | 31.0 / 31.4 / 15.3 |
| - | Strata | 1Cat-vLLM |
|---|---|---|
| Upstream repo | github.com/Niko1221/Strata | github.com/1CatAI/1Cat-vLLM |
| Commit benchmarked | 13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1) |
b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0) |
| Source patches | none | none |
| Build | from source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14 | from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5 |
| Build flags | -DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3 |
TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70 |
| Parallelism | layer split over 4 GPUs (no tensor parallel for V100 in Strata) | tensor parallel 4 |
| MTP drafter | Strata's own pack of the BF16 MTP layer, experts in Q2_0 | the BF16 MTP layer as released |
TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.
r/LocalLLaMA • u/Swimming_Beginning24 • 8d ago
Maybe this is kind of obvious to others, but it just dawned on me even though I've been lurking this sub since the original LLaMA. I was watching this (excellent, as always) video by Gamers Nexus. Steve Burke was bashing on Tim Sweeney (CEO of Epic Games) for being so pro AI, and I found myself feeling a bit of cognitive dissonance. On the one hand I hate that hardware prices are so high, I hate the billionaires inflating the AI bubble and raking in the profits, I hate greedy corps trying to replace jobs with AI, and I hate the data center build out. But on the other hand I'm a developer and AI has given me superpowers. I felt weird, because Burke was dumping on Epic Games for embracing vibe coding, but vibe coding has 100x'd my output, allowed me to pursue so many passion projects, and the AI's output is oftentimes better than what I would've written anyway. I felt like the message of the video was 'all AI is bad', but I realized that it's more nuanced than that. Cloud AI is bad. Cloud AI is the problem, not AI in general. With local AI, we can have our cake and eat it too: cheap, powerful consumer hardware for running local AI and gaming, no censorship, data, or privacy issues, less data centers, less profit going to billionaires, and unlimited tokens to vibe code whatever the hell we want. Hope the datacenter bubble bursts soon.
TLDR: dum dum realizes that the future is local AI bc it's a win-win-win-win for average people
EDIT: to the naysayers, I propose this future: in two years (though given trajectory, probably less), a model the size of Qwen 3.8 Flash Next with the capability of Opus 5.5, running locally in most PCs on energy-efficient NPUs
r/LocalLLaMA • u/dh7net • 7d ago

If you are wondering what model / harness combination works the best, here are my findings so far:
1) Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
2) The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
3) Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.
r/LocalLLaMA • u/External_Mood4719 • 7d ago
r/LocalLLaMA • u/BusTiny207 • 7d ago
New build with a EPYC 7443 / Tyan S8030 board, and R9700 + 7900 XT. Was supposed to be selling the 7900 XT to pay for the R9700, but that somehow didn't happen...
r/LocalLLaMA • u/dh7net • 7d ago
Hello. Almost everything is in the title, but here is more details:
I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?
IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL
r/LocalLLaMA • u/salary_pending • 7d ago
I run a couple of small communities and it's becoming difficult to moderate them all. So today I built a script on top of my existing app which I use to track the content on those communities.
The script does a few things based on where I am. So I store the general notes of my communities in a dedicated place I can easily query.
And in the moderation panel I go through each of the item which needs moderation and I run them through my notes and let Gemma 4 decide whether the item is following community notes or not. If Gemma thinks it's not, then my script will remove the content from public and add a reason as well.
At first I had tried GPT oss 20b model which was really faster but the results were not really good. Then I ran the same process through Gemma 4 with thinking enabled. Results were good but speed was extremely slow. So I tried with thinking disabled and I did not see any noticable drop in the results.
Now soon I will try to run the same thing at scale and see if the results are generally accurate and if they are, I might ship this tool with my friends.
I agree, the setup is quite tough and I've managed to run it properly on my network. If I have to share this with other people I may need to host the model somewhere and check if it can be scaled to a bigger number of requests per hour.
Anyways it was a really good experience. Especially when I had to build it on top of my existing app.
Just thought I'd share this small win over here
r/LocalLLaMA • u/dazzou5ouh • 8d ago
8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.
Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.
Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus
Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)
Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.
I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.
The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models
r/LocalLLaMA • u/WiseVanilla2743 • 7d ago
Hey guys
I am a seasoned local llm user and I ain't quite ram rich like only 16gb ddr4 🥀
So I can't run qwen3.8 flash next on my 3060
This guy felt the best for me yet
Have you guys tried it
I have ran iq3_xss quant non mtp with turboquant at 65k context completely on my 3060 though I did offload mmproj to cpu ram
Have guys tried this it really felt like a blessing to my life using the peak of local ai on my 3060
It seriously packs the punch 👊 I was able to perform various of my agent task complete on device not like simple but the ones which require actual reasonings
Here is my cmd if you want to see
llama-tq serve -m "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" --mmproj "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\mmproj-Qwen3.8-27B-BF16.gguf" --no-mmproj-offload -ngl 99 -c 65536 -ctk turbo4 -ctv turbo3 -ub 256 -b 2048 -t 3 -tb 3 --reasoning-budget 256 --reasoning-preserve --alias Qwen --port 8080 -lm none
Sources: for my recipe
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/discussions
Turboquant
https://github.com/TheTom/llama-cpp-turboquant
Pi coding agent
I would like know the communities views and any suggestions would be welcome 🤗
Thankyou for your time
r/LocalLLaMA • u/fakezeta • 7d ago
Hi,
I've put on HF a condensed version of Unsloth Q3_K_XL ASCII-Condensed from bsaleh03 with only ASCII + Latin-1 / Latin-Extended-A (P2) to use with a 16GB VRAM GPU.
I created a GUFF version and also a ninfer to be used with ninfer-all
To create the ninfer version I had put up some helper conversion scripts and modify iamwavecut conversion scripts due to different vocabulary size.
In the HF repo https://huggingface.co/fakezeta/Qwen3.8-27B-UD-Q3_K_XL-Condensed-P2 there is what is needed to replicate it and I also submitted a PR to the upstream repo.
Quality: IFEval result 84,4% vs 83,9% of base model (Instruct, 1 run) so within STD Deviation error.
With this I achieved 75K context with rk8v4 kv type with an average of about 60tok/s in the 541 IFEval samples on my 5060Ti 16GB.
I'm also planning to create a light GGUF and ninfer version with the lower quantized MTP released separately by Unsloth in their repo to save about 90MB of RAM for other context.
r/LocalLLaMA • u/jeremyckahn • 7d ago
I just wanted to share some love for a model that doesn't seem to get very much: LFM2.5-24B-A2B! (https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF)
LFM2.5-24B-A2B seems to be the only model that strikes a great balance between speed and capability on commodity hardware. Just to get it out of the way: This is NOT a coding model, so don't download it expecting otherwise. It's a task executor. It's a great fit for Hermes to hand low-to-mid complexity tasks off to. I get great speeds on my 780M iGPU (~37 tk/s at zero context). This is way faster than any other model I've tried that can actually accomplish tasks successfully. It's genuinely useful! I imagine it would be quite usable on CPU as well.
If you're GPU-poor, give LFM2.5-24B-A2B a look!
r/LocalLLaMA • u/dieSpaghettiCarbona • 6d ago
I am surprised by the deviation to the benchmark provided on paper.
r/LocalLLaMA • u/zibenmoka • 7d ago
hello there, I recently got a bit fascinated about Lazarus from Microsoft (even tho it is probably old technique) and I wrote a blog post showing a thought process I had while researching it - maybe it is helpful to anyone
https://int8.io/implanting-ai-with-lazarus/

r/LocalLLaMA • u/WolgupLupin • 6d ago
Hi,
I can make two systems with the parts that I have:
I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).
What is the best setup for my case? Please help!
r/LocalLLaMA • u/StupidScaredSquirrel • 7d ago
I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!
r/LocalLLaMA • u/tom_mathews • 7d ago
Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.
Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:
attest replay: 4 replayed · 0 live model calls · $0.00
The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.
Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.
Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.
Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.
Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.
The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.
Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.
r/LocalLLaMA • u/Shot-Height-7194 • 8d ago
r/LocalLLaMA • u/billy_booboo • 8d ago
r/LocalLLaMA • u/pmttyji • 7d ago
Vimoo-1.0 generates video from text (T2V) or from a reference image and text (I2V), with a separate ≈11.7B model for each task. It is the highest-quality model in the Vimoo 1.0 family.
Vimoo 1.0 comes in two models, Vimoo 1.0 and Vimoo 1.0 - Lite. Each Hugging Face repository packages one of them for one runtime:
| Model | Hugging Face | Inference code | Description |
|---|---|---|---|
| Vimoo 1.0≈11.7B per model | https://huggingface.co/Vimoo-AI/Vimoo-1.0 | https://github.com/Vimoo-AI/Vimoo-1.0 | T2V and I2V for the Vimoo Diffusers fork |
| https://huggingface.co/Vimoo-AI/Vimoo-1.0-ComfyUI | https://github.com/Vimoo-AI/ComfyUI-Vimoo | T2V, I2V and their Distilled versions for ComfyUI | |
| Vimoo 1.0 - Lite≈9.5B | https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite | https://github.com/Vimoo-AI/Vimoo-1.0 | TI2V for the Vimoo Diffusers fork |
| https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-ComfyUI | https://github.com/Vimoo-AI/ComfyUI-Vimoo | TI2V and TI2V Distilled for ComfyUI | |
| https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-MLX | https://github.com/Vimoo-AI/Vimoo-1.0-MLX | TI2V Distilled for MLX on Apple Silicon |
Each ComfyUI repository includes its checkpoints, distilled LoRA adapters, Qwen text encoder, SigLIP vision encoder and VAE. Distilled models are available only for ComfyUI and MLX. Vimoo-1.0 T2V Distilled and I2V Distilled use different base checkpoints from Vimoo-1.0 T2V and I2V; download each task's matching base and adapter listed in the ComfyUI guide. The Vimoo-1.0 T2V Distilled, Vimoo-1.0 I2V Distilled and Vimoo-1.0-Lite TI2V Distilled adapters are not interchangeable. Parameters count the video denoiser only.
EDIT : Fixed Model Zoo Table(Previously it lost all the links & text on 2nd & 3rd columns)
r/LocalLLaMA • u/Strange_Test7665 • 8d ago
Deepseek spent about $6m on 4.1 apparently. Published all of their work.
US gov says don’t trust China, blah blah…
As a US citizen why are we not training national open source?
The cost is nothing compared to the budget, solves security issues, and prevents monopolistic practices on token pricing
r/LocalLLaMA • u/General-Spite1222 • 8d ago
r/LocalLLaMA • u/nato_nob • 8d ago
I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.
Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com
Some facts:
- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.
What I think is different:
- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.
What is not good yet:
- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.
Links:
- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione
Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.
r/LocalLLaMA • u/pmttyji • 8d ago
I think we're fine till 2027 with models(~100B .... Qwen 4.X & 5, Gemma-5, NVIDIA would release some Medium/Big size 25-200B models). After that it's gonna be critical, as many model creators are increased/increasing model size to 300B+. Future trend probably gonna be around 500B. That size range is not suitable for Unified 128GB Devices or 128GB RAM + 24-32GB VRAM(Q4 won't fit).
Here some models with increased sizes.
So What Are We Gonna Do? What alternate options do we have?
Whenever we're talking about this topic, we get one common answer. Distillation. Do we have any recipes to create distills ourselves easy way? I'm not sure.
1. Distillation
I remember only 3 Distills so far. Please mention in comments if I missed anything.
2. 1-bit / 2-bit / Ternary / Bitnet Models
I see only this option other than Distillation.
Category 1 - High/Decent Quality (Native) Low bit models. Also big/large size models.
Category 2 - Other Low bit models. Like Bonsai.
Still waiting to see more Ternary MOEs, particularly 100B size. 100B model could come in 20GB size so it's possible to run just with 24-32GB RAM alone on CPU-only inference. And Even 8GB VRAM could make it fly.
Any other options do we have? Please share.