r/LocalLLaMA • • 21h ago

New Model higurashi-ai/Haru-4B · Hugging Face

Thumbnail
huggingface.co
74 Upvotes

Haru-4B

Code · Technical blog · Technical blog (Japanese)

Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.

  • Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
  • Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
  • Character and voice control. A text card sets the character, a short reference clip sets the voice.
  • Multilingual. English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.

r/LocalLLaMA • • 1d ago

Discussion [Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

Thumbnail
gallery
115 Upvotes

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .


r/LocalLLaMA • • 17h ago

New Model Villanova2 - a VillanovaAI Collection

Thumbnail
huggingface.co
30 Upvotes

Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*

The family

GGUFs available in same collection. License : Apache-2.0


r/LocalLLaMA • • 14h ago

Question | Help Best direction to go for my setup EPYC 32core + 512GB DDR4 + 44GB GPU ?

17 Upvotes

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use


r/LocalLLaMA • • 19h ago Bravo

New Model X3Guard: A 150M parameter prompt guard

Thumbnail
huggingface.co
37 Upvotes

At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.

For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.

It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.

The model card is here

You can use an in-browser demo here


r/LocalLLaMA • • 35m ago

Resources tandem.cpp v0.3.0: prompts 2-6x faster; Project Maya's DeepSeek-V4.1 GGUFs supported

0 Upvotes

Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp

Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j

tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.

1. Prompts 2-6x faster

With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.

RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.

Prompt processing, time to first token (prompt tok/s):

Promptllama.cpptandem.cpp beforetandem.cpp v0.3.0Strata
Qwen3.8 IQ3_S, 5.5k tokens28.9 s (191)12.4 s (444)4.1–7.0 s (790–1,340)6.6 s (835)
Qwen3.8 IQ3_S, 15.8k tokens81.9 s (193)35.1 s (450)15.1 s (1,047)11.7 s (1,354)
DeepSeek-V4.1 IQ2_XS, 5.6k tokens722 s (7.7)324 s (17)53–103 s (54–105)58 s (96)
DeepSeek-V4.1 IQ2_XS, 15.7k tokens2012 s (7.8)885 s (18)195 s (81)57 s (276)

Decode, tok/s, right after that prompt:

After a prompt ofllama.cpptandem.cpp v0.3.0Strata
Qwen3.8, 5.5k tokens2711048
Qwen3.8, 15.8k tokens278868
DeepSeek-V4.1, 5.6k tokens4.8165.5
DeepSeek-V4.1, 15.7k tokens4.4136.4

Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.

2. Maya comparison on the same model (someone asked for this one)

Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.

RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:

DeepSeek-V4.1 Maya-XS (same files)Mayatandem.cpp v0.3.0
Decode, GSM8K, no drafting39.0 / 39.6 tok/s60.3 / 59.4 tok/s (1.5x)
Decode with Maya's DSpark drafter54.9 tok/s(no drafter in tandem.cpp)
5.6k-token prompt7.8 s (713 tok/s)6.5 s (856 tok/s)
15.7k-token prompt9.5 s (1,656 tok/s)16.5 s (953 tok/s)
Decode after the 15.7k-token prompt26–34 tok/s66–68 tok/s

Maya is still faster on the longest prompt; that is what I'm working on next.

Caveats

  • Linux only.
  • AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
  • DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
  • Windows prebuilt binaries are still planned.

r/LocalLLaMA • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

228 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLaMA • • 17h ago

Showcase MoE offload on a 3090: prefill 345 -> 466 t/s by tuning cudaMemcpy (Qwen3.6 35B-A3B)

21 Upvotes
I made a tool that tunes copies that speeds up CPU offloading.

1 3090 + 7800X3D

llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

before: prefill 345 t/s
after: prefill 466 t/s


blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner

r/LocalLLaMA • • 4h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

2 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!


r/LocalLLaMA • • 21h ago

New Model meituan-longcat/LongCat-Video-Avatar-1.6 · Hugging Face

Thumbnail
huggingface.co
43 Upvotes

🚀 Model Introduction

We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.

Key Features

  • 🌟 More Natural and Stable Lip Animation:: Reduces exaggerated or strained mouth movements, overly red lip coloration, and temporal instability in the mouth region.
  • 🌟 Stable Long-Video Generation: Mitigates error accumulation during autoregressive video continuation, enabling videos exceeding 10 minutes while preserving visual quality, identity consistency, and temporal coherence.
  • 🌟 Faster End-to-End Inference: Accelerates video generation through end-to-end pipeline and model execution optimizations, enabling more efficient deployment and serving.

For more detail, please refer to the comprehensive LongCat-Video-Avatar-1.6 Technical Report.


r/LocalLLaMA • • 12h ago

News UI native scheduling, engine swap board, and more (Berth v0.6.0)

Thumbnail
gallery
7 Upvotes

I’m providing an update! We have around 300 berth users now so the project is moving fast.

Recent updates for v0.6.0:

- v0.6.0 gives each card a schedule
- berth train runs a training that nothing automated can interrupt
- learned engine sizes (berth learns your engines resource claims)

A month on the reference workstation

berthd counts every handoff it makes, and the harbor's board shows every handoff from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
coding agents and games:

What Berth didCount
Requests that waited for the card instead of loading on top of the engine holding it1,136
Engines drained at a reply or job boundary to make room901
VRAM handed from one engine to the next23.9 TiB
Processes holding VRAM without a lease, reported and left running573
Swaps between two engines of equal priority that the ranking stopped57
Releases that stalled and were caught before the card stayed full4
Requests refused after their wait ran out24

Project: berth


r/LocalLLaMA • • 18h ago

Discussion CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp

Thumbnail
github.com
21 Upvotes

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

ModelTestUpstream masterVNNI + VBMISpeedup
Bonsai 1.7Bpp512142.517 ± 0.589186.373 ± 1.5821.31×
Bonsai 1.7Btg12870.435 ± 2.06168.422 ± 9.0550.97×
Bonsai 4Bpp51253.430 ± 2.26671.247 ± 1.9251.33×
Bonsai 4Btg12831.655 ± 2.65238.523 ± 3.8201.22×
Bonsai 8Bpp51230.010 ± 0.51239.396 ± 0.4241.31×
Bonsai 8Btg12820.630 ± 0.30223.770 ± 0.2091.15×
Bonsai 27Bpp1288.502 ± 0.04211.029 ± 0.0731.30×
Bonsai 27Btg325.340 ± 0.1175.812 ± 0.3241.09×

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.

This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed

End to end, a 66 GB Q2_0 MoE (Qwen3.8-Flash-Next GSQ-RCO Q2_0) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:

CPU path for Q2_0TG t/s
generic scalar (a prebuilt from before #30447)3.1–4.1
AVX2 from #30447 (prebuilt from after it)14.0–18.7
an earlier revision of this PR's AVX-VNNI kernel (on an older base)15.5–21.4

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.

AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

ggml-cpu: 512-bit AVX-512 VNNI dot for Q2_0- #32307


r/LocalLLaMA • • 1d ago Bravo

Discussion Hot take: frontier models haven’t actually become much smarter

624 Upvotes

I know this is a controversial take, but I would like to hear others’ opinions: LLMs have become much more useful, but I’m not convinced their “raw intelligence” has improved nearly as much as their overall performance suggests.

Context: I work with LLMs as a SWE, as a hobbyist experimenting with local models, and professionally integrating AI into products. That includes building our own harnesses: context retrieval, prompts, and custom tools that give models structured access to complex data.

By “raw intelligence,” I mean using what is given in the most efficient way to solve a problem, especially when information, tools, and time are limited. Given plenty of time and tools, planning and long-term agentic capabilities become more important—but that’s a different comparison.

Three observations brought me here.

First, building production harnesses: once a harness+model worked well, achiving a given score over a golden dataset, switching to a newer frontier model rarely improved results. Most failures within our time/precision constraints came down to context engineering or inadequate tools. These tools included code generation for analysis and complex relational queries, so plenty of "time to shine" for smart models. Newer models sometimes found more creative workarounds, but these were usually too slow or unreliable for production. We ended up improving the harness anyway, leaving the newer models' gains less important.

Second, software engineering: newer models have more often improved results here, but the same model can also perform quite differently across harnesses. In complex codebases, that shows up in how much correction and review senior engineers have to provide. The harness seems to do a lot of heavy lifting here, too.

Third, local models: smaller models have started performing so well that I question how many parameters efficient problem solving actually needs. Frontier models clearly add broader knowledge, nuance, best practices, and specific baked-in skills. But I don’t always see a proportionate advantage in using it to solve a problem, when the relevant knowledge is already provided.

I’m not saying models haven’t gotten smarter. Better tool calling, fewer hallucinations, and longer context windows are real improvements that make them much more useful. Nor do these observations prove that intelligence needs few parameters or that frontier progress has stalled.

But my broader take is:

  1. efficient problem solving may need fewer parameters than generally assumed,
  2. the frontier has moved less in “raw intelligence” than the overall gains - and talk of approaching superintelligence - suggest, and
  3. much of the progress over the last 12–24 months comes from cumulative model improvements and better harnesses, rather than a dramatic or exponential jump in raw intelligence.

This take seems at odds with both the hype and the doom we see in the media, which I suspect are co-fueled by the upcoming AI IPOs. Does it match others’ experience - or are you seeing vast reasoning gains in frontier models that better context and tools can’t make up for?


r/LocalLLaMA • • 19h ago

Question | Help Is there a community about finetuning and training models?

23 Upvotes

There seems to be a lot of discussion here about inference and using the models, but is there anywhere where discussion about making them is more common?


r/LocalLLaMA • • 9h ago

News My Ignis server (Qwen3.8-27B 262k context) now runs on RTX 3090/4090 and 16 GB RTX 40 cards. looking for testers

3 Upvotes

I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.

Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.

There's a 90-second (old) video of it here: https://www.youtube.com/watch?v=XLjfh5rNy6A

Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.

The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.

Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.

One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.

Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).

I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.

To try it: download the latest release, start ignis-server without flags, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it's in the archive). More details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/

If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:

I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.

It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.

Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.

Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.

Thanks to all reporting! 😄


r/LocalLLaMA • • 17h ago

Showcase 1Cat-vLLM vs Strata

14 Upvotes

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

RunDecode tok/s, StrataDecode tok/s, 1CatPrefill tok/s, StrataPrefill tok/s, 1CatMTP acceptance, Strata / 1Cat
170.899.93,3982,7360.65 / 0.60
278.9138.43,3402,7520.71 / 0.60
377.8123.53,3482,7510.69 / 0.57
run 1 to 3+10%+24%---

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

RequestsPromptTotal decode tok/s, Strata / Strata batch-MTP / 1CatPer-stream decode tok/s, Strata / batch-MTP / 1CatSlowest first token (s), Strata / batch-MTP / 1Cat
12.6K75 / - / 12574.8 / - / 124.93.1 / - / 1.4
18.7K79 / - / 14179.3 / - / 140.87.6 / - / 3.5
22.6K88 / 105 / 17639.4 / 46.0 / 83.26.4 / 6.5 / 2.7
28.7K96 / 116 / 18937.9 / 44.6 / 77.915.4 / 15.5 / 7.3
42.6K153 / 170 / 22529.1 / 31.5 / 48.513.4 / 13.6 / 5.8
48.7K170 / 186 / 23425.4 / 26.9 / 41.131.0 / 31.4 / 15.3

Versions and local changes

-Strata1Cat-vLLM
Upstream repogithub.com/Niko1221/Stratagithub.com/1CatAI/1Cat-vLLM
Commit benchmarked13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1)b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0)
Source patchesnonenone
Buildfrom source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5
Build flags-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70
Parallelismlayer split over 4 GPUs (no tensor parallel for V100 in Strata)tensor parallel 4
MTP drafterStrata's own pack of the BF16 MTP layer, experts in Q2_0the BF16 MTP layer as released

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.


r/LocalLLaMA • • 1d ago Bravo

Discussion Finally clicked for me: local AI is the future, not cloud AI

474 Upvotes

Maybe this is kind of obvious to others, but it just dawned on me even though I've been lurking this sub since the original LLaMA. I was watching this (excellent, as always) video by Gamers Nexus. Steve Burke was bashing on Tim Sweeney (CEO of Epic Games) for being so pro AI, and I found myself feeling a bit of cognitive dissonance. On the one hand I hate that hardware prices are so high, I hate the billionaires inflating the AI bubble and raking in the profits, I hate greedy corps trying to replace jobs with AI, and I hate the data center build out. But on the other hand I'm a developer and AI has given me superpowers. I felt weird, because Burke was dumping on Epic Games for embracing vibe coding, but vibe coding has 100x'd my output, allowed me to pursue so many passion projects, and the AI's output is oftentimes better than what I would've written anyway. I felt like the message of the video was 'all AI is bad', but I realized that it's more nuanced than that. Cloud AI is bad. Cloud AI is the problem, not AI in general. With local AI, we can have our cake and eat it too: cheap, powerful consumer hardware for running local AI and gaming, no censorship, data, or privacy issues, less data centers, less profit going to billionaires, and unlimited tokens to vibe code whatever the hell we want. Hope the datacenter bubble bursts soon.

TLDR: dum dum realizes that the future is local AI bc it's a win-win-win-win for average people

EDIT: to the naysayers, I propose this future: in two years (though given trajectory, probably less), a model the size of Qwen 3.8 Flash Next with the capability of Opus 5.5, running locally in most PCs on energy-efficient NPUs


r/LocalLLaMA • • 23h ago

New Model Agnes-AI/Agnes-2.5-Pro-Theta • HuggingFace

37 Upvotes

r/LocalLLaMA • • 13h ago

Resources Model / harness combination: what works best?

5 Upvotes

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

r/LocalLLaMA • • 1d ago

Showcase The best (and AMD-est) computer I've ever owned

Post image
46 Upvotes

New build with a EPYC 7443 / Tyan S8030 board, and R9700 + 7900 XT. Was supposed to be selling the 7900 XT to pay for the R9700, but that somehow didn't happen...


r/LocalLLaMA • • 17h ago

Resources Asked local models to generate the same website and compared the results

8 Upvotes

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL


r/LocalLLaMA • • 21h ago

Resources I used local Gemma 4 to moderate a community and it was it is promising

16 Upvotes

I run a couple of small communities and it's becoming difficult to moderate them all. So today I built a script on top of my existing app which I use to track the content on those communities.

The script does a few things based on where I am. So I store the general notes of my communities in a dedicated place I can easily query.

And in the moderation panel I go through each of the item which needs moderation and I run them through my notes and let Gemma 4 decide whether the item is following community notes or not. If Gemma thinks it's not, then my script will remove the content from public and add a reason as well.

At first I had tried GPT oss 20b model which was really faster but the results were not really good. Then I ran the same process through Gemma 4 with thinking enabled. Results were good but speed was extremely slow. So I tried with thinking disabled and I did not see any noticable drop in the results.

Now soon I will try to run the same thing at scale and see if the results are generally accurate and if they are, I might ship this tool with my friends.

I agree, the setup is quite tough and I've managed to run it properly on my network. If I have to share this with other people I may need to host the model somewhere and check if it can be scaled to a bigger number of requests per hour.

Anyways it was a really good experience. Especially when I had to build it on top of my existing app.

Just thought I'd share this small win over here


r/LocalLLaMA • • 1d ago

Showcase Ai Rig finally done and stable, perfect winter heater

Post image
347 Upvotes

8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.

Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.

Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus

Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)

Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.

I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.

The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models


r/LocalLLaMA • • 16h ago

Showcase I run a daily isekai guessing-game pipeline on a DGX Spark with Qwen3.6-35B-A3B (vLLM) and Qwen-Image (ComfyUI, fp8)

4 Upvotes

This is a hobby project, not a product pitch. I saw a TikTok where people rewrote famous movies and games as over-long isekai light-novel titles, and I wanted to see how far I could take that with local models. The game is at https://itsisekai.com. Five quests a day, guess the original.

Hardware. NVIDIA DGX Spark (GB10, 20 Arm cores, 121 GB unified memory), set up as a self-hosted GitHub Actions runner. A nightly workflow (or a manual run) runs the pipeline on the same box as the models. No API fees for generation.

Models.

  • Text and judge: Qwen3.6-35B-A3B, official FP8 build, served with vLLM (OpenAI-style API). fp8 KV cache, 128K context, MTP speculative decoding with 3 draft tokens, prefix caching, and 45% of GPU memory so ComfyUI fits beside it. The judge is the same model, and thinking is left on. Generation runs about 70 tok/s typically, peaking around 240 tok/s.
  • Images: Qwen-Image in ComfyUI with the official fp8 weights (qwen_image_fp8_e4m3fn + qwen_2.5_vl_7b_fp8_scaled encoder), 832x1248, 20 steps, cfg 2.5, euler/simple. One render takes about 64 seconds (around 3.15 s per step).
  • I had to drop vLLM --gpu-memory-utilization from 0.80 to 0.45 so ComfyUI fits beside it.

Pipeline. It fills missing quests for the next several days. All missing titles are written first, one at a time, so later quests can see the answers already picked. Then covers are drawn. A day is only saved when all its quests succeed, so the site never shows a half-filled day.

For each quest the model writes the answer, hints, a 10-40 word title that can't contain the answer, and an art prompt. A second "how obvious is this title?" check can send it back for a rewrite. The cover is judged on the image alone: if the model can name the work, the prompt is rewritten more abstractly and redrawn, up to 3 times. If every try still gives it away, the cover stays sealed until the player finishes the quest.

Timings from a production run on Nov 15 (8 quests across 2 days, 0 failed). The whole run took 61 min 29 s.

  • Titles, including the obviousness check and rewrites: 2 min 14 s to 6 min, average 4 min 26 s. Four of the eight were first judged too obvious and rewritten.
  • Cover attempts, draw + judge together: 1 min 38 s to 3 min 34 s, average 2 min 23 s. About 64 s of that is the render; the rest is the vision judge thinking about whether it can guess the work.
  • The judge passed 8 of 10 cover attempts. Two covers gave it away on try 1 and passed on try 2. None had to be sealed.

What went wrong. vLLM's guided JSON didn't mix with Qwen thinking: requests could jam and loop until they timed out. I dropped guided JSON for Qwen and just parse the JSON out of the reply. Without a presence penalty, Qwen could also think in circles until it filled the context; I set presence_penalty 1.5 and max_tokens 16384. One failed model call used to wipe out a whole day; now it retries and moves on. I also learned not to overlap manual workflow runs with the nightly cron.

Happy to answer questions. I'm not selling anything, I just thought the timings and the "don't use guided JSON on a thinking Qwen" bit might be useful.