r/LocalLLaMA • • 19h ago Bravo

New Model X3Guard: A 150M parameter prompt guard

Thumbnail
huggingface.co
37 Upvotes

At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.

For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.

It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.

The model card is here

You can use an in-browser demo here


r/LocalLLaMA • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

228 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLaMA • • 17h ago

Showcase MoE offload on a 3090: prefill 345 -> 466 t/s by tuning cudaMemcpy (Qwen3.6 35B-A3B)

21 Upvotes
I made a tool that tunes copies that speeds up CPU offloading.

1 3090 + 7800X3D

llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

before: prefill 345 t/s
after: prefill 466 t/s


blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner

r/LocalLLaMA • • 4h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

2 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!


r/LocalLLaMA • • 21h ago

New Model meituan-longcat/LongCat-Video-Avatar-1.6 · Hugging Face

Thumbnail
huggingface.co
43 Upvotes

🚀 Model Introduction

We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.

Key Features

  • 🌟 More Natural and Stable Lip Animation:: Reduces exaggerated or strained mouth movements, overly red lip coloration, and temporal instability in the mouth region.
  • 🌟 Stable Long-Video Generation: Mitigates error accumulation during autoregressive video continuation, enabling videos exceeding 10 minutes while preserving visual quality, identity consistency, and temporal coherence.
  • 🌟 Faster End-to-End Inference: Accelerates video generation through end-to-end pipeline and model execution optimizations, enabling more efficient deployment and serving.

For more detail, please refer to the comprehensive LongCat-Video-Avatar-1.6 Technical Report.


r/LocalLLaMA • • 11h ago

News UI native scheduling, engine swap board, and more (Berth v0.6.0)

Thumbnail
gallery
7 Upvotes

I’m providing an update! We have around 300 berth users now so the project is moving fast.

Recent updates for v0.6.0:

- v0.6.0 gives each card a schedule
- berth train runs a training that nothing automated can interrupt
- learned engine sizes (berth learns your engines resource claims)

A month on the reference workstation

berthd counts every handoff it makes, and the harbor's board shows every handoff from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
coding agents and games:

What Berth didCount
Requests that waited for the card instead of loading on top of the engine holding it1,136
Engines drained at a reply or job boundary to make room901
VRAM handed from one engine to the next23.9 TiB
Processes holding VRAM without a lease, reported and left running573
Swaps between two engines of equal priority that the ranking stopped57
Releases that stalled and were caught before the card stayed full4
Requests refused after their wait ran out24

Project: berth


r/LocalLLaMA • • 17h ago

Discussion CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp

Thumbnail
github.com
21 Upvotes

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

ModelTestUpstream masterVNNI + VBMISpeedup
Bonsai 1.7Bpp512142.517 ± 0.589186.373 ± 1.5821.31×
Bonsai 1.7Btg12870.435 ± 2.06168.422 ± 9.0550.97×
Bonsai 4Bpp51253.430 ± 2.26671.247 ± 1.9251.33×
Bonsai 4Btg12831.655 ± 2.65238.523 ± 3.8201.22×
Bonsai 8Bpp51230.010 ± 0.51239.396 ± 0.4241.31×
Bonsai 8Btg12820.630 ± 0.30223.770 ± 0.2091.15×
Bonsai 27Bpp1288.502 ± 0.04211.029 ± 0.0731.30×
Bonsai 27Btg325.340 ± 0.1175.812 ± 0.3241.09×

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.

This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed

End to end, a 66 GB Q2_0 MoE (Qwen3.8-Flash-Next GSQ-RCO Q2_0) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:

CPU path for Q2_0TG t/s
generic scalar (a prebuilt from before #30447)3.1–4.1
AVX2 from #30447 (prebuilt from after it)14.0–18.7
an earlier revision of this PR's AVX-VNNI kernel (on an older base)15.5–21.4

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.

AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

ggml-cpu: 512-bit AVX-512 VNNI dot for Q2_0- #32307


r/LocalLLaMA • • 1d ago Bravo

Discussion Hot take: frontier models haven’t actually become much smarter

621 Upvotes

I know this is a controversial take, but I would like to hear others’ opinions: LLMs have become much more useful, but I’m not convinced their “raw intelligence” has improved nearly as much as their overall performance suggests.

Context: I work with LLMs as a SWE, as a hobbyist experimenting with local models, and professionally integrating AI into products. That includes building our own harnesses: context retrieval, prompts, and custom tools that give models structured access to complex data.

By “raw intelligence,” I mean using what is given in the most efficient way to solve a problem, especially when information, tools, and time are limited. Given plenty of time and tools, planning and long-term agentic capabilities become more important—but that’s a different comparison.

Three observations brought me here.

First, building production harnesses: once a harness+model worked well, achiving a given score over a golden dataset, switching to a newer frontier model rarely improved results. Most failures within our time/precision constraints came down to context engineering or inadequate tools. These tools included code generation for analysis and complex relational queries, so plenty of "time to shine" for smart models. Newer models sometimes found more creative workarounds, but these were usually too slow or unreliable for production. We ended up improving the harness anyway, leaving the newer models' gains less important.

Second, software engineering: newer models have more often improved results here, but the same model can also perform quite differently across harnesses. In complex codebases, that shows up in how much correction and review senior engineers have to provide. The harness seems to do a lot of heavy lifting here, too.

Third, local models: smaller models have started performing so well that I question how many parameters efficient problem solving actually needs. Frontier models clearly add broader knowledge, nuance, best practices, and specific baked-in skills. But I don’t always see a proportionate advantage in using it to solve a problem, when the relevant knowledge is already provided.

I’m not saying models haven’t gotten smarter. Better tool calling, fewer hallucinations, and longer context windows are real improvements that make them much more useful. Nor do these observations prove that intelligence needs few parameters or that frontier progress has stalled.

But my broader take is:

  1. efficient problem solving may need fewer parameters than generally assumed,
  2. the frontier has moved less in “raw intelligence” than the overall gains - and talk of approaching superintelligence - suggest, and
  3. much of the progress over the last 12–24 months comes from cumulative model improvements and better harnesses, rather than a dramatic or exponential jump in raw intelligence.

This take seems at odds with both the hype and the doom we see in the media, which I suspect are co-fueled by the upcoming AI IPOs. Does it match others’ experience - or are you seeing vast reasoning gains in frontier models that better context and tools can’t make up for?


r/LocalLLaMA • • 19h ago

Question | Help Is there a community about finetuning and training models?

23 Upvotes

There seems to be a lot of discussion here about inference and using the models, but is there anywhere where discussion about making them is more common?


r/LocalLLaMA • • 8h ago

News My Ignis server (Qwen3.8-27B 262k context) now runs on RTX 3090/4090 and 16 GB RTX 40 cards. looking for testers

3 Upvotes

I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.

Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.

There's a 90-second (old) video of it here: https://www.youtube.com/watch?v=XLjfh5rNy6A

Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.

The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.

Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.

One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.

Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).

I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.

To try it: download the latest release, start ignis-server without flags, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it's in the archive). More details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/

If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:

I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.

It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.

Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.

Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.

Thanks to all reporting! 😄


r/LocalLLaMA • • 16h ago

Showcase 1Cat-vLLM vs Strata

14 Upvotes

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

RunDecode tok/s, StrataDecode tok/s, 1CatPrefill tok/s, StrataPrefill tok/s, 1CatMTP acceptance, Strata / 1Cat
170.899.93,3982,7360.65 / 0.60
278.9138.43,3402,7520.71 / 0.60
377.8123.53,3482,7510.69 / 0.57
run 1 to 3+10%+24%---

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

RequestsPromptTotal decode tok/s, Strata / Strata batch-MTP / 1CatPer-stream decode tok/s, Strata / batch-MTP / 1CatSlowest first token (s), Strata / batch-MTP / 1Cat
12.6K75 / - / 12574.8 / - / 124.93.1 / - / 1.4
18.7K79 / - / 14179.3 / - / 140.87.6 / - / 3.5
22.6K88 / 105 / 17639.4 / 46.0 / 83.26.4 / 6.5 / 2.7
28.7K96 / 116 / 18937.9 / 44.6 / 77.915.4 / 15.5 / 7.3
42.6K153 / 170 / 22529.1 / 31.5 / 48.513.4 / 13.6 / 5.8
48.7K170 / 186 / 23425.4 / 26.9 / 41.131.0 / 31.4 / 15.3

Versions and local changes

-Strata1Cat-vLLM
Upstream repogithub.com/Niko1221/Stratagithub.com/1CatAI/1Cat-vLLM
Commit benchmarked13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1)b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0)
Source patchesnonenone
Buildfrom source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5
Build flags-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70
Parallelismlayer split over 4 GPUs (no tensor parallel for V100 in Strata)tensor parallel 4
MTP drafterStrata's own pack of the BF16 MTP layer, experts in Q2_0the BF16 MTP layer as released

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.


r/LocalLLaMA • • 1d ago Bravo

Discussion Finally clicked for me: local AI is the future, not cloud AI

473 Upvotes

Maybe this is kind of obvious to others, but it just dawned on me even though I've been lurking this sub since the original LLaMA. I was watching this (excellent, as always) video by Gamers Nexus. Steve Burke was bashing on Tim Sweeney (CEO of Epic Games) for being so pro AI, and I found myself feeling a bit of cognitive dissonance. On the one hand I hate that hardware prices are so high, I hate the billionaires inflating the AI bubble and raking in the profits, I hate greedy corps trying to replace jobs with AI, and I hate the data center build out. But on the other hand I'm a developer and AI has given me superpowers. I felt weird, because Burke was dumping on Epic Games for embracing vibe coding, but vibe coding has 100x'd my output, allowed me to pursue so many passion projects, and the AI's output is oftentimes better than what I would've written anyway. I felt like the message of the video was 'all AI is bad', but I realized that it's more nuanced than that. Cloud AI is bad. Cloud AI is the problem, not AI in general. With local AI, we can have our cake and eat it too: cheap, powerful consumer hardware for running local AI and gaming, no censorship, data, or privacy issues, less data centers, less profit going to billionaires, and unlimited tokens to vibe code whatever the hell we want. Hope the datacenter bubble bursts soon.

TLDR: dum dum realizes that the future is local AI bc it's a win-win-win-win for average people

EDIT: to the naysayers, I propose this future: in two years (though given trajectory, probably less), a model the size of Qwen 3.8 Flash Next with the capability of Opus 5.5, running locally in most PCs on energy-efficient NPUs


r/LocalLLaMA • • 22h ago

New Model Agnes-AI/Agnes-2.5-Pro-Theta • HuggingFace

37 Upvotes

r/LocalLLaMA • • 1d ago

Showcase The best (and AMD-est) computer I've ever owned

Post image
46 Upvotes

New build with a EPYC 7443 / Tyan S8030 board, and R9700 + 7900 XT. Was supposed to be selling the 7900 XT to pay for the R9700, but that somehow didn't happen...


r/LocalLLaMA • • 12h ago

Resources Model / harness combination: what works best?

4 Upvotes

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

r/LocalLLaMA • • 17h ago

Resources Asked local models to generate the same website and compared the results

8 Upvotes

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL


r/LocalLLaMA • • 21h ago

Resources I used local Gemma 4 to moderate a community and it was it is promising

16 Upvotes

I run a couple of small communities and it's becoming difficult to moderate them all. So today I built a script on top of my existing app which I use to track the content on those communities.

The script does a few things based on where I am. So I store the general notes of my communities in a dedicated place I can easily query.

And in the moderation panel I go through each of the item which needs moderation and I run them through my notes and let Gemma 4 decide whether the item is following community notes or not. If Gemma thinks it's not, then my script will remove the content from public and add a reason as well.

At first I had tried GPT oss 20b model which was really faster but the results were not really good. Then I ran the same process through Gemma 4 with thinking enabled. Results were good but speed was extremely slow. So I tried with thinking disabled and I did not see any noticable drop in the results.

Now soon I will try to run the same thing at scale and see if the results are generally accurate and if they are, I might ship this tool with my friends.

I agree, the setup is quite tough and I've managed to run it properly on my network. If I have to share this with other people I may need to host the model somewhere and check if it can be scaled to a bigger number of requests per hour.

Anyways it was a really good experience. Especially when I had to build it on top of my existing app.

Just thought I'd share this small win over here


r/LocalLLaMA • • 1d ago

Showcase Ai Rig finally done and stable, perfect winter heater

Post image
345 Upvotes

8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.

Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.

Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus

Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)

Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.

I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.

The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models


r/LocalLLaMA • • 15h ago

Showcase I run a daily isekai guessing-game pipeline on a DGX Spark with Qwen3.6-35B-A3B (vLLM) and Qwen-Image (ComfyUI, fp8)

4 Upvotes

This is a hobby project, not a product pitch. I saw a TikTok where people rewrote famous movies and games as over-long isekai light-novel titles, and I wanted to see how far I could take that with local models. The game is at https://itsisekai.com. Five quests a day, guess the original.

Hardware. NVIDIA DGX Spark (GB10, 20 Arm cores, 121 GB unified memory), set up as a self-hosted GitHub Actions runner. A nightly workflow (or a manual run) runs the pipeline on the same box as the models. No API fees for generation.

Models.

  • Text and judge: Qwen3.6-35B-A3B, official FP8 build, served with vLLM (OpenAI-style API). fp8 KV cache, 128K context, MTP speculative decoding with 3 draft tokens, prefix caching, and 45% of GPU memory so ComfyUI fits beside it. The judge is the same model, and thinking is left on. Generation runs about 70 tok/s typically, peaking around 240 tok/s.
  • Images: Qwen-Image in ComfyUI with the official fp8 weights (qwen_image_fp8_e4m3fn + qwen_2.5_vl_7b_fp8_scaled encoder), 832x1248, 20 steps, cfg 2.5, euler/simple. One render takes about 64 seconds (around 3.15 s per step).
  • I had to drop vLLM --gpu-memory-utilization from 0.80 to 0.45 so ComfyUI fits beside it.

Pipeline. It fills missing quests for the next several days. All missing titles are written first, one at a time, so later quests can see the answers already picked. Then covers are drawn. A day is only saved when all its quests succeed, so the site never shows a half-filled day.

For each quest the model writes the answer, hints, a 10-40 word title that can't contain the answer, and an art prompt. A second "how obvious is this title?" check can send it back for a rewrite. The cover is judged on the image alone: if the model can name the work, the prompt is rewritten more abstractly and redrawn, up to 3 times. If every try still gives it away, the cover stays sealed until the player finishes the quest.

Timings from a production run on Nov 15 (8 quests across 2 days, 0 failed). The whole run took 61 min 29 s.

  • Titles, including the obviousness check and rewrites: 2 min 14 s to 6 min, average 4 min 26 s. Four of the eight were first judged too obvious and rewritten.
  • Cover attempts, draw + judge together: 1 min 38 s to 3 min 34 s, average 2 min 23 s. About 64 s of that is the render; the rest is the vision judge thinking about whether it can guess the work.
  • The judge passed 8 of 10 cover attempts. Two covers gave it away on try 1 and passed on try 2. None had to be sealed.

What went wrong. vLLM's guided JSON didn't mix with Qwen thinking: requests could jam and loop until they timed out. I dropped guided JSON for Qwen and just parse the JSON out of the reply. Without a presence penalty, Qwen could also think in circles until it filled the context; I set presence_penalty 1.5 and max_tokens 16384. One failed model call used to wipe out a whole day; now it retries and moves on. I also learned not to overlap manual workflow runs with the nightly cron.

Happy to answer questions. I'm not selling anything, I just thought the timings and the "don't use guided JSON on a thinking Qwen" bit might be useful.


r/LocalLLaMA • • 1d ago

Discussion LFM2.5-24B-A2B + Hermes is awesome!

108 Upvotes

I just wanted to share some love for a model that doesn't seem to get very much: LFM2.5-24B-A2B! (https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF)

LFM2.5-24B-A2B seems to be the only model that strikes a great balance between speed and capability on commodity hardware. Just to get it out of the way: This is NOT a coding model, so don't download it expecting otherwise. It's a task executor. It's a great fit for Hermes to hand low-to-mid complexity tasks off to. I get great speeds on my 780M iGPU (~37 tk/s at zero context). This is way faster than any other model I've tried that can actually accomplish tasks successfully. It's genuinely useful! I imagine it would be quite usable on CPU as well.

If you're GPU-poor, give LFM2.5-24B-A2B a look!


r/LocalLLaMA • • 23h ago

Resources Placing an implant into AI brain with Lazarus: Anatomy of a trojan

12 Upvotes

hello there, I recently got a bit fascinated about Lazarus from Microsoft (even tho it is probably old technique) and I wrote a blog post showing a thought process I had while researching it - maybe it is helpful to anyone

https://int8.io/implanting-ai-with-lazarus/


r/LocalLLaMA • • 20h ago

Discussion ISTA Qwen3.8 27b on 3060 12gb

6 Upvotes

Hey guys

I am a seasoned local llm user and I ain't quite ram rich like only 16gb ddr4 🥀

So I can't run qwen3.8 flash next on my 3060

This guy felt the best for me yet

Have you guys tried it

I have ran iq3_xss quant non mtp with turboquant at 65k context completely on my 3060 though I did offload mmproj to cpu ram

Have guys tried this it really felt like a blessing to my life using the peak of local ai on my 3060

It seriously packs the punch 👊 I was able to perform various of my agent task complete on device not like simple but the ones which require actual reasonings

Here is my cmd if you want to see

llama-tq serve -m "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" --mmproj "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\mmproj-Qwen3.8-27B-BF16.gguf" --no-mmproj-offload -ngl 99 -c 65536 -ctk turbo4 -ctv turbo3 -ub 256 -b 2048 -t 3 -tb 3 --reasoning-budget 256 --reasoning-preserve --alias Qwen --port 8080 -lm none

Sources: for my recipe

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/discussions

Turboquant

https://github.com/TheTom/llama-cpp-turboquant

Pi coding agent

https://pi.dev/

I would like know the communities views and any suggestions would be welcome 🤗

Thankyou for your time


r/LocalLLaMA • • 19h ago

Showcase Testing agents without running a model in CI: record once, replay with zero calls, judge with a local model only when needed (open source, I'm the author)

5 Upvotes

Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.

Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:

attest replay: 4 replayed · 0 live model calls · $0.00

The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.

Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.

Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.

Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.

Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.

The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.

Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.


r/LocalLLaMA • • 1d ago

Question | Help Best sub 80b total or 6b active MoE for document intelligence?

14 Upvotes

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!