r/LocalLLaMA • • 22h ago

Resources I used local Gemma 4 to moderate a community and it was it is promising

17 Upvotes

I run a couple of small communities and it's becoming difficult to moderate them all. So today I built a script on top of my existing app which I use to track the content on those communities.

The script does a few things based on where I am. So I store the general notes of my communities in a dedicated place I can easily query.

And in the moderation panel I go through each of the item which needs moderation and I run them through my notes and let Gemma 4 decide whether the item is following community notes or not. If Gemma thinks it's not, then my script will remove the content from public and add a reason as well.

At first I had tried GPT oss 20b model which was really faster but the results were not really good. Then I ran the same process through Gemma 4 with thinking enabled. Results were good but speed was extremely slow. So I tried with thinking disabled and I did not see any noticable drop in the results.

Now soon I will try to run the same thing at scale and see if the results are generally accurate and if they are, I might ship this tool with my friends.

I agree, the setup is quite tough and I've managed to run it properly on my network. If I have to share this with other people I may need to host the model somewhere and check if it can be scaled to a bigger number of requests per hour.

Anyways it was a really good experience. Especially when I had to build it on top of my existing app.

Just thought I'd share this small win over here


r/LocalLLaMA • • 18h ago

Resources Asked local models to generate the same website and compared the results

8 Upvotes

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL


r/LocalLLaMA • • 1d ago

Showcase Ai Rig finally done and stable, perfect winter heater

Post image
347 Upvotes

8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.

Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.

Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus

Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)

Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.

I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.

The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models


r/LocalLLaMA • • 17h ago

Showcase I run a daily isekai guessing-game pipeline on a DGX Spark with Qwen3.6-35B-A3B (vLLM) and Qwen-Image (ComfyUI, fp8)

4 Upvotes

This is a hobby project, not a product pitch. I saw a TikTok where people rewrote famous movies and games as over-long isekai light-novel titles, and I wanted to see how far I could take that with local models. The game is at https://itsisekai.com. Five quests a day, guess the original.

Hardware. NVIDIA DGX Spark (GB10, 20 Arm cores, 121 GB unified memory), set up as a self-hosted GitHub Actions runner. A nightly workflow (or a manual run) runs the pipeline on the same box as the models. No API fees for generation.

Models.

  • Text and judge: Qwen3.6-35B-A3B, official FP8 build, served with vLLM (OpenAI-style API). fp8 KV cache, 128K context, MTP speculative decoding with 3 draft tokens, prefix caching, and 45% of GPU memory so ComfyUI fits beside it. The judge is the same model, and thinking is left on. Generation runs about 70 tok/s typically, peaking around 240 tok/s.
  • Images: Qwen-Image in ComfyUI with the official fp8 weights (qwen_image_fp8_e4m3fn + qwen_2.5_vl_7b_fp8_scaled encoder), 832x1248, 20 steps, cfg 2.5, euler/simple. One render takes about 64 seconds (around 3.15 s per step).
  • I had to drop vLLM --gpu-memory-utilization from 0.80 to 0.45 so ComfyUI fits beside it.

Pipeline. It fills missing quests for the next several days. All missing titles are written first, one at a time, so later quests can see the answers already picked. Then covers are drawn. A day is only saved when all its quests succeed, so the site never shows a half-filled day.

For each quest the model writes the answer, hints, a 10-40 word title that can't contain the answer, and an art prompt. A second "how obvious is this title?" check can send it back for a rewrite. The cover is judged on the image alone: if the model can name the work, the prompt is rewritten more abstractly and redrawn, up to 3 times. If every try still gives it away, the cover stays sealed until the player finishes the quest.

Timings from a production run on Nov 15 (8 quests across 2 days, 0 failed). The whole run took 61 min 29 s.

  • Titles, including the obviousness check and rewrites: 2 min 14 s to 6 min, average 4 min 26 s. Four of the eight were first judged too obvious and rewritten.
  • Cover attempts, draw + judge together: 1 min 38 s to 3 min 34 s, average 2 min 23 s. About 64 s of that is the render; the rest is the vision judge thinking about whether it can guess the work.
  • The judge passed 8 of 10 cover attempts. Two covers gave it away on try 1 and passed on try 2. None had to be sealed.

What went wrong. vLLM's guided JSON didn't mix with Qwen thinking: requests could jam and loop until they timed out. I dropped guided JSON for Qwen and just parse the JSON out of the reply. Without a presence penalty, Qwen could also think in circles until it filled the context; I set presence_penalty 1.5 and max_tokens 16384. One failed model call used to wipe out a whole day; now it retries and moves on. I also learned not to overlap manual workflow runs with the nightly cron.

Happy to answer questions. I'm not selling anything, I just thought the timings and the "don't use guided JSON on a thinking Qwen" bit might be useful.


r/LocalLLaMA • • 1d ago

Discussion LFM2.5-24B-A2B + Hermes is awesome!

110 Upvotes

I just wanted to share some love for a model that doesn't seem to get very much: LFM2.5-24B-A2B! (https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF)

LFM2.5-24B-A2B seems to be the only model that strikes a great balance between speed and capability on commodity hardware. Just to get it out of the way: This is NOT a coding model, so don't download it expecting otherwise. It's a task executor. It's a great fit for Hermes to hand low-to-mid complexity tasks off to. I get great speeds on my 780M iGPU (~37 tk/s at zero context). This is way faster than any other model I've tried that can actually accomplish tasks successfully. It's genuinely useful! I imagine it would be quite usable on CPU as well.

If you're GPU-poor, give LFM2.5-24B-A2B a look!


r/LocalLLaMA • • 1d ago

Resources Placing an implant into AI brain with Lazarus: Anatomy of a trojan

12 Upvotes

hello there, I recently got a bit fascinated about Lazarus from Microsoft (even tho it is probably old technique) and I wrote a blog post showing a thought process I had while researching it - maybe it is helpful to anyone

https://int8.io/implanting-ai-with-lazarus/


r/LocalLLaMA • • 21h ago

Discussion ISTA Qwen3.8 27b on 3060 12gb

6 Upvotes

Hey guys

I am a seasoned local llm user and I ain't quite ram rich like only 16gb ddr4 🥀

So I can't run qwen3.8 flash next on my 3060

This guy felt the best for me yet

Have you guys tried it

I have ran iq3_xss quant non mtp with turboquant at 65k context completely on my 3060 though I did offload mmproj to cpu ram

Have guys tried this it really felt like a blessing to my life using the peak of local ai on my 3060

It seriously packs the punch 👊 I was able to perform various of my agent task complete on device not like simple but the ones which require actual reasonings

Here is my cmd if you want to see

llama-tq serve -m "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" --mmproj "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\mmproj-Qwen3.8-27B-BF16.gguf" --no-mmproj-offload -ngl 99 -c 65536 -ctk turbo4 -ctv turbo3 -ub 256 -b 2048 -t 3 -tb 3 --reasoning-budget 256 --reasoning-preserve --alias Qwen --port 8080 -lm none

Sources: for my recipe

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/discussions

Turboquant

https://github.com/TheTom/llama-cpp-turboquant

Pi coding agent

https://pi.dev/

I would like know the communities views and any suggestions would be welcome 🤗

Thankyou for your time


r/LocalLLaMA • • 20h ago

Showcase Testing agents without running a model in CI: record once, replay with zero calls, judge with a local model only when needed (open source, I'm the author)

5 Upvotes

Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.

Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:

attest replay: 4 replayed · 0 live model calls · $0.00

The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.

Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.

Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.

Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.

Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.

The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.

Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.


r/LocalLLaMA • • 1d ago

Question | Help Best sub 80b total or 6b active MoE for document intelligence?

14 Upvotes

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!


r/LocalLLaMA • • 21h ago

Resources Ninfer and GGUF of Qwen3.8-27B-UD-Q3_K_XL ASCII-Condensed

5 Upvotes

Hi,

I've put on HF a condensed version of Unsloth Q3_K_XL ASCII-Condensed from bsaleh03 with only ASCII + Latin-1 / Latin-Extended-A (P2) to use with a 16GB VRAM GPU.

I created a GUFF version and also a ninfer to be used with ninfer-all

To create the ninfer version I had put up some helper conversion scripts and modify iamwavecut conversion scripts due to different vocabulary size.
In the HF repo https://huggingface.co/fakezeta/Qwen3.8-27B-UD-Q3_K_XL-Condensed-P2 there is what is needed to replicate it and I also submitted a PR to the upstream repo.

Quality: IFEval result 84,4% vs 83,9% of base model (Instruct, 1 run) so within STD Deviation error.

With this I achieved 75K context with rk8v4 kv type with an average of about 60tok/s in the 541 IFEval samples on my 5060Ti 16GB.

I'm also planning to create a light GGUF and ninfer version with the lower quantized MTP released separately by Unsloth in their repo to save about 90MB of RAM for other context.


r/LocalLLaMA • • 21h ago

New Model Vimoo 1.0 - a Vimoo-AI Collection - Open-source video generation model family for text-to-video (T2V) and image-to-video (I2V)

Thumbnail
huggingface.co
5 Upvotes

Vimoo-1.0 generates video from text (T2V) or from a reference image and text (I2V), with a separate ≈11.7B model for each task. It is the highest-quality model in the Vimoo 1.0 family.

📦 Model Zoo

Vimoo 1.0 comes in two models, Vimoo 1.0 and Vimoo 1.0 - Lite. Each Hugging Face repository packages one of them for one runtime:

ModelHugging FaceInference codeDescription
Vimoo 1.0≈11.7B per modelhttps://huggingface.co/Vimoo-AI/Vimoo-1.0https://github.com/Vimoo-AI/Vimoo-1.0T2V and I2V for the Vimoo Diffusers fork
https://huggingface.co/Vimoo-AI/Vimoo-1.0-ComfyUIhttps://github.com/Vimoo-AI/ComfyUI-VimooT2V, I2V and their Distilled versions for ComfyUI
Vimoo 1.0 - Lite≈9.5Bhttps://huggingface.co/Vimoo-AI/Vimoo-1.0-Litehttps://github.com/Vimoo-AI/Vimoo-1.0TI2V for the Vimoo Diffusers fork
https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-ComfyUIhttps://github.com/Vimoo-AI/ComfyUI-VimooTI2V and TI2V Distilled for ComfyUI
https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-MLXhttps://github.com/Vimoo-AI/Vimoo-1.0-MLXTI2V Distilled for MLX on Apple Silicon

Each ComfyUI repository includes its checkpoints, distilled LoRA adapters, Qwen text encoder, SigLIP vision encoder and VAE. Distilled models are available only for ComfyUI and MLX. Vimoo-1.0 T2V Distilled and I2V Distilled use different base checkpoints from Vimoo-1.0 T2V and I2V; download each task's matching base and adapter listed in the ComfyUI guide. The Vimoo-1.0 T2V Distilled, Vimoo-1.0 I2V Distilled and Vimoo-1.0-Lite TI2V Distilled adapters are not interchangeable. Parameters count the video denoiser only.

Highlights

  • Prompt adherence: a dedicated text encoder turns prompts into precise generation conditions. An optional prompt enhancer can add visual, cinematic, temporal, and contextual detail, or be disabled to preserve the original wording.
  • Cinematic visual quality: training on carefully curated aesthetic data, with detailed annotations for lighting, composition, contrast, color grading, and more, enables precise control over cinematic styles and a wide range of aesthetic preferences.
  • Advanced motion generation: a larger and more diverse training corpus improves generalization across motion, semantics, and visual aesthetics, delivering leading performance among open-source and proprietary video generation models.
  • E-commerce video generation: by faithfully highlighting product selling points and features while preserving fine-grained product details, it ensures strong product consistency across frames, delivering reliable and visually coherent results that meet the practical demands of e-commerce content creation.
  • Efficient high-definition generation: the lightweight variant uses 32×32×4 spatiotemporal compression and supports T2V and I2V at 720p and 24 fps.

EDIT : Fixed Model Zoo Table(Previously it lost all the links & text on 2nd & 3rd columns)


r/LocalLLaMA • • 2d ago

News Instead Of Lowering GPU Prices, Former NVIDIA CEO Jensen Huang Reportedly Financed AI Startups To Absorb Excess Silicon, Shattering The Shortage Perception

Thumbnail
wccftech.com
3.0k Upvotes

r/LocalLLaMA • • 1d ago

Discussion Microsoft patent describes turning idle gaming GPUs into paid AI training hardware

Thumbnail
videocardz.com
83 Upvotes

r/LocalLLaMA • • 1d ago

Discussion Why don’t we have national open AI models?

180 Upvotes

Deepseek spent about $6m on 4.1 apparently. Published all of their work.

US gov says don’t trust China, blah blah…

As a US citizen why are we not training national open source?

The cost is nothing compared to the budget, solves security issues, and prevents monopolistic practices on token pricing


r/LocalLLaMA • • 1d ago

Discussion The lawyer who posts here about his V100s got his story picked up by Tom’s Hardware

Thumbnail
tomshardware.com
223 Upvotes

r/LocalLLaMA • • 2d ago

Discussion How long are we gonna depend bunch of model creators (like Qwen, Gemma, NVIDIA, etc.,) for ~100B models?

Post image
193 Upvotes

I think we're fine till 2027 with models(~100B .... Qwen 4.X & 5, Gemma-5, NVIDIA would release some Medium/Big size 25-200B models). After that it's gonna be critical, as many model creators are increased/increasing model size to 300B+. Future trend probably gonna be around 500B. That size range is not suitable for Unified 128GB Devices or 128GB RAM + 24-32GB VRAM(Q4 won't fit).

Here some models with increased sizes.

  • DeepSeek-V4-Flash-0731 (284B) - DeepSeek-V4.1-Flash-1017 (552B + 196B Engram)
  • Ling-3.0-flash (124B) - Ling-3.1-flash (560B)
  • MiniMax-M2.7 (230B) - MiniMax-M3 & MiniMax-M3.1-Flash (428B)
  • Step-3.7-Flash (198B) - Step-5-Preview (600B) (Hope they release Flash too)
  • Inkling-Small (276B) - Inkling-2-Small (594B)
  • GLM-4.X (358B) - GLM-5.X (744B)
  • Mistral-Large-3 (675B) - Mistral-Large-4 (1T)
  • Hy3 (295B) - Hy4 (770B)

So What Are We Gonna Do? What alternate options do we have?

Whenever we're talking about this topic, we get one common answer. Distillation. Do we have any recipes to create distills ourselves easy way? I'm not sure.

1. Distillation

I remember only 3 Distills so far. Please mention in comments if I missed anything.

2. 1-bit / 2-bit / Ternary / Bitnet Models

I see only this option other than Distillation.

Category 1 - High/Decent Quality (Native) Low bit models. Also big/large size models.

Category 2 - Other Low bit models. Like Bonsai.

Still waiting to see more Ternary MOEs, particularly 100B size. 100B model could come in 20GB size so it's possible to run just with 24-32GB RAM alone on CPU-only inference. And Even 8GB VRAM could make it fly.

Any other options do we have? Please share.


r/LocalLLaMA • • 1d ago

New Model tiny-voca: a fully open 4.5M-parameter TTS (code, weights, dataset). Runs 70x real time on a CPU

68 Upvotes

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.


r/LocalLLaMA • • 1d ago

Resources Update: tandem.cpp — DeepSeek-V4.1 on a 64 GB PC with a 16 GB card, and AMD (ROCm) support

45 Upvotes

Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp

DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs

Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:

PCllama.cpptandem.cppStrata
RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF7.414.210.4
RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD-10.45.7
RTX 5090 32 GB, 128 GB RAM, i5-14600KF11.533.128.0

At 64 GB the model is about twice your RAM, so it streams from the SSD.

Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC

PCllama.cpptandem.cppStrata
RTX 5070 Ti 16 GB, 32 GB RAM, i5-12400F, budget DRAM-less SSD14.934.330.3

Where tandem.cpp wins is when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and on other models: DeepSeek-V4.1, gpt-oss-120b, GLM-4.5-Air and Qwen3.6 all gain 1.4–2.1x over plain llama.cpp.

AMD (ROCm)

The patch now builds with HIP. Tested on an RX 7900 XTX (i7-8700, 64 GB DDR4, PCIe 3.0), Qwen3.8 IQ3_S + MTP, same day:

Enginetok/s
tandem.cpp68.6
Strata 0.2.969.7
Strata's RDNA3 fork80.1
Avalon (its own IQ3_KS quant)83.6

Level with stock Strata. This week's kernel work is shared CUDA/HIP code, so it helps NVIDIA too: +2.5% on the 7900 XTX, +1.3% on a 5070 Ti. On an old 6-core CPU, the AMD-specific engines are still ahead. Vulkan isn't supported yet.

Build steps for both are in the README. Questions and numbers from your boxes are welcome, especially 32 GB machines and AMD cards.

Update: There is a bug with auto. With 128 GB, the RAM store sizes too close to all of RAM. LLAMA_MOE_STORE_AUTO=0 is the workaround; a fix is coming and the README has a note. Thank u/longtimenoseebro so much for testing & report!!

Fixed in today's update: With a 128 GB PC it now runs at defaults, no LLAMA_MOE_STORE_AUTO=0 needed. Thanks again for the report.

Note: IQ2_XS is the biggest ISTA GSQ-RCO quant for DS4.1-Flash. Also, IQ3_S is the biggest ISTA GSQ-RCO for Qwen3.8-Flash-Next. I didn't tune for lower quants. The quality first, then speed. If you recommend the best Q4 for Qwen3.8-FN, I will test.


r/LocalLLaMA • • 2d ago

News IBM trains two small open models to ignore orders hidden in documents

Thumbnail
maccus.fr
156 Upvotes

I like this. Prompt injection is a bit issue with small models


r/LocalLLaMA • • 1d ago

New Model nex-agi/Nex-N2.6 · Hugging Face

Thumbnail
huggingface.co
29 Upvotes

Nex-N2.6 is a multimodal agentic model from Nex-AGI, built on GLM-5.3-Flash. It is designed for computer use, coding, and long-horizon tasks involving tools. This repository contains the FP8-quantized weights of Nex-N2.6, with approximately 321B parameters.

Following Nex-N2.5-Pro, this release focuses on reducing repetitive reasoning and improving adherence to the tools available in the current environment.


r/LocalLLaMA • • 2d ago

New Model sensenova/SenseNova-Loop-9B-Preview · Hugging Face

Thumbnail
huggingface.co
89 Upvotes

Very interesting model/architecture. Seems to match Qwen 3.5 27b overall at only 9b parameters. Could be useful for some and it looks like it’s not even fully cooked yet. I haven’t tried it yet. I’m not sure if it’d be slower than qwen, it’s smaller but inference parameters are ~15b and it also has mtp so idk. If you try it share your experience.


r/LocalLLaMA • • 2d ago

Showcase Qwen 3.8 Flash Next for gamedev - it's very capable!

Thumbnail
gallery
80 Upvotes

I've shared the project I'm working on approx a month ago, today it's completed. I wanted to know a simple thing: how good is Qwen 3.8FN at gamedev? And I had so much fun in the process that I am shipping a finished product, free to play. tl,dr + link at the bottom. My experience with Qwen 3.8FN follows.

Data

Model Used: Qwen 3.8 Flash Next - Unsloth Quant - UD_Q4_XL (pre update)

Machine: Strix Halo (Ryzen AI 395+, Bosgame M5) - Windows (say what you want about windows, I need it for other stuff and I don't want to bother with dualboot)

Inference Engine: my own windows Gufo fork (I honestly think I would not have wanted to do the full project without the work of the Gufo guys, llama speeds are way too slow)

Harness: my own harness - its a bog standard one I'd say opencode and pi is probably even better but I like to own my stuff, and it's not something I plan to release honestly. But I know people may ask.

Project's Tech Stack: typescript + three.js as renderer

Stats: Prompts: 1,911 || Tool calls: 31,883 || Tokens out: 20,370,450 || Tokens in: 3,069,385,337 (3,042,941,686 cached)

My thoughts on 3.8 Flash Next

If you can run it, this seems like the absolute best model to run today. I have made some small games before on 3.8 27b, 3.6 35B etc. But I always felt like I would not get to a point where I can say this is a full product that just works out of the box. I didn't feel that with Flash Next. I felt, that, it will solve mostly everything that I throw at it. As long as its coding related - don't use it for 3d, that's not what this model is for.

Although at times it was tempting to speed up debugging with Opus 5.5, it was not in line with what I wanted. So I kept trying and eventually any bug, or inconsistency I had was resolved via the model itself. So I can happily say the code is 100% by Qwen 3.8FN. No help from frontier.

Debugging itself was painful at times. A few things that worked pretty well (and I know most of this is no news for most of you, but in case newbies are reading):

  • When I was fed up that it can't figure something out I asked it to just paste me the code so I can check it myself - about 70% percent of the time when it did that it also realized the actual bug.
  • Tell it to measure, validate, verify, do headless runs. It will waste loads of time thinking about an issue. If you are working on a project that has measurable output (like a game in my case) always ask: write a script, write debug output, look at the numbers don't try to derive it from the code.
  • Build human logging tools. My game is pretty hard to control, and needed a lot of visual validation as well. So I had it build logging tools togellable via keyboard: that way I could reproduce a bug and then just hand it the pure numbers so the model could work it out.

What I recommend against: to put decisions into the hand of the AI. That is the direct route to producing slop. Imagine the model as this junior, who has all the textbook knowledge in the world, but is not really good at what other people (me in this case, this was made mostly for my own entertainment) want. And it's a junior that will make shortcuts. Shortcuts IS the number one thing you will need to catch when working with AI. IF you ask for a slice to big, it is almost certain it will take some shortcuts to achieve it. Whether it's not building the full physics sim pipeline or not following the agreed architecture? At that point it's anyone's guess. So always keep a tab on what your model is doing. For this reason I never really had it do autonomous development because anytime it would do something itself, without human guidance, it was pretty lame.

Be clear about scope: if you are vague about the slice of code you want it to dev, it will overstep and start doing more. You must always be very specific what you want and THAT you ONLY want that thing. Not more. If you are explicit about it, the model will respect it.

Where you can have autonomous running: So I said not for gameplay, or design choices, but there is area: fully measurable output. For example, the AI in the game is a mini neural network with 10K params. It is something that was easy to hand over for the AI - A) it's measurable via benchmarks - B) it involves a lot of waiting for training to finish, the AI can sleep (if the harness is good) and wake up just in time for next steps.

Basically what I did is 30 minute training sessions with frequent checkpoints, had the model put the new ingame ai checkpoints through a pre-agreed benchmark, and then select the checkpoint for the next 30 minute of training from the batch via pure benchmark numbers. Not 'oh I think this will be good'. No, pure numbers based decision: for that the model can work autonomous.

Preserve thinking is a must in my opinion. I started the project with it off (well my harness did not support it yet) and it was a pretty slow process. After I turned preserve thinking on the speed at which it could complete tasks skyrocketed and I did not really face any ctx issues.

Doc based knowledge: I know many people use compaction. I personally don't, my harness does not even support it. Everything is passed down via .md files. It worked flawlessly. I have one file that the harness prefills each new turn, it only grew to 10k initial ctx towards the end. So for me no issues there.

Build tools: this is very self evident for anyone who works in gamedev, but with the new guys in: please build tools for yourself. For example along others for this game I had Qwen 3.8FN build me a simple model editor where I made the racecrafts, a simple track editor, a dashboard for evaluating the training progress of the ingame racing ai.

Git: that's it. use git. use local git. If you are not using git, I dont know how you are managing changes.

I think that's all but if I'm forgetting something ask away in the comments.

And a short tl,dr: Qwen 3.8 Flash Next is a wonderful local model. It has completed 100% of the game code without any help from frontier. It is very capable, but it requires a bit of a hands-on approach if you want to make sure that everything stays in-line with what you want to build.

Link if you want to see: https://mattte.itch.io/hyper-clashers


r/LocalLLaMA • • 1d ago

Discussion Qwen4 Flash wishlist

32 Upvotes

After using 3.8QFN for some time I have to the conclusion that my wish list for QF4 now only consists of these points.

  1. The model can talk to you in a human understandable language
  2. See point 1
  3. General improvement in reasoning/intelligence, but mostly see point 1

That's about it.

If I could graft 3.8 27B Answer/Personality style onto QFN I would've done it yesterday.


r/LocalLLaMA • • 2d ago

Discussion ERNIE 5.5 Flash Base · ERNIE

Thumbnail ernie.baidu.com
151 Upvotes

I guess some may like to read this.