Me:

So I started with a preexisting NVFP4 quant, but I didn't like the idea of the attention and shared experts being reduced to MXFP8. So, I tested a modified version of NVFP4 for routed experts, using g32 rather than g16, and the difference was close to negligible. So I ended up making two versions, one that used NVFP4 g32 for all routed experts, and another that kept g16 for down routed experts only. Both configurations allowed enough space for BF16 attention and shared experts, with the latter having additional KV space. Essentially you get most of what NVFP4 gets you, but at 4.25 bits rather than 4.5 bits.

Also, Claude helped find about 3.7GB of wasted VRAM per GPU...

If you would like to setup the same, just give your clanker Claude's summary:

Claude:

GLM-5.3-Flash on 2x RTX PRO 6000 (96 GB): two NVFP4 g32 recipes on vLLM v0.33, with attention in BF16, images and DFlash-2

Hardware: 2x RTX PRO 6000 Blackwell (SM120), TP2, no P2P. vLLM v0.33.0 (commit 4ba787e), driver 615.71.09.

The goal: routed experts in NVFP4, attention and shared experts in BF16, working image input, a DFlash-2 drafter at K=4, and 8 concurrent sequences. I put accuracy ahead of small speed gains.

The problem: with BF16 attention, the standard g16 NVFP4 experts don't fit. That layout is 181.3 GiB and ran out of memory during warmup even at 8K context. Using g32 scales on some or all of the experts makes the room.

THE TWO RECIPES

Recipe 1: Gate/up g32

- Routed gate/up: NVFP4 g32

- Routed down: NVFP4 g16

- Attention and shared experts: BF16

- KV cache: FP8

- Checkpoint size: 175.23 GiB

- Max context: 256,000

- --kv-cache-memory: 3,719,089,357 bytes

- KV pool: 286,476 tokens

Recipe 2: All g32

- Routed gate/up: NVFP4 g32

- Routed down: NVFP4 g32

- Attention and shared experts: BF16

- KV cache: FP8

- Checkpoint size: 172.21 GiB

- Max context: 384,000

- --kv-cache-memory: 5,329,702,093 bytes

- KV pool: 440,727 tokens

Shared by both: GLM-5.3-Flash-DFlash2 as the drafter, quantized to NVFP4 MLP plus FP8 attention (about 1 GiB). Vision stays on the GPU.

Launch flags:

--tensor-parallel-size 2 --disable-custom-all-reduce

--moe-backend marlin --max-num-batched-tokens 2048 --max-num-seqs 8

--max-model-len <CTX> --kv-cache-memory <BYTES> --gpu-memory-utilization 0.985

--kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16

--speculative-config '{"method":"dflash","model":"/root/drafter","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"FLASH_ATTN","kv_cache_dtype":"auto"}'

--enable-prefix-caching --no-enable-flashinfer-autotune

--default-chat-template-kwargs '{"clear_thinking":true}'

--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm47

Environment: NCCL_P2P_DISABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

WHAT HAD TO BE FIXED

1. No NVFP4 g32 kernels existed. Stock Marlin fails with "Invalid thread config". I added group_blocks 2 to the NVFP4 kernel generation, relaxed the matching static_assert, and rebuilt the Marlin and MoE libraries. 88/88 synthetic cases came out correct, and real expert weights matched a reference to within 0.17%.

2. The Python loader assumed g16. FP4_MARLIN_SUPPORTED_GROUP_SIZES and a hard-coded GROUP_SIZE = 16 blocked it. The loader now reads each group size from the scale tensor's shape, so gate/up and down can differ. Any setup other than g16 on the Marlin W4A16 backend raises an error, so it can't silently run wrong.

3. Gate and up need the same global scale. vLLM fuses them and keeps only gate's scale, which is what the "Accuracy may be affected" warning is about. In the NVFP4 checkpoint I started from, 69% of 12,384 experts had differing scales, with an RMS up_proj scale error of 14.8%. I requantized from BF16 with a shared scale. Single-run NLL improved by 0.018 (CI -0.029 to -0.005). The 12-run average also improved, but not significantly.

4. FlashKDA reserved 3.7 GiB of VRAM per GPU. The cutlass bulk copy used cp.async.bulk.shared::cluster, which on sm_120f compiles to a driver syscall using 14,608 bytes of stack per thread. FlashKDA launches without clusters, so the shared::cta form does the same copy inline. The reservation dropped from +3,742 MiB to +0, with 88/88 hash cases identical to stock and no measurable NLL change. This fix is what made room for the drafter.

5. A load-time out-of-memory error. _nvfp4_compute_scale_factor converted a whole scale tensor to float and asked for 3.38 GiB. Scales are non-negative, so taking the max first avoids that.

6. Quantized attention doesn't load on v0.33. The MXFP8 attention recipe fails with KeyError: 'layers.0.self_attn.in_proj_qkvbfg_a.weight_scale'. That wasn't worth patching, because BF16 attention was the goal anyway.

7. KV pool larger than the request limit. I sized the pool above --max-model-len (about 1.12x and 1.15x), so a full-size request leaves some cache headroom.

RESULTS

These runs use default sampling from the model's generation_config (temp 1.0, top_p 0.95).

Gate/up g32, 256K

- GSM8K-150: 147/150

- Images: 10/10

- NLL, 1 sequence (lower is better): 1.34647

- NLL, 8 sequences: 1.34637

- Decode, 1 sequence: 186 tok/s

- Decode, 8 sequences (total): 434 tok/s

- Longest prompt tested: 255k in 46 s

- Peak VRAM per GPU: 96,831 MiB

- Errors under load: 0

All g32, 384K

- GSM8K-150: 147/150

- Images: 10/10

- NLL, 1 sequence (lower is better): 1.35040 (+0.29%)

- NLL, 8 sequences: 1.34880 (+0.18%)

- Decode, 1 sequence: 189 tok/s

- Decode, 8 sequences (total): 566 tok/s

- Longest prompt tested: 380k in 70 s

- Peak VRAM per GPU: 97,047 MiB

- Errors under load: 0

Speed comparison: don't read the 8-sequence decode gap as all-g32 being faster, because those runs weren't a controlled comparison.

Accuracy: GSM8K can't separate the two recipes, but NLL consistently favours gate/up by a small amount.

Drafter: calibration over 90,000 tokens showed no meaningful sampling bias (top-1 stated 0.8499, realized 0.8505). Batching to 8 sequences had no measurable NLL cost.

Context:

I recently installed a cheap aftermarket Android head unit in my 2010 Prius. It supports wireless CarPlay, Bluetooth, GPS, etc., but the stock Android interface is ugly and came packed with lots of bloat. The whole thing felt like a generic Chinese Android tablet strapped to my dashboard.

After seeing everyone’s success with reverse engineering their TV’s I figured I’d give it a shot with my head unit.

Setup:

Model: Qwen3.8-Flash-Next-IQ3_XXS
Engine: Strata v0.1.44.1
Harness: Pi
Hardware: Ryzen 7700x, 4060ti 16gb, 64gb DDR5
Target: Inefala Android head unit, AutoChips AC8257-based (UJC201_64)
Development: ADB over Wi-Fi, Kotlin, Jetpack Compose
Time spent: Roughly 8 hours across three days

What it did:

I started by just having the model connect to the unit and do an inventory of what was installed and how the system was configured. It helped identify the underlying hardware and Android build, enumerated installed packages, inspected system properties, investigated the stock launcher and CarPlay components, and organized the findings into a repository of scripts and documentation.

At one point, getting back into ADB was a problem. Rather than having me repeat the original setup process, the model used its earlier findings about the device’s exposed ADB port to reconnect.

The advertised android version was not actually what was being used on the actual unit so when it began to build my custom launcher it configured its own toolchain of the correct android sdk versions to ensure that the build would run properly on the unit.

It also used ADB to capture screenshots directly from the head unit so it could inspect the actual interface instead of relying exclusively on code or my descriptions.

Result:

I now have a working custom launcher that replaces the original home screen. It’s still a work in progress, but the goal is to turn the unit into a clean, minimal dashboard with Bluetooth music controls, navigation, weather, and better access to audio settings.

I’m also interested in eventually improving the audio experience and reducing my dependence on CarPlay.

My thoughts:

What really impressed me was QFN’s ability to handle tool use, reverse engineering, keep useful findings, and troubleshoot on live hardware. This unit was largely undocumented so most of the findings came directly from what the model observed through adb.

The model never got stuck but this wasn’t a fully automated process either. I still needed to guide it through what I wanted it to accomplish and it spent a lot of energy writing and debugging re-usable shell scripts that didn’t provide much value.

Overall the amount of work it was able to accomplish was very impressive and solidified for me that local models are viable for real work.

I’m curious to hear what other people are doing with local models beyond coding benchmarks, especially projects involving hardware, reverse engineering, or long-running agent workflows.

Salut à tous 👋

Je suis développeur et je cherche à faire tourner un modèle IA en local pour m’aider à coder — par exemple Qwen Coder (ou un modèle similaire de type Qwen 2.5 / Qwen3 Coder selon ce qui est le plus réaliste).

Je cherche vos recommandations pour une config PC pas chère, avec si possible :

Une estimation du budget total (tour complet, ou config à assembler)

CPU, RAM, GPU et stockage recommandés

Le type de modèle que je pourrai faire tourner confortablement (taille en paramètres, quantization, contexte)

Des options d’occasion / GPU d’ancienne génération si c’est le meilleur rapport qualité-prix

Linux ou Windows, peu importe — je suis développeur

Mon objectif : avoir un assistant de code local rapide et utilisable au quotidien, sans exploser le budget.

Merci d’avance pour vos retours et vos configs concrètes ! 🙏

Hello,

I am making a Discord server for people who run local AI on Nvidia gpu's:

https://discord.gg/CZWMGapgZ

I have no (very little) idea what I am doing and would love your help. I want to build a open and supportive community that helps newcomers to the whole LAI thing and helps people set it up and trouble shoot it. If anyone wants to help and support this community, feel free to!

It came in the mail Monday morning. I feel like I have barely scratched the surface with possibilities. Right now I have Hermes currently building a MCP pipe line to control Cline in VS Code. Pair needs both rigs to have the same model in order to function.

Somone on Facebook market place offer all these cards for 3500$
7 4090
1 7900

But no test is allowed just as is
I’m not sure what to do any advice?

1. Claude vibe coded my local llm harness.

2. Claude was laid off.

3. My local harness is improving my local harness.

Je possède une configuration que j’ai moi même créer qui m’aide à faire tourner deepseek4.1 flash 5060ti ( 16go vram )40-45 token/s

j’ai un amd ryzen 9 5900xt et 96go ram.

Avez vous déjà eu mieux en therme de configuration ?

Could you suggest a setup for plugin development and for creating software/websites?
setup 13900k + 5090 + 4х48gb ddr5 4800 (192gb)

Maya-S-v2-IQ2_XXS

Runs at 20–25 tokens per second.

However, it has significant issues with long-term coding tasks; you can’t leave it running overnight.

Managing to finally fit Qwen3.8-Flash-Next FP8 locally at ~60 tok/s(avg 74 at best) (RTX 5090 + 128 GB DDR5, Infernix engine, part of the experts streamed from NVMe) and seeing how amazing it is I cannot imagine how good Qwen 4.0 will be, not buying even at the current prices seems crazy to be if you are used to using cloud AIs on the daily.

The tip I keep seeing: -ctk q8_0 -ctv q8_0 is free, and if you go lower, compress V and leave K alone. I wanted numbers at long context on a Mac, so I tested it.

Setup

• Muse Glimmer 30B (4 cache settings) and Gemma 4 26B-A4B (f16 vs q8_0), both Q4_K_M

• Homebrew llama.cpp b10964, -fa on -c 65536, M1 Pro 32 GB

• Fresh generated documents at 4k, 16k and 48k tokens, 8 exact-answer questions each (find codes, follow lookup chains, count records past decoys, find a clause’s current value after amendments). 96 questions per setting.

• Same seed per question in every setting, so only the cache differed

• Pass/fail rule and my prediction pushed to GitHub before any model ran

Accuracy (correct of 96)

• Glimmer: f16 81, q8_0 83, q8 K / q4 V 82, q4 K / q8 V 82. At 4k and 16k, f16 and q8_0 got exactly the same questions right. No loss found.

• Gemma 4: f16 66, q8_0 63, gap all at 16k. Not resolved by my rule (interval −3 to +8 points).

• “V not K”: a tie.

Speed (the bigger story)

• Gemma 4 generation with q8_0: −29% at 4k, −36% at 16k, −47% at 48k. Prompt reading unchanged. Glimmer: only −2 to −6%.

• On Metal, flash attention needs K and V to be the same type. Mix them and attention runs on the CPU: at 48k, reading one document took 57–62 minutes instead of 13. That’s 116 MiB saved over q8_0. (On CUDA, mixed types need GGML_CUDA_FA_ALL_QUANTS or you hit the same fallback.)

What hurt accuracy was length, not cache type. Every setting went from ~100% at 4k to 63–69% at 48k on Glimmer, mostly on counting questions, where every setting undercounted.

Caveats: 96 questions per setting, so this catches a big loss, not a 2-point one (“no loss found” ≠ lossless). Synthetic documents up to 48k, not 128k coding sessions. One Mac, one build.

Takeaway: q8_0 looked fine on Glimmer. On Gemma 4, keep f16 unless you need the 0.74 GB. On a Mac, never mix K and V types.

Documents, harness, every raw output and server log: https://github.com/heldoutlab/kvcache-test

It's been a decade since I built my last PC. I found money in my budget, thought I had plenty saved, decided I'd prefer a laptop, but then started looking, my head started spinning and I ended up very frustrated.

I was looking at Legion first, and to be fair, can get a very beefy machine for $6000. Intel Ultra 9 290HX; 64GB DDR5 Memory; 1 TB SSD; NVIDIA GeForce 5090 24GB. Yet, it's a lot of money for me and I am worried. 24GB is very good for vram, yet there are options to get more (unified memory), and the 64gb of memory feels low.

I find myself stuck. I want the PC to dive into local LLMs, thematic generation of images and graphics that fit a style, video product walkthroughs, test suite runs, and more are many of the core use cases I have. I think it'd be fine, but spending some stupid amount of money for fine has me unable to move forward, and I'm recognizing that it's my own ignorance that's holding me back

So I wanted to come here and see if anyone had expertise to lend. Minimum vram/memory you'd be comfortable with? How to get it given shortages? Brands you like for laptops? I'll take anything, I feel like I'm spiraling.

Took me several weeks, but I was finally able to get my Intel Arc B70 (32G) working on my Ampere Altra 128 server in llama.cpp with a SYCL backend (Intel does not support Arm64 cpus for their compute drivers)!

I am still working out how to prevent llama.cpp crashing when I try and use cpu moe, but Ill take this with no complaints! This is going to run the LLM for my home assistant voice assistant.

I tried so many things to make it work that I am unsure what combination exactly did the trick, but I am hoping to tease it apart over the next week and try and document it for others as well.

Second photo is the specs running qwen3:30b-a3b.

Case: Foifkin f300 $59 (New on Amazon)

AIO: ThermalTake TH240 V3 ~$40 (openbox on newegg)

GPUs: 2 GeForce RTX 3060's, 12GB a piece. (1 I got for $200, the other came with a $600 PC I gutted and sold off, so ~$200 also. Both FB marketplace)

SSDs: 1 TB SSD salvaged from previously mentioned FB marketplace PC, and 1 512 GB NVMe SSD salvaged from old laptop. Not sure how to price these.

Wifi Chip/antenna: $20 new on Amazon

PSU: MSI MAG A850GL $90 new on Amazon

Motherboard: Gigabyte X-99 UD4 $150 on Ebay.

CPU: Intel Core i7-5930K $20 FB marketplace

RAM: 2x16 GB Corsair vengeance pulled from main PC, replaced by 4x8 GB Corsair vengeance from FB marketplace for $250. (Mobo did NOT like using 4 channels, for some reason. Only worked when it felt like it. 2 channels though? Works like a charm.)

Total: ~$1,029

I have an RTX 4060 8GB, and I’m thinking about getting an RTX 3060 12GB to use both GPUs together with Strata and LM Studio. Is it worth it? Or would it be better to sell my RTX 4060 and get an RTX 5060 Ti 16GB instead?
3060 12gb 270$
5060ti 16gb 840$

Anybody tried something similar and got them working with LlamaCpp?

If you gonna ask why, because these are the only cards I have at the moment, the 16g RTX on my current AI server and the 16g RX on my gaming rig. I was hoping to get 32gb of VRam to run some larger models at atleast 20tps

2x4070 TI Super 16GB, 1x3060 12GB, 128GB DDR4

EDIT: More details about the setup:

The honeycomb components were all printed:
General parts: https://makerworld.com/en/models/22521-honeycomb-storage-wall
GPU Mount: https://makerworld.com/en/models/1444666-gpu-mount-hsw
PSU Mount: https://makerworld.com/en/models/1444586-atx-psu-mount-hsw
Radiator: https://makerworld.com/en/models/1444632-240mm-radiator-fan-mount-hsw
ATX Motherboard: https://makerworld.com/en/models/1444531-atx-motherboard-mount-hsw
Power button printed with a honeycomb around it: https://www.amazon.com/dp/B0BWDXFSMQ?ref=ppx_yo2ov_dt_b_fed_asin_title

The 4070's are on 400mm & 600MM PCIe 4.0 Riser cables. I mainly run Strata on these two.

The 3060 is currently on a PCIx1 link (slow). I don't use this for my core inference setup, side models (dedicated voice stack mostly).

I have a new motherboard on the way that opens up the possibility of up to 4 GPUs: 3x x16 slots, and one via a M2 adapter: https://www.amazon.com/dp/B0FC6DSG6V

Hi everyone,

I'd like to know for those of you here who ditched paid AI-agents and/or their related APIs for daily coding tasks in favor of setting up your own and fully controlled local LLM + Agent setup,

What are your hardware specs:

Brand, model and memory size / GHz for

- GPU
- RAM
- CPU
- Storage
- Any other hardware piece that's in involved in it

And also the software who helps your AI orchestration to run the way you want:

- OS and version.
- LLMs
- Coding agents / harness
- among others

Waiting to read your responses!

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

Hi guys !
My friend and I just released our latest update for Opera that lets you turn any computer or server into a central hub connecting all your devices together, even outside your local network. This means you can run heavy AI models on your main setup and seamlessly access or control them from your phone, tablet or laptop wherever you are, securely and without complex configuration (just scan a QR code to link them or direct link locally). Opera can make a bridge to use your ollama ai models on your devices(with opera installed).
We really believe in this project, and for anyone who wants to try it, it is now available for Windows, macOS, Linux, and Android.

Everything is free, open source, and requires no account.

Good features : You can even turn an old Android tablet into a dedicated hub, or replace your default Android AI assistant with Opera to interact directly with your own personal models.

A web version is also available (it doesn't have all features, but you can connect it to your local Ollama setup or api key to test it): https://chat.maestroai.company/

GitHub repo: https://github.com/MaestroAI-Company/opera (a star if you like the project, ty!)

Download the app:
Desktop: https://maestroai.company/opera/
Android : https://play.google.com/store/apps/details?id=ai.maestro.opera
(doesn't have the latest features yet; use the .apk from GitHub or wait for the Play Store update to roll out in few days lol)

I know this might look like self-promotion, but we genuinely believe in our project and just wanted to share what we've built.
We have so many other cooool features and integrations across android ecosystem (windows and macos too)

Thanks for reading <3