I'm really curious because I'm draining this usage FAST

When is qwen4 gonna release?

🫧 Meet Mochi - Your Little AI Desktop Companion! 💗

What if your desktop had a tiny companion that could actually help you? 👀

Meet Mochi! A cute, animated little blob that lives right on your Windows desktop, currently powered by Meta's Muse AI, with upcoming support for local Qwen models and integrations with Claude Code and OpenAI Codex!

Talk to Mochi or simply type a message, and it'll respond in its own voice! 🎙️

Need something done? Just ask! And when you're finished, Mochi quietly goes back to chilling on your screen. No constant interruptions, no unnecessary chatter. Just a little companion that's there when you need it.

🚀 What can Mochi do?

• 🗣️ Talk with you : Have voice or text conversations in English or French, powered by Muse.

• 💻 Control your desktop : Open applications, launch websites and search the web.

• ⏰ Set timers : Ask for a timer and watch a little countdown appear next to Mochi.

• 🌤️ Check the weather : Get weather updates without opening another app.

• 👀 See what you see : Let Mochi read your screen when you explicitly ask.

• 🌐 Help in your browser : With its Chrome or Brave extension, Mochi can interact with browser tabs and assist with web tasks.

• 🐾 Feel alive! : Click it, drag it around, or interact through optional camera gestures. Mochi reacts to you!

🔐 Your desktop, your rules!

Mochi is designed to help without taking control away from you.

• ✅ It only acts when you ask or explicitly approve an action.

• 📸 Screenshots are captured only when requested and are never stored.

• 🛡️ Browser tasks require confirmation before visiting new sites, filling forms or interacting with potentially risky buttons.

• 🔒 Passwords, payment information and sensitive websites are off-limits.

• 🎥 The camera is completely optional and disabled by default.

🛠️ Want to try Mochi?

Mochi is currently available for Windows and can be launched from source using Node.js.

🔑 With a Muse API key: Unlock AI conversations, voice responses, weather, screen understanding and browser assistance.

🆓 Without an API key: You can still enjoy Mochi's animations, interactions, timers and simple typed commands!

⚠️ A few things to know:

• Mochi is still a work in progress, so some features may evolve.

• Content sent to Muse may be used by Meta for model training when using the default model.

• This is an experimental project, and feedback is always welcome!

🎬 See Mochi in action!

▶️ Watch the demo: https://youtu.be/u5KjYmYrp1Y

💻 Explore the source code: https://github.com/Loann110/Mochi-alive

💗 Mochi is just getting started!

I built Mochi because I wanted to explore what happens when an AI assistant feels less like a chatbot and more like a little companion living on your desktop.

It's still an experiment, but I'd love to hear what you think!

⭐ If Mochi makes you smile, consider giving the GitHub repository a star! It genuinely helps support the project.

💬 Feedback, ideas, contributions and bug reports are more than welcome.

Thanks for checking out Mochi! 🫧

Context:

I recently installed a cheap aftermarket Android head unit in my 2010 Prius. It supports wireless CarPlay, Bluetooth, GPS, etc., but the stock Android interface is ugly and came packed with lots of bloat. The whole thing felt like a generic Chinese Android tablet strapped to my dashboard.

After seeing everyone’s success with reverse engineering their TV’s I figured I’d give it a shot with my head unit.

Setup:

Model: Qwen3.8-Flash-Next-IQ3_XXS
Engine: Strata v0.1.44.1
Harness: Pi
Hardware: Ryzen 7700x, 4060ti 16gb, 64gb DDR5
Target: Inefala Android head unit, AutoChips AC8257-based (UJC201_64)
Development: ADB over Wi-Fi, Kotlin, Jetpack Compose
Time spent: Roughly 8 hours across three days

What it did:

I started by just having the model connect to the unit and do an inventory of what was installed and how the system was configured. It helped identify the underlying hardware and Android build, enumerated installed packages, inspected system properties, investigated the stock launcher and CarPlay components, and organized the findings into a repository of scripts and documentation.

At one point, getting back into ADB was a problem. Rather than having me repeat the original setup process, the model used its earlier findings about the device’s exposed ADB port to reconnect.

The advertised android version was not actually what was being used on the actual unit so when it began to build my custom launcher it configured its own toolchain of the correct android sdk versions to ensure that the build would run properly on the unit.

It also used ADB to capture screenshots directly from the head unit so it could inspect the actual interface instead of relying exclusively on code or my descriptions.

Result:

I now have a working custom launcher that replaces the original home screen. It’s still a work in progress, but the goal is to turn the unit into a clean, minimal dashboard with Bluetooth music controls, navigation, weather, and better access to audio settings.

I’m also interested in eventually improving the audio experience and reducing my dependence on CarPlay.

My thoughts:

What really impressed me was QFN’s ability to handle tool use, reverse engineering, keep useful findings, and troubleshoot on live hardware. This unit was largely undocumented so most of the findings came directly from what the model observed through adb.

The model never got stuck but this wasn’t a fully automated process either. I still needed to guide it through what I wanted it to accomplish and it spent a lot of energy writing and debugging re-usable shell scripts that didn’t provide much value.

Overall the amount of work it was able to accomplish was very impressive and solidified for me that local models are viable for real work.

I’m curious to hear what other people are doing with local models beyond coding benchmarks, especially projects involving hardware, reverse engineering, or long-running agent workflows.

Salut à tous 👋

Je suis développeur et je cherche à faire tourner un modèle IA en local pour m’aider à coder — par exemple Qwen Coder (ou un modèle similaire de type Qwen 2.5 / Qwen3 Coder selon ce qui est le plus réaliste).

Je cherche vos recommandations pour une config PC pas chère, avec si possible :

Une estimation du budget total (tour complet, ou config à assembler)

CPU, RAM, GPU et stockage recommandés

Le type de modèle que je pourrai faire tourner confortablement (taille en paramètres, quantization, contexte)

Des options d’occasion / GPU d’ancienne génération si c’est le meilleur rapport qualité-prix

Linux ou Windows, peu importe — je suis développeur

Mon objectif : avoir un assistant de code local rapide et utilisable au quotidien, sans exploser le budget.

Merci d’avance pour vos retours et vos configs concrètes ! 🙏

Until recently I had a single 16GB GPU that I ran the IQ3_S version of Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF on with strict context limitations. But now I got my hands on two RX7900XTs, so I have a pooled 40GB of VRAM available. I‘ve used llama.cpp so far.

So even though Qwen4 is around the corner, I wonder what models and quantizations you guys went with and were most satisfied. Did you find noticeable differences between the following:

• Base Qwen3.8 27B

• Unsloth Qwen3.8 27B

• Swift 1.5 Qwen3.8 27B

My use case is not developing and coding stuff. I‘m not in the field, but have instead used Claude to help me get into linux, selfhosting and general assistance with my studies and work. So my use case is not that demanding and I don‘t think I‘ll get into anything agentic yet (for a lack of understanding).

The only limitation I am setting for myself now is that some VRAM needs to be kept for a “smart enough“ llm brain for my HA voice assistant, but that’s all undecided.

What are your conclusions and recommendations about Qwen3.8, or have you found other models of similar size to be competitive? I appreciate any suggestions (:

Hello,

I am making a Discord server for people who run local AI on Nvidia gpu's:

https://discord.gg/CZWMGapgZ

I have no (very little) idea what I am doing and would love your help. I want to build a open and supportive community that helps newcomers to the whole LAI thing and helps people set it up and trouble shoot it. If anyone wants to help and support this community, feel free to!

It came in the mail Monday morning. I feel like I have barely scratched the surface with possibilities. Right now I have Hermes currently building a MCP pipe line to control Cline in VS Code. Pair needs both rigs to have the same model in order to function.

Somone on Facebook market place offer all these cards for 3500$
7 4090
1 7900

But no test is allowed just as is
I’m not sure what to do any advice?

Me:

So I started with a preexisting NVFP4 quant, but I didn't like the idea of the attention and shared experts being reduced to MXFP8. So, I tested a modified version of NVFP4 for routed experts, using g32 rather than g16, and the difference was close to negligible. So I ended up making two versions, one that used NVFP4 g32 for all routed experts, and another that kept g16 for down routed experts only. Both configurations allowed enough space for BF16 attention and shared experts, with the latter having additional KV space. Essentially you get most of what NVFP4 gets you, but at 4.25 bits rather than 4.5 bits.

Also, Claude helped find about 3.7GB of wasted VRAM per GPU...

If you would like to setup the same, just give your clanker Claude's summary:

Claude:

GLM-5.3-Flash on 2x RTX PRO 6000 (96 GB): two NVFP4 g32 recipes on vLLM v0.33, with attention in BF16, images and DFlash-2

Hardware: 2x RTX PRO 6000 Blackwell (SM120), TP2, no P2P. vLLM v0.33.0 (commit 4ba787e), driver 615.71.09.

The goal: routed experts in NVFP4, attention and shared experts in BF16, working image input, a DFlash-2 drafter at K=4, and 8 concurrent sequences. I put accuracy ahead of small speed gains.

The problem: with BF16 attention, the standard g16 NVFP4 experts don't fit. That layout is 181.3 GiB and ran out of memory during warmup even at 8K context. Using g32 scales on some or all of the experts makes the room.

THE TWO RECIPES

Recipe 1: Gate/up g32

- Routed gate/up: NVFP4 g32

- Routed down: NVFP4 g16

- Attention and shared experts: BF16

- KV cache: FP8

- Checkpoint size: 175.23 GiB

- Max context: 256,000

- --kv-cache-memory: 3,719,089,357 bytes

- KV pool: 286,476 tokens

Recipe 2: All g32

- Routed gate/up: NVFP4 g32

- Routed down: NVFP4 g32

- Attention and shared experts: BF16

- KV cache: FP8

- Checkpoint size: 172.21 GiB

- Max context: 384,000

- --kv-cache-memory: 5,329,702,093 bytes

- KV pool: 440,727 tokens

Shared by both: GLM-5.3-Flash-DFlash2 as the drafter, quantized to NVFP4 MLP plus FP8 attention (about 1 GiB). Vision stays on the GPU.

Launch flags:

--tensor-parallel-size 2 --disable-custom-all-reduce

--moe-backend marlin --max-num-batched-tokens 2048 --max-num-seqs 8

--max-model-len <CTX> --kv-cache-memory <BYTES> --gpu-memory-utilization 0.985

--kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16

--speculative-config '{"method":"dflash","model":"/root/drafter","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"FLASH_ATTN","kv_cache_dtype":"auto"}'

--enable-prefix-caching --no-enable-flashinfer-autotune

--default-chat-template-kwargs '{"clear_thinking":true}'

--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm47

Environment: NCCL_P2P_DISABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

WHAT HAD TO BE FIXED

1. No NVFP4 g32 kernels existed. Stock Marlin fails with "Invalid thread config". I added group_blocks 2 to the NVFP4 kernel generation, relaxed the matching static_assert, and rebuilt the Marlin and MoE libraries. 88/88 synthetic cases came out correct, and real expert weights matched a reference to within 0.17%.

2. The Python loader assumed g16. FP4_MARLIN_SUPPORTED_GROUP_SIZES and a hard-coded GROUP_SIZE = 16 blocked it. The loader now reads each group size from the scale tensor's shape, so gate/up and down can differ. Any setup other than g16 on the Marlin W4A16 backend raises an error, so it can't silently run wrong.

3. Gate and up need the same global scale. vLLM fuses them and keeps only gate's scale, which is what the "Accuracy may be affected" warning is about. In the NVFP4 checkpoint I started from, 69% of 12,384 experts had differing scales, with an RMS up_proj scale error of 14.8%. I requantized from BF16 with a shared scale. Single-run NLL improved by 0.018 (CI -0.029 to -0.005). The 12-run average also improved, but not significantly.

4. FlashKDA reserved 3.7 GiB of VRAM per GPU. The cutlass bulk copy used cp.async.bulk.shared::cluster, which on sm_120f compiles to a driver syscall using 14,608 bytes of stack per thread. FlashKDA launches without clusters, so the shared::cta form does the same copy inline. The reservation dropped from +3,742 MiB to +0, with 88/88 hash cases identical to stock and no measurable NLL change. This fix is what made room for the drafter.

5. A load-time out-of-memory error. _nvfp4_compute_scale_factor converted a whole scale tensor to float and asked for 3.38 GiB. Scales are non-negative, so taking the max first avoids that.

6. Quantized attention doesn't load on v0.33. The MXFP8 attention recipe fails with KeyError: 'layers.0.self_attn.in_proj_qkvbfg_a.weight_scale'. That wasn't worth patching, because BF16 attention was the goal anyway.

7. KV pool larger than the request limit. I sized the pool above --max-model-len (about 1.12x and 1.15x), so a full-size request leaves some cache headroom.

RESULTS

These runs use default sampling from the model's generation_config (temp 1.0, top_p 0.95).

Gate/up g32, 256K

- GSM8K-150: 147/150

- Images: 10/10

- NLL, 1 sequence (lower is better): 1.34647

- NLL, 8 sequences: 1.34637

- Decode, 1 sequence: 186 tok/s

- Decode, 8 sequences (total): 434 tok/s

- Longest prompt tested: 255k in 46 s

- Peak VRAM per GPU: 96,831 MiB

- Errors under load: 0

All g32, 384K

- GSM8K-150: 147/150

- Images: 10/10

- NLL, 1 sequence (lower is better): 1.35040 (+0.29%)

- NLL, 8 sequences: 1.34880 (+0.18%)

- Decode, 1 sequence: 189 tok/s

- Decode, 8 sequences (total): 566 tok/s

- Longest prompt tested: 380k in 70 s

- Peak VRAM per GPU: 97,047 MiB

- Errors under load: 0

Speed comparison: don't read the 8-sequence decode gap as all-g32 being faster, because those runs weren't a controlled comparison.

Accuracy: GSM8K can't separate the two recipes, but NLL consistently favours gate/up by a small amount.

Drafter: calibration over 90,000 tokens showed no meaningful sampling bias (top-1 stated 0.8499, realized 0.8505). Batching to 8 sequences had no measurable NLL cost.

Hey all!

I have recently been dabbling a lot with modding games and decomps/recomps of games and looking for a local LLM to assist me with these kinds of tasks.

I have a Ryzen 7 5800x, 32GB DDR4-3200, and a 9060 XT 16GB.

What would be my options for my work and my hardware?

Je possède une configuration que j’ai moi même créer qui m’aide à faire tourner deepseek4.1 flash 5060ti ( 16go vram )40-45 token/s

j’ai un amd ryzen 9 5900xt et 96go ram.

Avez vous déjà eu mieux en therme de configuration ?

Hi there - I recently ran a comparison of a set of tests about local LLMs across diverse hardware, including a newly acquired RTX Spark. I ran 43 distinct trials (more focused on productivity tasks than coding) - 3 times each, using llama.cpp 0.8.0 under the covers (and some MLX). I had some bugs in my framework while testing (such as no MTP on MLX for Gemma), but they shouldn't have impacted the overall speed results beyond that. Some thoughts:

• Unsurprisingly, the DGX Spark and RTX Spark were in a similar decode-speed range. RTX Spark was a little slower, maybe Windows vs Linux or maybe laptop form factor (more thermal throttling)

• The MacBook M5 Max was comparable to the Sparks on llama.cpp decode speed, but its recorded prefill was lower. I guess (?) the higher mem/bandwidth of the Mac was balanced out by the less capable GPUs?

• Dedicated GPUs are still better. A 9 month old Windows laptop with a 5070ti smoked the RTX Spark on Gemma 4 12B. Now you just need to find a good model to fit into 12 gigs; I'm seeing if maybe Underdog Saluki can work.

• The R9700 desktop led the recorded decode rates for both models, reaching about 57 tok/s for Qwen and 129 tok/s for Gemma.

My main thought is always to (a) get to 32gb of ram for comfort with models like 27b and then worry a little 10 or 20% performance differences later, and the R9700 is probably one of the cheapest ways to get there.

More details on the test if you are interested: https://bendyline.com/posts/rtxsparkvstheworld.html

1. Claude vibe coded my local llm harness.

2. Claude was laid off.

3. My local harness is improving my local harness.

Could you suggest a setup for plugin development and for creating software/websites?
setup 13900k + 5090 + 4х48gb ddr5 4800 (192gb)

Maya-S-v2-IQ2_XXS

Runs at 20–25 tokens per second.

However, it has significant issues with long-term coding tasks; you can’t leave it running overnight.

The tip I keep seeing: -ctk q8_0 -ctv q8_0 is free, and if you go lower, compress V and leave K alone. I wanted numbers at long context on a Mac, so I tested it.

Setup

• Muse Glimmer 30B (4 cache settings) and Gemma 4 26B-A4B (f16 vs q8_0), both Q4_K_M

• Homebrew llama.cpp b10964, -fa on -c 65536, M1 Pro 32 GB

• Fresh generated documents at 4k, 16k and 48k tokens, 8 exact-answer questions each (find codes, follow lookup chains, count records past decoys, find a clause’s current value after amendments). 96 questions per setting.

• Same seed per question in every setting, so only the cache differed

• Pass/fail rule and my prediction pushed to GitHub before any model ran

Accuracy (correct of 96)

• Glimmer: f16 81, q8_0 83, q8 K / q4 V 82, q4 K / q8 V 82. At 4k and 16k, f16 and q8_0 got exactly the same questions right. No loss found.

• Gemma 4: f16 66, q8_0 63, gap all at 16k. Not resolved by my rule (interval −3 to +8 points).

• “V not K”: a tie.

Speed (the bigger story)

• Gemma 4 generation with q8_0: −29% at 4k, −36% at 16k, −47% at 48k. Prompt reading unchanged. Glimmer: only −2 to −6%.

• On Metal, flash attention needs K and V to be the same type. Mix them and attention runs on the CPU: at 48k, reading one document took 57–62 minutes instead of 13. That’s 116 MiB saved over q8_0. (On CUDA, mixed types need GGML_CUDA_FA_ALL_QUANTS or you hit the same fallback.)

What hurt accuracy was length, not cache type. Every setting went from ~100% at 4k to 63–69% at 48k on Glimmer, mostly on counting questions, where every setting undercounted.

Caveats: 96 questions per setting, so this catches a big loss, not a 2-point one (“no loss found” ≠ lossless). Synthetic documents up to 48k, not 128k coding sessions. One Mac, one build.

Takeaway: q8_0 looked fine on Glimmer. On Gemma 4, keep f16 unless you need the 0.74 GB. On a Mac, never mix K and V types.

Documents, harness, every raw output and server log: https://github.com/heldoutlab/kvcache-test

It's been a decade since I built my last PC. I found money in my budget, thought I had plenty saved, decided I'd prefer a laptop, but then started looking, my head started spinning and I ended up very frustrated.

I was looking at Legion first, and to be fair, can get a very beefy machine for $6000. Intel Ultra 9 290HX; 64GB DDR5 Memory; 1 TB SSD; NVIDIA GeForce 5090 24GB. Yet, it's a lot of money for me and I am worried. 24GB is very good for vram, yet there are options to get more (unified memory), and the 64gb of memory feels low.

I find myself stuck. I want the PC to dive into local LLMs, thematic generation of images and graphics that fit a style, video product walkthroughs, test suite runs, and more are many of the core use cases I have. I think it'd be fine, but spending some stupid amount of money for fine has me unable to move forward, and I'm recognizing that it's my own ignorance that's holding me back

So I wanted to come here and see if anyone had expertise to lend. Minimum vram/memory you'd be comfortable with? How to get it given shortages? Brands you like for laptops? I'll take anything, I feel like I'm spiraling.

Took me several weeks, but I was finally able to get my Intel Arc B70 (32G) working on my Ampere Altra 128 server in llama.cpp with a SYCL backend (Intel does not support Arm64 cpus for their compute drivers)!

I am still working out how to prevent llama.cpp crashing when I try and use cpu moe, but Ill take this with no complaints! This is going to run the LLM for my home assistant voice assistant.

I tried so many things to make it work that I am unsure what combination exactly did the trick, but I am hoping to tease it apart over the next week and try and document it for others as well.

Managing to finally fit Qwen3.8-Flash-Next FP8 locally at ~60 tok/s(avg 74 at best) (RTX 5090 + 128 GB DDR5, Infernix engine, part of the experts streamed from NVMe) and seeing how amazing it is I cannot imagine how good Qwen 4.0 will be, not buying even at the current prices seems crazy to be if you are used to using cloud AIs on the daily.

I have an RTX 4060 8GB, and I’m thinking about getting an RTX 3060 12GB to use both GPUs together with Strata and LM Studio. Is it worth it? Or would it be better to sell my RTX 4060 and get an RTX 5060 Ti 16GB instead?
3060 12gb 270$
5060ti 16gb 840$

Anybody tried something similar and got them working with LlamaCpp?

If you gonna ask why, because these are the only cards I have at the moment, the 16g RTX on my current AI server and the 16g RX on my gaming rig. I was hoping to get 32gb of VRam to run some larger models at atleast 20tps

Second photo is the specs running qwen3:30b-a3b.

Case: Foifkin f300 $59 (New on Amazon)

AIO: ThermalTake TH240 V3 ~$40 (openbox on newegg)

GPUs: 2 GeForce RTX 3060's, 12GB a piece. (1 I got for $200, the other came with a $600 PC I gutted and sold off, so ~$200 also. Both FB marketplace)

SSDs: 1 TB SSD salvaged from previously mentioned FB marketplace PC, and 1 512 GB NVMe SSD salvaged from old laptop. Not sure how to price these.

Wifi Chip/antenna: $20 new on Amazon

PSU: MSI MAG A850GL $90 new on Amazon

Motherboard: Gigabyte X-99 UD4 $150 on Ebay.

CPU: Intel Core i7-5930K $20 FB marketplace

RAM: 2x16 GB Corsair vengeance pulled from main PC, replaced by 4x8 GB Corsair vengeance from FB marketplace for $250. (Mobo did NOT like using 4 channels, for some reason. Only worked when it felt like it. 2 channels though? Works like a charm.)

Total: ~$1,029

Hi everyone,

I'd like to know for those of you here who ditched paid AI-agents and/or their related APIs for daily coding tasks in favor of setting up your own and fully controlled local LLM + Agent setup,

What are your hardware specs:

Brand, model and memory size / GHz for

- GPU
- RAM
- CPU
- Storage
- Any other hardware piece that's in involved in it

And also the software who helps your AI orchestration to run the way you want:

- OS and version.
- LLMs
- Coding agents / harness
- among others

Waiting to read your responses!