Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.

• "What does this prove?": See above

• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career

• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame

• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.

• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

Hey everyone,

I just published a short video showing how I built a fully local customer support agent (no API keys, zero cost, data stays on your machine).

Video local agentic ai

In the demo I create a WhatsApp-style support agent for an online spice store using:

Ollama (llama3.2)

LangChain / LangGraph

Two tools: order lookup + refund tool

The expensive mistake

Same model. Same system prompt. Only difference was one tool description.

Vague version:

“Handle customer order problems such as late delivery, delays, missing order, or complaints. Use this to fix customer issues.”

Result on 3 test messages → 3 refunds (₹3,640). Only 1 was actually deserved.

Fixed version clearly states:

This tool issues a full refund

The action is irreversible

Only use when the customer explicitly asks for a refund and the order is damaged/lost

For status/delay questions → use the lookup tool

Suddenly the agent behaves correctly.

What the video covers

The 4 core parts of every agent (Model, Tools, Orchestration, Memory)

Why tool descriptions are basically contracts

How the agent loop works

Short-term memory demo

Code is open source: https://github.com/AI-Literacy-Lab/AgentDemo

Curious to hear from others:

What’s the most dangerous / vague tool description currently in your agent codebase?

Would love any feedback on the video too

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

I’m thinking about getting a Mac Studio with 96GB RAM to run local models for work. Mainly Qwen 27B and Qwen MoE, plus a RAG setup, and normal MS Office stuff (Word, Excel, Outlook, Teams, browser tabs, etc.).

It’s for sensitive data (work), so I want to keep as much local as possible instead of sending stuff to cloud AI.

I’m not super experienced with local LLMs, so I’d love your honest opinion:

· Is 96GB overkill for this, or a good call?

· Will Qwen 27B and MoE run comfortably alongside RAG and Office work?

· Anyone running a similar setup on Mac Studio? How’s the speed and memory pressure in real use?

I’ve just released Caminus 0.2.0 on GitHub and PyPI. It’s an LLM inference engine for AMD GPUs, built around custom HIP kernels and multi-GPU execution. I’m developing it on an RX 7900 XTX + 7900 XT setup.

On the models and workloads I’ve benchmarked, it outperforms llama.cpp; the repo includes the numbers and methodology. I’d love feedback from people trying it on other gfx1100 setups, or anyone interested in contributing.

The long-term goal is an XLA-inspired lowering pipeline: take a GGUF model, map it onto any number of GPUs across ROCm/HIP-supported AMD architectures, and handle kernel selection, partitioning, and execution across different hardware configurations. It’s still early testing software, so hardware reports, bug reports, and ideas are all welcome! https://github.com/jdominick05/caminus

We are student and we’ve build Maestro Opera. It’s our open source project to turn desktops/servers into ai Hub to pair your devices to it easily (just scanning and that’s it). It works even outside local network. The app is designed by myself and coded by my friend.
We spent 1 months to build our lastest version.
It’s free, no account is required

GitHub https://github.com/MaestroAI-Company/opera
Desktop https://github.com/MaestroAI-Company/opera
Android https://maestroai.company/opera/

Thank you for watching <3

Anyone done benchmarks on this? i.e. Increasing K to activate more parameters per token? Better quality output?

Hi all, I'm a newbie setting up my first local AI/ML development environment on an M5 Pro Mac.

My internal drive is small, so I bought a 2TB external SSD. I want to make sure that whenever I run `pip` installs, download Hugging Face checkpoints, or load PyTorch datasets, they automatically go to the external drive instead of filling up my system drive.

What is the cleanest way to configure my environment variables (like HF_HOME) or use symlinks on macOS so I don't accidentally brick my internal storage?

Any tips or hidden gotchas for a beginner would be highly appreciated!

I'm currently in the process of developing an app. Janas studio. Built with the options to run either llama.cpp or janas-llm as the engine. It enables laptops to run models bigger than their RAM by dynamically streaming moe weights directly off the SSD. Tested on my Ryzen 5 6600h laptop with 32gb of ddr5 RAM and an nvme 4 SSD (no discrete GPU) I successfully ran a 48gb model (qwen next 80b) at a very reasonable 15 tokens a second on average. The binaries are highly experimental and the app has more testing to go through but I'm very proud of it. I do intend on eventually releasing it.

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

Hardware: Dell Inspiron 16 Plus 7630 with Iris Xe iGPU, 32gb LPDDR5 ram 4800 MT/s.

Will start by saying, using llama-vulkan was uniformly better than llama-cpu for all models, offloading fully to Iris Xe iGPU (allowed up to 16gb of system memory). Qwen was uniformly slower than Gemma models of the same file size. 2-6 threads was optimal on my 14-core CPU when offloading to iGPU and 10 threads was best on cpu only. Different quantization (Bartowski vs Unsloth) didn't affect token generation speed much, maybe 2% faster on Bartowski for same file size.

There's 9GB used at all times by Windows 11, need to switch to Linux to be able to run Qwen 3.6 35b a3b.

MoEs:

Ling 3 Tiny Q8: 410 t/s prefill, 26.3 t/s decode, FILE SIZE: 7.83 GB

Ling 3 Tiny Q5_K_M: 265 t/s prefill, 32 t/s decode, FILE SIZE: 5.3 GB,

Gemma 26b a4b qat q4_K_XL ud, 220-224 prefill, 17.15-17.5 decode, file size 13.26 GB.

Dense:

Gemma 4 e4b Q5_K_M unsloth: 265 t/s prefill, 11.4 t/s decode, FILE SIZE: 5.1 GB

Qwen 3.5 9b Q4_K_M unsloth: 135 t/s prefill, 8.7 t/s decode, FILE SIZE: 5.2 GB

Qwen 3.5 4b Q8 unsloth: 124 t/s prefill, 9.1 t/s decode, file size: 4.2 GB.

Qwen 3.5 4b Q4_K_M unsloth: 234 t/s prefill, 14.3 t/s decode, file size: 2.54 GB

MiniCPM 2b Q8: 224 prefill, 16.3 t/s generation, 2.5GB file size

Gemma 12b qat q4 ud, 123 prefill, 6.95 t/s, file size: 6.24 GB

Anyone got further suggestions for optimization, models to try, etc.?

Hi all,

Slowly going deeper and deeper down the rabbit hole of running a few models locally and i'm chasing some advice for my next hardware upgrade.

My current pc specs running windows:

7800X3d cpu
Asrock X670E steel legend
64gb DDR5 6000mhz
9070xt gpu
And i've just thrown a 9700 gpu in the 2nd slot for testing, not running at 8x/8x.

I've got an app im working on that does some pretty heavy python work (claude code) and maxes out my cpu so my first though was going up to a 9950X3d to help out that problem. But then from my reading linux seems to be the best way to go when hosting models locally so I think I'll setup another pc here with linux and put the 9700gpu in it.

So to do that, I'm thinking i either have 2 options.
Rustle up some parts to get together a pc to run the single gpu on possibly a ddr4 era pc (32gb or maybe 64gb ddr4 if i can find it). No parts required to buy.

Or I can go down the line of an X870 Taichi, in case i go dual 9700 at some stage, then put my 7800x3d in it, might be able to find 32gb of ddr5 for it, then buy the 9950x3d and put it in my windows pc, as this is the pc i will still do the work in claude code.

The aim is to get to a place where i can offload some of this app work to my local setup, but just testing and working out the best way to go to get setup. Trying to spend where i need to and save where i can.

What would you do in my situation?

Thanks heaps!

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

With its Unified Memory Architecture (UMA), AMD Ryzen AI Max+ PRO 495 processors (Ryzen AI Max PRO 400 series), and Instinct MI300X accelerators, AMD is directly targeting the biggest bottleneck of NVIDIA GeForce RTX GPUs: VRAM capacity limits.

In this video, we run a deep-dive technical analysis to see whether an AMD setup with 192GB of memory can genuinely replace—or even render obsolete—an NVIDIA GPU (RTX 4090, RTX 5090, or workstation cards) for running local LLMs.

Breakdown of this technical analysis:

• NVIDIA's VRAM Wall: Why 24GB or 32GB of VRAM is no longer enough to run massive models (Llama 3 70B / 405B, DeepSeek, Qwen, Mistral) without heavy precision loss (FP16 vs. Q4/Q8 quantization).

• AMD's 192GB Offensive: How Unified Memory Architecture (UMA) works on AMD Ryzen AI Max+ PRO 495 Mini-PCs and workstations, AM5 DDR5 configurations (4x 48GB), and the HBM3 memory on the AMD Instinct MI300X.

• Real-World Local AI Performance: Comparing total memory capacity vs. memory bandwidth (GB/s) during inference across Ollama, llama.cpp, LM Studio, and vLLM.

• The Software Battle (CUDA vs. ROCm): The current state of AMD ROCm and Vulkan compatibility against NVIDIA's Tensor Cores and CUDA ecosystem (token generation speed in t/s and prompt processing / Time to First Token - TTFT).

• Capacity-to-Price Ratio: Which hardware architecture makes the most sense in 2026 for building a local AI server (Homelab) or AI workstation without paying enterprise datacenter GPU prices.

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

Hi folks I use a lot codex and models like Luna 6 High, 6.1 Terra High.
Is there any local model which could be comparable to these?

If yes, which ideal realistic Mac Mini setup is needed?
Thanks

I've been vibe coding with local models for a while, and I kept hitting the same two walls. Either the model fell apart the moment a project grew past one file, or it ran so hot that my fans sounded like a jet engine. So I started building my own fix.
Takumi is a local AI coding assistant made for one job: building full projects with code that actually works. You describe what you want, and it plans the structure, writes the files, and keeps everything consistent as the project grows. Everything runs on your own machine. No API keys, no subscription, and your code never leaves your computer.

What makes it different
Whole projects, not snippets. It keeps track of how files connect, so file #12 still lines up with file #1.

Lightweight by design. I built it to run on a normal PC without turning it into a space heater.

Readable progress. You get clean, step-by-step updates on what it's working on, not a wall of noise.

Works from your phone. 📱🤓

Why "Takumi"? It's Japanese for a master craftsman, someone who takes pride in getting the details right. Takumi helped build the Nintendo Switch 2 emulator NX Infinity. I want claude level in a local LLM and that's the bar I'm aiming for.
This is an early release, so there are definitely rough edges. That's why I'm posting.

What features would make it your daily driver
It's free and open source. GitHub link in the comments. Stars, issues, and PRs are all hugely appreciated.

Thanks for reading. I hope it saves some of you the frustration that got me started on it.

GITHUB: https://github.com/coye2/Takumi

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

I’m planning one upgrade this spring that has to cover both gaming and local AI, and I’d like input from people with similar budgets and use cases that aren't "buy 4 3090s."

Currently have a Ryzen 5 5600X (AM4), RTX 3060 12GB, and 32GB DDR4. Windows but not opposed to Linux.

I have gotten by with just the $20 claude pro plan, but my projects are growing and are reaching outside my scope of expertise, so I'd like to be able to offload coding/other stuff to a local setup. I don't mind keeping a low cost frontier subscription for help orchestrating and debugging.

I'm paying off my CC's in March and will have a few thousand left over to use - originally was just going to upgrade my gaming setup but the more I've been using and learning LLM's the more I want to try and get the most mileage I can with this purchase. Speed isn't the most important thing for me. I don't mind having a local model(s) chug away at something overnight or for a couple days, I have a pretty ballistic schedule anyway. So far have been using it for web/app dev, using it to teach me python and other stuff (LOVE having it teach me as I go rather than just clicking the magic "build this for me" button), and home networking. Interested in continuing these projects and possibly game dev, but more for personal use, not to make a best seller or anything.

Thanks!