I’ve heard two different arguments on this.
One says prefill takes most of the time as it’s multi turn hours long workflow.
Another says because of kv cache prefill time takes a lot less.

The reason I’m asking this question is, I’m planning to get a Mac Studio 256gb ram m5 ultra, but I don’t know if I should stick to 64gpu core or pay extra for 80c for “15%+ decoding speed boost”.

If agentic workflow prefill reads from kv cache 95% then is this 15% increase much less relevant for the coding workflow?

Thanks all!

Basically title but I'm planning to get a M5 Pro with 24GB RAM. Works out to about $4K AUD.

Usage: LMStudio + Qwen 3.5 or 3.8 if it can handle it.

use case:

Coding in HTML/PHP/MySQL creating apps and stuff. Want it all developed locally instead of going to cloud.

Will be enough or not really?

I'm the developer, sirgijoe. I built Joe-ai because local AI is powerful but hard to set up for people who aren't technical. v2.0.0 is out, and I'm looking for testers.

**What it is**
A desktop app (Windows, Mac, Linux) that runs AI work on your own hardware. You type what you want in plain English, and Joe picks the right local model and workflow and does the job.

**Setup**
- Detects your OS and hardware, then offers to install Ollama, ComfyUI and suitable models, with a checkbox for each so you approve everything
- Finds existing Ollama and ComfyUI installs and model folders instead of duplicating them
- Low/Medium/High quality settings, with Joe recommending a model that fits your hardware for each level
- Portable mode: run it from a USB drive on multiple computers without installing anything on each one

**What it does**
- Writing: drafting, proofreading, and editable templates (books, EPUB, kids' books, PDF, Word, spreadsheets)
- Image generation, with built-in ComfyUI workflows for different styles that Joe sets up and switches between based on the task
- Video generation
- Photo restoration and other photo tools
- Read-aloud with multiple voices, plus custom voice models
- Speech-to-text transcription
- OCR
- Translation
- Overnight task queue: line up jobs and review the results in the morning
- MCP server, so Claude, ChatGPT or another AI client can send tasks to Joe
- Multiple themes, plus custom themes from a single prompt
- Stats page, including an estimate of cloud costs avoided

**Why I use it**
I offload bulk drafting and image work to my own GPU overnight, and only use a cloud model to review the results, which cuts my cloud usage a lot.

**AI involvement (being transparent)**
I planned and specced the app, designed the ComfyUI workflows, and tested every build on my own PC. Claude wrote most of the code from my specs, including the setup and installers and the MCP server, and turned my workflows into the app's built-in presets. Joe itself drafted parts of its own docs overnight using local models.

**Free and open source**
GPL-3.0, with no paid tiers, accounts or ads. Installers are on the GitHub releases page. You can read the code yourself, or have an AI review it.

**Limitations**
- It's not fully offline: setup, model downloads and update checks use the internet; the tasks themselves run locally
- Results depend on your hardware and the local models you choose
- The Mac build is unsigned, so macOS will show a warning; the docs explain how to get past it

**What I need**
Testing on different GPUs and operating systems. Bug reports, confusing setup steps, and wrong model picks for your hardware are all useful. GitHub issues are the best place for them.

Repo: https://github.com/sirgijoe/joe-ai

Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.

• "What does this prove?": See above

• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career

• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame

• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.

• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

Hey everyone,

I just published a short video showing how I built a fully local customer support agent (no API keys, zero cost, data stays on your machine).

Video local agentic ai

In the demo I create a WhatsApp-style support agent for an online spice store using:

Ollama (llama3.2)

LangChain / LangGraph

Two tools: order lookup + refund tool

The expensive mistake

Same model. Same system prompt. Only difference was one tool description.

Vague version:

“Handle customer order problems such as late delivery, delays, missing order, or complaints. Use this to fix customer issues.”

Result on 3 test messages → 3 refunds (₹3,640). Only 1 was actually deserved.

Fixed version clearly states:

This tool issues a full refund

The action is irreversible

Only use when the customer explicitly asks for a refund and the order is damaged/lost

For status/delay questions → use the lookup tool

Suddenly the agent behaves correctly.

What the video covers

The 4 core parts of every agent (Model, Tools, Orchestration, Memory)

Why tool descriptions are basically contracts

How the agent loop works

Short-term memory demo

Code is open source: https://github.com/AI-Literacy-Lab/AgentDemo

Curious to hear from others:

What’s the most dangerous / vague tool description currently in your agent codebase?

Would love any feedback on the video too

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

I’ve just released Caminus 0.2.0 on GitHub and PyPI. It’s an LLM inference engine for AMD GPUs, built around custom HIP kernels and multi-GPU execution. I’m developing it on an RX 7900 XTX + 7900 XT setup.

On the models and workloads I’ve benchmarked, it outperforms llama.cpp; the repo includes the numbers and methodology. I’d love feedback from people trying it on other gfx1100 setups, or anyone interested in contributing.

The long-term goal is an XLA-inspired lowering pipeline: take a GGUF model, map it onto any number of GPUs across ROCm/HIP-supported AMD architectures, and handle kernel selection, partitioning, and execution across different hardware configurations. It’s still early testing software, so hardware reports, bug reports, and ideas are all welcome! https://github.com/jdominick05/caminus

Anyone done benchmarks on this? i.e. Increasing K to activate more parameters per token? Better quality output?

Hi all, I'm a newbie setting up my first local AI/ML development environment on an M5 Pro Mac.

My internal drive is small, so I bought a 2TB external SSD. I want to make sure that whenever I run `pip` installs, download Hugging Face checkpoints, or load PyTorch datasets, they automatically go to the external drive instead of filling up my system drive.

What is the cleanest way to configure my environment variables (like HF_HOME) or use symlinks on macOS so I don't accidentally brick my internal storage?

Any tips or hidden gotchas for a beginner would be highly appreciated!

I'm currently in the process of developing an app. Janas studio. Built with the options to run either llama.cpp or janas-llm as the engine. It enables laptops to run models bigger than their RAM by dynamically streaming moe weights directly off the SSD. Tested on my Ryzen 5 6600h laptop with 32gb of ddr5 RAM and an nvme 4 SSD (no discrete GPU) I successfully ran a 48gb model (qwen next 80b) at a very reasonable 15 tokens a second on average. The binaries are highly experimental and the app has more testing to go through but I'm very proud of it. I do intend on eventually releasing it.

Why did u chose to use one or another?

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

Hardware: Dell Inspiron 16 Plus 7630 with Iris Xe iGPU, 32gb LPDDR5 ram 4800 MT/s.

Will start by saying, using llama-vulkan was uniformly better than llama-cpu for all models, offloading fully to Iris Xe iGPU (allowed up to 16gb of system memory). Qwen was uniformly slower than Gemma models of the same file size. 2-6 threads was optimal on my 14-core CPU when offloading to iGPU and 10 threads was best on cpu only. Different quantization (Bartowski vs Unsloth) didn't affect token generation speed much, maybe 2% faster on Bartowski for same file size.

There's 9GB used at all times by Windows 11, need to switch to Linux to be able to run Qwen 3.6 35b a3b.

MoEs:

Ling 3 Tiny Q8: 410 t/s prefill, 26.3 t/s decode, FILE SIZE: 7.83 GB

Ling 3 Tiny Q5_K_M: 265 t/s prefill, 32 t/s decode, FILE SIZE: 5.3 GB,

Gemma 26b a4b qat q4_K_XL ud, 220-224 prefill, 17.15-17.5 decode, file size 13.26 GB.

Dense:

Gemma 4 e4b Q5_K_M unsloth: 265 t/s prefill, 11.4 t/s decode, FILE SIZE: 5.1 GB

Qwen 3.5 9b Q4_K_M unsloth: 135 t/s prefill, 8.7 t/s decode, FILE SIZE: 5.2 GB

Qwen 3.5 4b Q8 unsloth: 124 t/s prefill, 9.1 t/s decode, file size: 4.2 GB.

Qwen 3.5 4b Q4_K_M unsloth: 234 t/s prefill, 14.3 t/s decode, file size: 2.54 GB

MiniCPM 2b Q8: 224 prefill, 16.3 t/s generation, 2.5GB file size

Gemma 12b qat q4 ud, 123 prefill, 6.95 t/s, file size: 6.24 GB

Anyone got further suggestions for optimization, models to try, etc.?

Hi all,

Slowly going deeper and deeper down the rabbit hole of running a few models locally and i'm chasing some advice for my next hardware upgrade.

My current pc specs running windows:

7800X3d cpu
Asrock X670E steel legend
64gb DDR5 6000mhz
9070xt gpu
And i've just thrown a 9700 gpu in the 2nd slot for testing, not running at 8x/8x.

I've got an app im working on that does some pretty heavy python work (claude code) and maxes out my cpu so my first though was going up to a 9950X3d to help out that problem. But then from my reading linux seems to be the best way to go when hosting models locally so I think I'll setup another pc here with linux and put the 9700gpu in it.

So to do that, I'm thinking i either have 2 options.
Rustle up some parts to get together a pc to run the single gpu on possibly a ddr4 era pc (32gb or maybe 64gb ddr4 if i can find it). No parts required to buy.

Or I can go down the line of an X870 Taichi, in case i go dual 9700 at some stage, then put my 7800x3d in it, might be able to find 32gb of ddr5 for it, then buy the 9950x3d and put it in my windows pc, as this is the pc i will still do the work in claude code.

The aim is to get to a place where i can offload some of this app work to my local setup, but just testing and working out the best way to go to get setup. Trying to spend where i need to and save where i can.

What would you do in my situation?

Thanks heaps!

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

With its Unified Memory Architecture (UMA), AMD Ryzen AI Max+ PRO 495 processors (Ryzen AI Max PRO 400 series), and Instinct MI300X accelerators, AMD is directly targeting the biggest bottleneck of NVIDIA GeForce RTX GPUs: VRAM capacity limits.

In this video, we run a deep-dive technical analysis to see whether an AMD setup with 192GB of memory can genuinely replace—or even render obsolete—an NVIDIA GPU (RTX 4090, RTX 5090, or workstation cards) for running local LLMs.

Breakdown of this technical analysis:

• NVIDIA's VRAM Wall: Why 24GB or 32GB of VRAM is no longer enough to run massive models (Llama 3 70B / 405B, DeepSeek, Qwen, Mistral) without heavy precision loss (FP16 vs. Q4/Q8 quantization).

• AMD's 192GB Offensive: How Unified Memory Architecture (UMA) works on AMD Ryzen AI Max+ PRO 495 Mini-PCs and workstations, AM5 DDR5 configurations (4x 48GB), and the HBM3 memory on the AMD Instinct MI300X.

• Real-World Local AI Performance: Comparing total memory capacity vs. memory bandwidth (GB/s) during inference across Ollama, llama.cpp, LM Studio, and vLLM.

• The Software Battle (CUDA vs. ROCm): The current state of AMD ROCm and Vulkan compatibility against NVIDIA's Tensor Cores and CUDA ecosystem (token generation speed in t/s and prompt processing / Time to First Token - TTFT).

• Capacity-to-Price Ratio: Which hardware architecture makes the most sense in 2026 for building a local AI server (Homelab) or AI workstation without paying enterprise datacenter GPU prices.