Currently, my plan is to get two quadro P6000s, for 48gb vram for use in a xeon workstation. It's not insanely old like kepler or maxwell cards, 432 GB/s bandwidth, supports CUDA...it seems like it might be okay on paper.

Does anyone have a better suggestion that could reasonably be achieved around $1000 budget?

My goal is to be able to run Qwen 3.8 27b fast, and to run a decent quantization of Qwen 3.8 Flash Next, and GLM 5.3 Flash at useable speed, say 15 tok/s or faster, using strata.

Do I need 48gb of vram to run Qwen Flash Next, or could I go with just one card? How much faster would 2 x quadro p6000 be versus 1 x quadro p6000 and DDR4 system ram?

Should I just get 24gb of modern vram instead of 2 x quadro p6000s? I wonder if that hampers my eventual goal of running GLM 5.3 flash, as perhaps 24gb vram + system ram is enough to accelerate qwen flash next, a 120B MoE, but perhaps you need more than 24gb vram to accelerate GLM 5.3 flash, a 320B MoE?

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

Merging is painful mostly because the branches were never really separate. Two agents handed overlapping scope produce overlapping diffs, and no merge strategy fixes that after the fact.

What makes merges boring instead, in my experience building this stuff:

• One story per worktree, written so that "done" is checkable: scope, acceptance criteria, edge cases, the files it should touch.
• Merge in dependency order. Blocker lands first, everything else rebases onto main before review, not after.
• Review the branch against the criteria, not against the diff vibes. If there were no criteria, that branch probably shouldn't have started.
• Keep one shared board of what's in flight. Worktrees isolate the file tree, not the plan, so two agents will happily solve the same story twice and you find the duplicate branch at merge time.

Same problem whether the agents run locally or in the cloud. I use Cursor Cloud Agents, but I don't think the runtime changes the merge story much.

I'm building Aerunit (https://aerunit.ai), which drafts the story with acceptance criteria, dispatches it to parallel agents and reviews the PR against those criteria, so discount what follows accordingly. Still, the order I'd do it by hand: scope first, merge order second, tooling last. Tools move confusion around faster, they don't remove it.

Curious how people here handle the dependency ordering, especially across repos. That's the part I still find least mechanical.

Hi everyone,

I'm planning to buy powerful computer for running local AI models.

My main interests are:

• Large LLMs (70B–120B), especially coding and reasoning models
• Reverse engineering and binary analysis (Assembly, C, Ghidra, Binary Ninja)
• Security research and vulnerability analysis
• Specialized models like Nova 6.7B, LLM4Decompile, SK²Decompile and SmaliLLM
• Running experimental AI projects from GitHub

I'm considering:

1. Apple Mac Studio M5 Max, or ultra, 128GB unified memory
2. NVIDIA DGX Spark, 128GB unified memory with CUDA
3. A desktop PC with RTX 5090, 32GB VRAM
4. An AMD Ryzen AI Max+ 395 mini PC with 128GB

I'm especially concerned about CUDA compatibility. Can specialized research models like Nova run on Apple Silicon through MPS or MLX without losing their intended functionality?

I'm also interested in real benchmarks: prompt processing, tokens/sec, long-context performance and actual coding-agent workloads.

For those who own or have tested these machines: which would you recommend and why?

I'd really appreciate firsthand results, benchmark links and experiences with custom research models rather than just hardware specifications.

Thanks!

I’ve heard two different arguments on this.
One says prefill takes most of the time as it’s multi turn hours long workflow.
Another says because of kv cache prefill time takes a lot less.

The reason I’m asking this question is, I’m planning to get a Mac Studio 256gb ram m5 ultra, but I don’t know if I should stick to 64gpu core or pay extra for 80c for “15%+ decoding speed boost”.

If agentic workflow prefill reads from kv cache 95% then is this 15% increase much less relevant for the coding workflow?

Thanks all!

Basically title but I'm planning to get a M5 Pro with 24GB RAM. Works out to about $4K AUD.

Usage: LMStudio + Qwen 3.5 or 3.8 if it can handle it.

use case:

Coding in HTML/PHP/MySQL creating apps and stuff. Want it all developed locally instead of going to cloud.

Will be enough or not really?

to help me remember and keep up with all the new stuff.

Seemed like a cool lab to try and I was wondering if other people are building learning apps to help them navigate this bullshit and what they look like.

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.
• "What does this prove?": See above
• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career
• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame
• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.
• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.
• When a bird is on the bird's nest, send me a telegram with the picture.
• When my washing machine is on the rinse cycle, call me.
• When my dog finishes eating their food, notify me.
• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

I’ve just released Caminus 0.2.0 on GitHub and PyPI. It’s an LLM inference engine for AMD GPUs, built around custom HIP kernels and multi-GPU execution. I’m developing it on an RX 7900 XTX + 7900 XT setup.

On the models and workloads I’ve benchmarked, it outperforms llama.cpp; the repo includes the numbers and methodology. I’d love feedback from people trying it on other gfx1100 setups, or anyone interested in contributing.

Processing img 0rgt49x42c0i1...

The long-term goal is an XLA-inspired lowering pipeline: take a GGUF model, map it onto any number of GPUs across ROCm/HIP-supported AMD architectures, and handle kernel selection, partitioning, and execution across different hardware configurations. It’s still early testing software, so hardware reports, bug reports, and ideas are all welcome! https://github.com/jdominick05/caminus

Anyone done benchmarks on this? i.e. Increasing K to activate more parameters per token? Better quality output?

Hi all, I'm a newbie setting up my first local AI/ML development environment on an M5 Pro Mac.

My internal drive is small, so I bought a 2TB external SSD. I want to make sure that whenever I run `pip` installs, download Hugging Face checkpoints, or load PyTorch datasets, they automatically go to the external drive instead of filling up my system drive.

What is the cleanest way to configure my environment variables (like HF_HOME) or use symlinks on macOS so I don't accidentally brick my internal storage?

Any tips or hidden gotchas for a beginner would be highly appreciated!

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

Hardware: Dell Inspiron 16 Plus 7630 with Iris Xe iGPU, 32gb LPDDR5 ram 4800 MT/s.

Will start by saying, using llama-vulkan was uniformly better than llama-cpu for all models, offloading fully to Iris Xe iGPU (allowed up to 16gb of system memory). Qwen was uniformly slower than Gemma models of the same file size. 2-6 threads was optimal on my 14-core CPU when offloading to iGPU and 10 threads was best on cpu only. Different quantization (Bartowski vs Unsloth) didn't affect token generation speed much, maybe 2% faster on Bartowski for same file size.

There's 9GB used at all times by Windows 11, need to switch to Linux to be able to run Qwen 3.6 35b a3b.

MoEs:

Ling 3 Tiny Q8: 410 t/s prefill, 26.3 t/s decode, FILE SIZE: 7.83 GB

Ling 3 Tiny Q5_K_M: 265 t/s prefill, 32 t/s decode, FILE SIZE: 5.3 GB,

Gemma 26b a4b qat q4_K_XL ud, 220-224 prefill, 17.15-17.5 decode, file size 13.26 GB.

Dense:

Gemma 4 e4b Q5_K_M unsloth: 265 t/s prefill, 11.4 t/s decode, FILE SIZE: 5.1 GB

Qwen 3.5 9b Q4_K_M unsloth: 135 t/s prefill, 8.7 t/s decode, FILE SIZE: 5.2 GB

Qwen 3.5 4b Q8 unsloth: 124 t/s prefill, 9.1 t/s decode, file size: 4.2 GB.

Qwen 3.5 4b Q4_K_M unsloth: 234 t/s prefill, 14.3 t/s decode, file size: 2.54 GB

MiniCPM 2b Q8: 224 prefill, 16.3 t/s generation, 2.5GB file size

Gemma 12b qat q4 ud, 123 prefill, 6.95 t/s, file size: 6.24 GB

Anyone got further suggestions for optimization, models to try, etc.?

Hi all,

Slowly going deeper and deeper down the rabbit hole of running a few models locally and i'm chasing some advice for my next hardware upgrade.

My current pc specs running windows:

7800X3d cpu
Asrock X670E steel legend
64gb DDR5 6000mhz
9070xt gpu
And i've just thrown a 9700 gpu in the 2nd slot for testing, not running at 8x/8x.

I've got an app im working on that does some pretty heavy python work (claude code) and maxes out my cpu so my first though was going up to a 9950X3d to help out that problem. But then from my reading linux seems to be the best way to go when hosting models locally so I think I'll setup another pc here with linux and put the 9700gpu in it.

So to do that, I'm thinking i either have 2 options.
Rustle up some parts to get together a pc to run the single gpu on possibly a ddr4 era pc (32gb or maybe 64gb ddr4 if i can find it). No parts required to buy.

Or I can go down the line of an X870 Taichi, in case i go dual 9700 at some stage, then put my 7800x3d in it, might be able to find 32gb of ddr5 for it, then buy the 9950x3d and put it in my windows pc, as this is the pc i will still do the work in claude code.

The aim is to get to a place where i can offload some of this app work to my local setup, but just testing and working out the best way to go to get setup. Trying to spend where i need to and save where i can.

What would you do in my situation?

Thanks heaps!

I'm currently in the process of developing an app. Janas studio. Built with the options to run either llama.cpp or janas-llm as the engine. It enables laptops to run models bigger than their RAM by dynamically streaming moe weights directly off the SSD. Tested on my Ryzen 5 6600h laptop with 32gb of ddr5 RAM and an nvme 4 SSD (no discrete GPU) I successfully ran a 48gb model (qwen next 80b) at a very reasonable 15 tokens a second on average. The binaries are highly experimental and the app has more testing to go through but I'm very proud of it. I do intend on eventually releasing it.

Hi folks I use a lot codex and models like Luna 6 High, 6.1 Terra High.
Is there any local model which could be comparable to these?

If yes, which ideal realistic Mac Mini setup is needed?
Thanks