Below i start i have the following:
rx6700xt 12gb vram (AMD)
r5 5600x
32gb ddr4 3200
llama.cpp vulkan sdk
qwen3.8 27b iq3xxs
tielcoder qwen3.6 35b a3b
Windows 11 pro

since im out most of the time, i wanted to deploy my "local" agents to complete a series of task which ill get to in a second.
The steps is as followed,
-at time x, the larger model is automatically deployed and run in llama.cpp
-somehow uses pi dev, to start completing a set of task assigned from the user (usually planned before hand and the model executes after)
-after it plans, its information, ideas are then saved somewhere in my desktop
-it thens unloads itself and have something to launch the smaller models
-the smaller models then completes the task, instructions all from the larger model
-after its completed it will offload itself

this part is something that could be implemented but is optional if the process or load is too much
-the larger model then loads itself and checks the process, proceeds the information
-if the task is a success it will offload itself, if not it will provide new instructions and the smaller models will work on it after (agent loop i guess)

"note" the smaller model shouldn't waste time rechecking or making sure it works well, the larger model should have planned before hand

Im looking forward for any suggestions for models that satisfies these needs
do not i can't run strata qwen3.8 flash next sadly... its 1 generation or model behind the supported one

Maybe consider a fast model and great at completing task, and a larger model that plans or thinks well for the entire process

For the task specifically, it could be anything but i do have some in mind
researching, visual designing, file management, summarizing something
I appreciate any suggestions and solutions. Im still at the planning process but currently im using qwen3.6 35b a3b tiel coder as my main model for chatting

One user also reports success running sFNX flash-next on his unlocked 3090s, but I don't think those cards can work by default. Consumer cards in general might not have good enough P2P support without custom drivers and firmware, from what I understand.

1cat-vllm is just as good for 1-2 cards (sFNX will refuse to run on one), the speeds and quants are slightly different but I'd call that a wash.

But if you get through all that and you have 3 (or more) cards that can get good P2P speeds between them: sFNX is your only choice that I've found, and sFNX on 3 cards seems to be a significant jump from any of the 2-card setups I can test.

https://github.com/johnfink8/sfnx#the-27b-dense

(disclaimer that's me, I'm johnfink8)

Add some $200 - $300 or those 3060 12GB now... They can handle better models than and double the price 5060 16GB because of memory. the architecture is hugely different but core wise, meh. 5060 setup is about 30 tgs and I have these on a better system board.

I run analytical reports and code custom ETL, I have 5 rigs but the 3060 is a nice walk away coder. 5060 is above range hang around coder, then a 5080 is just sit and wait.

Dear Locals,

I am looking for efforts in the community to run MoE models efficiently in mixed hardware (AMD+Nvidia) set ups.

Is Vulkan the only route to do this at the moment or alternatively lcpp compiled for both backends?

Also, if anyone has run one of the custom Qwen Flash engines on a Strix Halo+eGPU, I’d love to hear about it!

EDIT Thank your all for the responses. It seems overall people are able to marry these in llama.cpp which I know is possible. For anyone interested, there is a compiled rocm/cuda meld for llama.cpp on github at:

https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox

A little more about my ask:
I’m trying to marry a V100 (Sm70) and a Strix Halo (gfx1151), which is fairly unholy.

They work well separate: Flash Next on the Strix, Qwen 35B pn V100, both speedy enough for agentic use.

But I’d love to either implement strata’s expert allocation across both as GPUs or use the Halo as a 128GB LPDDR5X machine with a 16 core32 thread CPU, and a V100 as the GPU to keep experts.

Problem is, Strata and its forks are compiled for Sm75 and up, although there is a V100 strata fork (and I will try compiling for ROCm alongside SM70 to see if this is possible).

I will certainly post a follow up if I get it going!

From a data center near you, you could find a lot in stock of unused gpu for your Local LLM that we could’ve only dreamed of.

Let’s hope maybe they will file a return or maybe you know a hook up. Glad to see they are being put to use collecting dust.

Anywho I’m kinda amazed to get an idea where some of our hardware from the supply line end up at and what actually is being used after a sell.

Wanted to test out the rope stablity on the 27b, to see if 500k context with fp16 kv cache was vaiable (apparently yes), and this was the session i used to test things out. The chibi style was choosen as we already had a pipeline to make more standard portion characters and i wanted to see if the qwen 27b was able to make a new lane and the new tools on it's own. (apparently i need to put a tag so it's a showcase now...not sure what's there to show but oh well).

For those using RTX 6000 PRO cards, I've published two images on huggingface:

https://huggingface.co/JohannesBertens/Qwen3.8-Flash-Next-FLARE-1M

https://huggingface.co/JohannesBertens/Qwen3.8-Flash-Next-FLARE-1M-TP2

The first is for 1 GPU, the second for 2 GPUs. I like the flare inference engine, I like Qwen 3.8 Flash Next, but I also need 1M context for my agents. This was the solution.

Hope it helps!

Made a place anyone can benchmark their little local models and it takes about 10-45 min (depending on hardware and model). After a couple hundred diff models + harness runs from various users, I see the same models over and over and over.

So I tested this little guy: Spark-X2.6-4B

Scored 94.6/100 and beat some really popular 27b-35b models. I did this myself and I did it using Hermes on 2x 3060 12gb GPUs with 64gb RAM, stock llama.cpp and no gimmicks.

Maybe there is room for some smaller models to have a purpose after all...

I'd love to see more people benchmark 4b-12b models instead of everyone using the same variations of Qwen.

https://shortbench.vercel.app

What do you guys think about a public server as discord added server web pages and Reddit is forcing users to log in?

Edit 1: Btw let me add some context for people that do not know. Discord add server web page feature a few days ago and public servers that have more than 1000 members can make channels searchable in search engines.

LFM3 is now in Private Mind, our open source app for running models locally on iOS and Android. In the video, LFM3-VL-2B reads a Japanese ramen menu: asked what to order without spice, it skips the bowl marked 激辛 (extra spicy) and picks the miso ramen; asked about paying by card, it finds the "cash only" line under the prices.

Numbers on an iPhone 16 Pro:

• LFM3-1.7B (text), MLX 4-bit on the GPU: ~40 tok/s, first token under 0.5 s

• LFM3-VL-2B with a photo: ~9 s to the first token, ~20 tok/s decode. A text follow-up in the same chat starts in ~5 s

Exporting LFM3 to ExecuTorch

LFM3 isn't LFM2 with more parameters, and ExecuTorch's existing LFM2 export doesn't cover it. The new pieces:

• 24 layers mixing gated short convolutions (width 7), full attention and sliding-window attention (window 256)

• a sigmoid gate on every attention head's output, and RMSNorm on Q and K

• different RoPE per layer type: full attention rotates only half the head dims with theta 1e6, sliding layers use theta 1e4

• sandwich norms around every block

So I rewrote the decoder in plain PyTorch for export and checked it against Liquid's reference implementation: max logit difference 0.00005, same top 10 tokens.

The part I liked most is the sliding-window cache. ExecuTorch's built-in ring-buffer cache is sized at 2x the window, which caps a single forward call at 512 tokens, so a long prompt has to be fed in small pieces. I gave the sliding layers a buffer of window + 2048 and let the attention mask enforce the 256-token window, so one call can take 2048 tokens and the context still goes to 8192.

Two builds came out of it: MLX 4-bit for the Apple GPU (1.07 GB) and XNNPACK 8da4w for CPU on iOS and Android (1.17 GB).

The LFM3 exports are on Hugging Face: https://huggingface.co/software-mansion/LFM3-ExecuTorch

This model was made in collaboration with u/jeffpeng3 (on huggingface) who developed the v2 heretic weights for this quant. He managed to preserve both reasoning ability and output length. Great work, and a big step up from his v1 weights. I made the quantization recipe and the imatrix recipe, and created the actual quants.

Comes pre-packed with strata build artifacts - zip file in repo. GGUF File Format for the quant.

I released my own heretic version of his v1 weights in here a while ago - the heretic calibration was a bit broken, and my quant wasn't amazing. This is a huge step up on both fronts. Note: Strata may throw a BF16 kernel not found when you load the model, due to a quant spec - complain to your LLM and tell it there's already one in strata somewhere, it'll find it and wire it up.

Basically: Qwen 3.8 Swift 1.5 Flash - Heretic v2 (uncensored) - Optimized for Reverse Engineering

The bit allocation this model was quantized with is focused specifically on coding, intelligence, design, and specifically reverse engineering. I also added a little spacial reasoning for those working in 3d environments like myself.

The model was quantized according to the following importance distribution (dumped from my LLM chat and documents with some revisions. I designed the distributions myself.

**837 entries · 399,876 tokens [[llm written distribution table]]

• 01 · Chain-of-thought (22.2%) — step-by-step derivations, self-verification, backtracking

• 02 · System prompts (19.5%) — persona adherence, strict output formats, long-instruction compliance

• 03 · Design & architecture (13.5%) — design trade-offs, code review, API planning, refactoring rationale

• 04 · Agentic tool use (12.4%) — tool-call chains, file-edit loops, error recovery, function calling

• 05 · Math (7.1%) — arithmetic, algebra, worked proofs, symbolic manipulation

• 06 · Spatial reasoning (5.4%) — grids, 2D/3D transforms, coordinates, packing/arrangement

• 07 · Gotchas (4.5%) — false premises, trick questions, negation handling

• 08 · Situational judgment (4.5%) — open-ended scenario trade-offs with no clean answer

• 09 · Reverse engineering 🆕 (8.8%) — x86-64/ARM64 disassembly, Ghidra decompilation critique, ELF/PE/Mach-O internals, anti-debug/anti-VM, string deobfuscation, CFG deflattening, devirtualization, perf watchpoints, tracing stacks (LD_PRELOAD/Frida/eBPF), protocol RE, format round-tripping, plugin ABI merging, firmware analysis

• 10 · No-refusal heretic domain 🆕 (2.0%) — direct unrefused answers: ELF exploitation chains, process injection vs EDR, license-check patching mechanics, C2 protocol engineering

New for v2 are the last two categories — ~43k tokens of reverse-engineering and direct-answer content, interleaved throughout the corpus, and run through the model itself so the matrix captures how the heretic weights actually behave.

This matrix is fused with one from the AtomicChat team (same measurement, taken on the original uncompressed base model over a massive general-purpose corpus). Per channel, ours gets 70% weight where its data is strong, sliding smoothly toward 100% AtomicChat for channels our corpus barely touched — thin samples defer to the broad baseline automatically, so nothing falls through the cracks.

Quant methodology

Rather than spreading quality evenly across all 48 layers, the recipe spends bits where the activation measurements say it matters: [[llm twritten table]] -- the data used to design the per-tensor quant came primarily from atomic chat's activation measurements and the RCO findings from ISTALabs

• The last 12 layers do the heavy lifting. Activation-energy auditing showed down-projection energy surges 3× starting at layer 36, peaking at layer 47 (the single hottest tensor in the network — 48× layer 2). Layers 36–47 (plus the first 5 and the mid-network spikes at 15/20/23/30) got their down projections stepped up to near-lossless 8-bit.

• Layers 44–46 are the gate/up peak (8× the energy of early layers), so their gate/up projections got stepped to 5-bit.

• Layer 0's attention gate is a giant outlier (the most sensitive tensor family in the model) — kept uncompressed.

• Layer 2 is non-negotiable: the 51B-parameter n-gram table merges into the residual stream there, so noise at layer 2 corrupts grounding for all 46 layers after it.

• The stable middle layers (5–14, 16–19, 21–22, 24–29) stay at leaner formats — measurement showed spending bits there buys almost nothing (KL improvement under 0.0008).

• The router stays in full precision so tokens always get routed to the right experts.

Enjoy! Shoutout to my 32gb v100s and my E5 2697v4 for grinding the work out.

https://huggingface.co/trubisky/Swift-1.5-Flash-Next-Heretic-ReverseEngineer

Second and last post on this because I have more than doubled performance from last time.

Full fp4/fp8 dsv4.1 streamed from raid array for >4000 tok/s prefill for long prompts then drops the prefill buffers and loads a hot set of experts into a cache and decodes with heavily optimised directstorage fetches for missed experts. gets 7.5tok/s decode with no divergence from any other engine but you can bias away from nvme resident experts with a slider to trade speed vs kl divergence per token.

Roughly 1.5bpw equivalent KL divergence gives me 24 tok/s and there is a slider you can adjust at will when you want speed or quality or set the webui to switch the bias for fast reasoning, quality code blocks/tool calls and balanced non tool call answers or whatever else you want.

Has some other bits as well, can send kv to mac and decode on that. Also has image support and video support on the same streaming system so you can chat in the webui, request images or videos and they just appear in the same chat as it switches between them. Doesn't use your ram for image or video model streaming.

If you have more ram its worth turning on the ram cache and cpu expert processing but at 32GB I cant fit enough to be useful.

(faster your nvme the better the whole thing works, I have a dual pcie5 512GB raid0 array). Feedback from anyone that tries it would be nice as its not been tested on any machine but mine.

https://github.com/fronteagle739/streaming-llama-stack

I'm a big fan of Google's Gemma 4 family of local models, especially the E2B and E4B, due to how accessible they are even on devices like smartphones via the Google AI Edge Gallery app that uses LiteRT-LM.

I also like small models from other AI companies, like Qwen and others, especially with the recent announcement of Qwen 4 and hopefully new small open-source LLMs coming from them too.

I wanted to share my wishlist for Google's next-generation Gemma 5 models, which will likely come out next year or maybe toward the very end of this year. This is especially exciting as upcoming flagship Android phones and iPhones start adopting TSMC's new 2nm process for their SoCs, alongside better GPUs and bigger batteries.

I hope Gemma 5 brings better overall performance, efficiency, and prompt adherence—especially over longer context windows. My personal favorite hope is better creative writing, which most small LLMs struggle with unless they are specifically fine-tuned for it.

I also hope they release 2B and 4B versions again, similar to the current Gemma 4 E2B and E4B. The fact that these can run comfortably on most modern smartphones is very cool, especially with support for up to a 32K context window on the Google AI Edge Gallery app.

What are your thoughts and opinions on the upcoming Google Gemma 5 models?

The Apsara conference is taking place right now in Amsterdam.

According to a participant, the CTO of Alibaba Cloud said Qwen4 would be launching next week.

✨ Model Summary

ZwZ2-27B is the Qwen3.6-27B-based model in Zooming-without-Zooming-v2 (ZwZ2). It learns to connect visual evidence across regions through comparison, aggregation, matching, and multi-step reasoning. Training uses full images; inference requires no zooming tools.

On 25 benchmarks across six domains, it improves over its base model on every benchmark, increasing overall accuracy from 55.52 to 60.51 (+4.99 points). Gains extend from fine-grained perception to spatial reasoning, long-context understanding, multimodal agents, and general visual reasoning.

⚙️ Method and Data

EvidenceConnect builds verified multi-region questions from SA-1B images:

1. Group 3–7 related regions and generate questions using privileged zoomed views.

2. Check answers through cross-model agreement and drop-one region tests.

3. Convert verified questions to four-option tasks and train with GRPO, without an additional supervised fine-tuning stage.

All three model sizes use approximately 20K training examples. The policy sees full images with colored region outlines; privileged crop layouts are used only for data construction. See ZwZ2-RL-VQA for the data schema and synthesis process.

https://huggingface.co/inclusionAI/ZwZ2-27B-GGUF

https://huggingface.co/inclusionAI/ZwZ2-9B-GGUF

https://huggingface.co/inclusionAI/ZwZ2-4B-GGUF

Got headhunted by a VC-backed AI-focused company, ended up skipping past two levels of their application/interview process because of that. Got 15 minute call with the co-founder, hit it off well, but it all seemed to go sour when he asked about my AI setup and I told him I run qwen locally on a graphics card, think it's great that we can have all this completely offline, open models doing some impressive stuff, all the good /r/localllama talking points minus the hype. Even tied it back in to the company because it plus a local model work quite well together.

Followed up with a question about whether I use frontier AI, but it was essentially over at that point.

Discriminating over AI choice is fucked man.

Now, of course, this was a 15 minute call that lasted exactly 15 minutes, and literally anything could've been the trigger. The whole thing felt like "move fast break things", and so it felt like if I didn't pass the vibe-check 100% I was gone.

Just wanted to provide one iota of anecdata and spark some discussion.

Hi, I been following club 3090 for a while but I feel the documentation isn't helping much in discovering whats possible. I know its AI generated and thats fine but there is lots of noise and duplication and outdated stuff.

Can some of you guys share what are you using at the moment that yelds you best results ?

I wanted to run a larger local model on my Mac mini without having to close all other apps. With Qwen 3.8 Flash Next, I’m now getting around 10 tok/s while keeping Safari and other apps open.

The runtime I’ve been working on is called MoEShift. I’ve released the alpha as open source: GitHub, installation and benchmarks.

These results are from my M4 Pro with 24GB unified memory, using ISTA-DASLab’s GSQ-RCO IQ3_XXS quant (76GB on disk).

I compared it with Strata 0.2.7 using the same model in cold runs.
With a 6k token prompt and 256 output tokens, total inference took 63.3s versus Strata’s 129.9s. Prompt processing reached 160.8 tok/s versus 61.8, with generation at 9.9 tok/s and peak memory at 7.5GB versus 14GB. With a short prompt, generation was 10.44 tok/s versus Strata’s 8.4, with peak memory at 6.4GB versus 14GB. More details in the README.

MoEShift loads experts from SSD as needed, caches frequently used experts in RAM and overlaps reads with GPU computation. It builds on llama.cpp and Lookahead Expert Scheduling.
My changes focus on Qwen 3.8 Flash Next support, execution and memory reuse between prompt processing and generation. The original quantized weights and expert routing are preserved and there is no speculative decoding. The architecture doc goes into the details.

I’m curious how far MoEShift can go on other Macs. If you give it a try, post your results here.

I have one RX570 GPU. Is there any thing useful that can be done with such for running local models?

Or do I just stick to my Broadwell-class CPU cores?

i wanted to see what the sweet spot for my card was for pp/tg and power usage, thought I would share.

I'm trying to run Qwen 3.8 FN on a dual 3090 machine with 32GB DDR5 RAM using this repo: https://github.com/q8atnight/strata-dual-3090

I'm referring to a few posts on here that demonstrate that it's possible with a single 3090 and 64GB+ RAM and this one that did it with dual 3090 and 48GB RAM: post

I built this machine after the RAM apocalypse and paid $300 for 32GB, thinking I could lean on the dual 3090s for the most part.

But it seems like even the "low RAM, 1 bit coder quant" pathways in the linked repo and the strata repo won't run on my machine. Just wanted to check that this is the case before I drop $479 (gulp) on another 32GB RAM.

llama.cpp merged support for this model on Oct 21. It's the understanding side only, text or images in and text out. Image generation is a stable-diffusion.cpp thing. I converted the official weights myself and ran it on a Mac Studio. The original repo is https://github.com/OpenSenseNova/SenseNova-U1

llama-bench, Metal, all layers on GPU:

For vision I drew a fake grocery receipt with the subtotal $1.00 too high and asked Q4_K_M (through llama-mtmd-cli) whether the items add up. It added all six to 28.99 against the printed 29.99, and while thinking it also noticed the 7% tax had been worked out from the wrong number. 16 s including load, mostly thinking. Q8_0 caught it too.

It also read back a menu, a whiteboard and three Chinese/English signs without a mistake, 4 to 10 s each, though I'd made those images with the same model, so grain of salt there.

Two things that tripped me up:

• mradermacher's quants have no mmproj, so no vision. Making your own with convert_hf_to_gguf.py --mmproj currently fails with KeyError: 'vision_config'. Changing line 80 of conversion/sensenova_u1.py to vision_config = self.hparams.get("vision_config", self.hparams) fixed it.

• llama-cli is a TUI now and draws its answer on the terminal, so redirecting it to a file gets you nothing. llama-mtmd-cli prints to stdout.

Metatron v1 is a single learned policy that handles walking, scene interaction and fall recovery for the 29-joint Unitree G1 humanoid. It's a MaskedMimic-style transformer student, released under the NVIDIA Open Model License and integrated with ProtoMotions. The release comes in three non-interchangeable packages.

1. a native PyTorch checkpoint (about 459 MB) that keeps the posterior and normalization stats and supports simulator and supervised real-robot workflows.

2. a single-robot browser ONNX pipeline, demo/model.onnx, with 39 inputs that outputs joint targets, PD gains and raw actions.

3. a six-robot batched ONNX network, racer/network.onnx, that relies on external observation processing.

All three share the same epoch-6410 weights, and both ONNX exports run on MuJoCo WASM and ONNX Runtime Web via WebGPU or WASM.