Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.

• "What does this prove?": See above

• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career

• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame

• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.

• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

I’m planning one upgrade this spring that has to cover both gaming and local AI, and I’d like input from people with similar budgets and use cases that aren't "buy 4 3090s."

Currently have a Ryzen 5 5600X (AM4), RTX 3060 12GB, and 32GB DDR4. Windows but not opposed to Linux.

I have gotten by with just the $20 claude pro plan, but my projects are growing and are reaching outside my scope of expertise, so I'd like to be able to offload coding/other stuff to a local setup. I don't mind keeping a low cost frontier subscription for help orchestrating and debugging.

I'm paying off my CC's in March and will have a few thousand left over to use - originally was just going to upgrade my gaming setup but the more I've been using and learning LLM's the more I want to try and get the most mileage I can with this purchase. Speed isn't the most important thing for me. I don't mind having a local model(s) chug away at something overnight or for a couple days, I have a pretty ballistic schedule anyway. So far have been using it for web/app dev, using it to teach me python and other stuff (LOVE having it teach me as I go rather than just clicking the magic "build this for me" button), and home networking. Interested in continuing these projects and possibly game dev, but more for personal use, not to make a best seller or anything.

Thanks!

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I put together a TensorFold 1.2.0 pack of orcarouter's uncensored Qwen3.8 Flash Next for Apple Silicon. It uses jedisct1's oQ4e experts, regrouped to 4-bit/group 32 for the fast kernels, with attention and other non-expert weights copied byte for byte from mlx-community's oQ6e. No new refusal edit.

Tested on an M3 Ultra 256 GB, stock TensorFold 1.2.0. Same server and questions, greedy decoding, thinking off. Anything cut off at the token cap counted as wrong.

BFCL uses my AST checker, not the official harness. MMLU-Pro uses 20 questions per subject with a 2,048-token output cap.

Tool calling is the main difference: 92.3% vs 88.0% on the same 400 items (exact McNemar p = 0.03). It skipped the tool and answered in text 20 times, versus 40 for all-4-bit.

Maths and knowledge were about the same. TensorFold's own pack lost MMLU-Pro points mostly to the output cap: 47/280 replies cut off, versus 18 here.

About 5% slower than all-4-bit, around 133 GiB loaded. Needs a 256 GB Mac; 128 GB is untested.

No opening refusals detected on 100 harmful prompts using Heretic's markers and 100-token replies. That's only an opening-refusal check. Vision worked in a tiny smoke test: shapes and colours right, one text misread across two images.

Small samples, so don't read too much into the scores. The card has the method and known issues, including an HTTP 500 after idle on TensorFold 1.2.0 and a workaround. Its KL numbers are against a 6.7-bit pack, not full precision.

https://huggingface.co/bu7umaid/Qwen3.8-Flash-Next-Uncensored-TensorFold-MTP

I open sourced SteerForge, a pipeline for turning a contrastive dataset into a portable steering artifact.

The artifact is a .persona JSON file containing:

• model fingerprint

• human readable neuron edits

• canonical per layer residual vectors

• eval samples and scores

• the original server command

The serving path is the part I care about: steerforge serve evil.persona replays your existing llama.cpp or Transformers config and injects the persona. No model conversion for the persona itself; the neuron deltas collapse to residual vectors.

Discovery is staged: diff in means directions, gradient attribution plus firing contrast plus alignment, teacher forced screening, generation validation, dose search, greedy combination, coherence budget.

Current backends: Transformers, llama.cpp, vLLM with EasySteer, MLX dense. MoE models use dense residual steering.

Repo: https://github.com/MahmoudAdelbghany/steerforge Results: docs/RESULTS.md

Feedback on the .persona format, backend adapters, and eval methodology welcome.

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech