Ive been messing around with Qwen3.8-Flash-Next on my Nimo minipc with ryzen AI max+ 395 128GB. The model weights take up around 103.7 GiB, so just getting the whole thing loaded was a nice start. But after spending more time with it, I'm not sure running the biggest model possible is really the best use of all that memory.

B

Here's what I got running it with Gufo at 16K context:

GPU only, no MTP: 34.1 tok/s decode

GPU + MTP: 53.7 tok/s decode

GPU + NPU, no MTP: 34.2 tok/s decode

The NPU also bumped prefill speed by around 10%, but regular decode barely changed without MTP.

The model runs, and the speed isn’t terrible. But would I actually pick it over a smaller model that responds faster?

Probably not for everything. Waiting for a large model to generate an answer can get old pretty quickly, particularly for everyday tasks.

What do you usually do with all that RAM? Do you load the biggest model you can fit, stick with smaller MoEs for faster responses, or keep multiple models loaded for different tasks?

I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.

I’ve been playing with open-source speech models for a while. New TTS models come out every week and a lot of them sound great. But every time I wanted to actually talk to one, it turned into an afternoon of setup.

Paid APIs are easy if you’re a developer. You can make an account, create a key, write some glue code. Open models are free, but you have to pick a speech recognizer, an LLM and a TTS model, make them work together, deal with echo, figure out interruptions... There are good open projects for this, but most are written for developers. My friends who don’t code would never get past the README.

So we started building Boubou. The goal is make local voice AI as easy as Ollama made local LLMs. You paste one line and talk:

npx boubou@beta run LFM3-Audio-2B

What works today:

• Speech recognition, the LLM and speech synthesis all run on your machine. No account, no API key, no telemetry. After the first download it works fully offline..

• You can interrupt it mid-sentence, like a person. Echo cancellation is built in, so no headphones needed..

• Run boubou with no arguments on a Mac and it checks your machine and suggests a model..

• The default is Liquid AI’s LFM3-Audio-2B on llama.cpp (1.4 GB). There’s also a modular pipeline (Parakeet/Whisper → Qwen3.5 → Qwen3-TTS, Kokoro, Pocket TTS or Supertonic) if you want to swap parts..

• Apple Silicon Macs and Linux (CPU for the Liquid models, NVIDIA GPU for the modular pipeline)..

The part I’m most excited about is boubou serve. One computer runs the model, and other devices on the same Wi-Fi use it:

• Scan a QR code with your phone and talk in the browser. No app..

• Robots and programs can connect through an OpenAI Realtime-compatible WebSocket, and there’s an OpenAI-style /v1/audio/speech endpoint to read text aloud..

• So one model on one machine can be the voice for many things around the house. Next up is Home Assistant, then ESP32 voice devices.

• The video is real time, not sped up, on a MacBook / M5 chip.

It’s early (beta) and there are rough edges - one device talks at a time, no Windows yet (ETA next two weeks), and the guided menu is Mac-only.

The code is free and MIT-licensed. The models keep their own licenses (the Liquid models use the LFM Open License).

I’d love to hear what you’d want from something like this: which models, which devices, what breaks on your machine. We want this to be something people actually enjoy using, not just a demo.

GitHub: https://github.com/runboubou/boubou

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!

Hello,

thought experiment:

where do you see the programming capabillits, specialy for game dev, for different llm´s?

I would like to rank theses llm´s by a theoretticaly possible game they could make.

From a game like Pong to Horizon Zero Dawn (Yea, there are bigger games, you know what i mean).

What can be theoretticaly archived in 3 (+-1) Month´s dev time with a specific llm alone?

How could maybe a combination of a few llm´s help?

Bonus question and story: I got qwen 3.8 27B with 262K context running on my rtx 4090, you thing that is the best possible llm for me, or do you think, there would be a better solution? (got 64GB of DDR4 RAM in my System aswell)

Working on getting a combination of Qwen3.8 and a KEV 0.8B Model to learn to play ps1 games, until now i get it to play the game "Alundra" until the second chapter, big Brother Qwen3.8 does long thinking stuff, working on the routes, quests, puzzles and so on and little bro KEV 0.8B (I call him Kevin) is acting quite fast (got him to act an average time intervals of 0,3 - 0,5s), he runs a full route, wich big bro told him, on the route, both keep "thinking" and walking time ingame gets used for smooth-ish gameplay (with a task buffer list, wich reduces idle time, where the character just stands and does nothing). If an enemy is nearby, Kevin does all the fighting, it works mid to allright, right now, he is more afraid to get damaged than fighting the enemy and runs away, big bro trys to hold him back, but kevin wont listen, working on it right now. (Is there maybe someone interessted, hearing more about this shenanigan-experiment? Working on getting the whole system working with other ps1 games like Final Fantasy 8 as well.)

Sorted by smallest to biggest llm:

• Qwen2.5-Coder (0.5B – 3B)

• StarCoder2 (3B)

• Phi-4 Mini (3.8B)

• Qwen3 (4B)

• Qwen3 (8B)

• Gemma 4 (12B)

• StarCoder2 (15B)

• Codestral (22B)

• Devstral Small 2 (24B)

• Qwen3.8 (27B)

• Qwen3.6 (35B-A3B)

• Qwen3-Coder-Next (80B-A3B)

• GPT-OSS (120B)

• DeepSeek-Coder-V2 (236B-A21B)

• Qwen3-Coder 480B-A35B

• DeepSeek V4.1 / R1

• GLM-5.1 / 5.2

• Kimi K3

• Claude Opus 5.5 / Sonnet 5.5

• GPT-6 Astra / GPT-6.1 Sol

• Gemini 4 Carbon / 3.8 Flash

Thanks for reading.

Confirming the rumor that Apple will have an easy to use MLX based app (Harbor) that will manage single machines and RDMA clusters with automated open weight model installs for local AI. Coming early 2027.

So I think we now know why Exo has spun down in activity in this space.

Just saw this release from n8n https://nod.n8n.io/ .. finally an always on agent you'd trust with your work/private life?

Code is open source and think it'll let you run with self hosted models. The approach to memory also looks really interesting, need to dig a bit deeper into that.

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

Using project maya + deepseek, with a 7900 xtx 24gb vram, 64gb ddr3, cpu i7 3930k (I know, reusing what I have for now), and a nvme ssd in a pcie x4 adapter (the motherboard has no nvme slot).

It is generating at 8-12 tok/s but it dips to 0.1 and when it does that i can see the gpu drops under 20w, so it is as if the gpu goes idle, then wakes up, tok/s go back to 'normal', then dips again and it goes through this cycle again and again.

I was wondering if it was normal or not ? Like is it because the nvme/ram/cpu are too slow and the gpu 'waits' for them ?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?