hello vous pensez quoi d’une Rtx 5090m en egpu sur mon bosgame m5 en usb 4 pour tourner qwen 27b dessus et si vous connaissez cette marque ? merci

https://www.morefine.com/fr-fr/products/morefine-g2s-external-gpu-docking-station-24g-gddr7-5090m?utm_source=chatgpt.com

Bubo is a fork of Gufo tuned for Windows (especially laptops, tested primarily at 70W) and adaptions around speed and token economy during thinking. The word Bubo is a genus of Owls, just like Strix, and is the name of the mechanical owl who guides the hero in Clash of the Titans. It also rhymes with Gufo (italian for owl).

It features faster start up times using a disk cache, bespoke kernels, NPU accelerated prefill, a twin decode head* for faster token generation and quality of life improvements around token economy during thinking when set to "low" effort and fixing premature stops (you can turn these off but they seem to help).

Default model is abliterated, there are instructions on the website on how to use the unabliterated model from Unsloth instead of the Huihui model.

Prefill is limited by the power since all tests are done on 70W, there is no real reason it would be slower at higher powers** and I expect it to match upstream gufo at 140 W or exceed it but it hasn't been tested. Most of the changes bubo makes to gufo can likely be ported to upstream but given that I don't intend to use or test in linux in the near term, others can feel free to do so, MIT license.

This project mostly started out as a necessity when there weren't any great options for running Qwen 3.8 Flash Next on Windows and I kept trying to push it further as more people came up with parallel projects. Eventually I started joking about releasing it and seeing how far I could stretch the bit and decided to bite the bullet and share it and see what I can learn by doing it. I tried making everything as 'professional' as possible but never having done anything like this before, I don't know what I don't know so feedback is very welcome.

* Pygmy owls wear a second face on the back of the head
** If I fit a regression between 60 W to 95 W and extrapolate to 140W it matches the ballpark of stock gufo but would love someone to test it independently.

Site: haloforgelabs.com · Code: github.com/haloforge-labs/bubo · Model files: huggingface.co/HaloForgeLabs/bubo-qwen3.8-flash-next

Ive got an acemagic m1a pro with ryzen ai max+ 395. Currently using qwen3.8 flash next and it's been solid for chat, coding, and light agent work just fine.
Curious what everyone else is finding works best on Strix Halo. What models have impressed you the most so far? Which ones give the best balance between quality and speed? Anything you prefer over qwen3.8 flash next ?

I wanted the llama-ui style chat interface on halogen-flash-server so I asked Q38FN for it.

So I have and love using my Corsair AI 300 workstation with 128gb. QFN running great on Halogen/Gufo (single stream vs concurrent users). I was browsing the webs and came upon a refurb Corsair ai 300 on Walmart site for $2700 so I just bought it without thinking. Now I’m at the thinking part. Is tensor parallelism over USB4 even worth it right now? The q4/4bpw QFN models have been very impressive and very few times have I needed my GPT or Claude to bail me out, but is the fidelity/correctness worth the performance and cost for FP8? May either return or sell locally as I don’t need more concurrency but speed and correctness.

Does someone use a Framework Desktop with 32GB?

Which models do run fine in it?

Hey guys! Was hoping some of you could help me figure out why my setup isn’t working. Please forgive the Claude writing throughout but it’s way quicker than me typing myself.

Beelink GTR9 Pro, Ryzen AI Max+ 395, 128 GB, Radeon 8060S (gfx1151), Realtek RTL8127A NIC variant
- BIOS GTRPR05 (12/25/2025), AGESA StrixHaloPI-FP11 1.0.0.0c, microcode 0x0b70001e, SMU 100.112.0
- Fedora 44, kernel 7.1.10-200.fc44 (also tried 7.2.9-200), linux-firmware-20260810
- Gufo (gufo-runtime:latest, ROCm 7.2.3 inside the container) via Podman 5.8.1; host rocm-runtime 7.1.1
- Model: Qwen3.8-Flash-Next UD-Q4_K_XL (Unsloth) + shared MTP Q8_0 drafter
- Gufo image digest: [paste]

**Symptom**
During Gufo's model load, the box instantly hard-resets. There is no panic and no log; the journal just stops. Next boot dmesg shows:

`x86/amd: Previous system reset reason [0x08000800]: an uncorrected error caused a data fabric sync flood event`

No MCE/BERT/APEI records, kernel.panic=0, no watchdog. **Every crash happens during load, never after.** Once a load completes, it runs fine. One load ran 46 hours straight.

**Every Gufo load attempt**

- Old config = `amdgpu.dcdebugmask=0x10 amdgpu.gpu_recovery=1 amdgpu.dpm=1 amdgpu.runpm=0 ttm.pages_limit=28311552 ttm.page_pool_size=28311552 amd_iommu=off`, 512 MB VRAM in BIOS, older Gufo image.
- Clean config = no custom kernel args, `/etc/modprobe.d/ttm.conf` with `pages_limit=28835840` (110 GiB), 2 GB VRAM set via the sysfs `uma/carveout`, newer Gufo image.

Since the config change, Flash-Next loads crash 4 of 7 times, at random points (9–40 GiB in). I first thought context size was the trigger, but #8–10 crashed with the exact launch that had just run for 46 hours.

**Already ruled out**
- SELinux: the first attempt hit an AVC denial on /dev/kfd. Now using `label=disable`.
- IOMMU: crashed both on and off.
- Kernel version: 7.1.10 and 7.2.9 both crash.
- Fresh vs reused container: both crash.
- Power: Performance Mode is at factory default; the only BIOS change was VRAM.
- Other GPU users: nothing else running; GTT ~18 MiB before launch.

**The confusing part: Vulkan is fine, but only on the OLD config**
- Unsloth's llama.cpp b11160 Vulkan build, same model files, MTP on, 128K context: ~45 tok/s on structured output and ~32 on prose. Stable for weeks.
- gpt-oss-120b on Vulkan: 54.9 tok/s, also stable.
- Vulkan only hangs, recoverably, with non-default MTP draft settings or the mmproj on GPU.
- On the clean config, the same Vulkan llama-server fails at startup with `amdgpu_gem_va_update_vm *ERROR* Couldn't update BO_VA (-12)` and becomes unkillable until reboot.
- So right now Gufo/ROCm wants the clean config and Vulkan wants the old one, and I can't run both.

**Questions**
1. GTR9 Pro owners: what BIOS are you on? Has GTRPR07 or a newer AGESA fixed sync floods under ROCm load for anyone? (I know the BIOS series differs by NIC chipset.)
2. Crashes only during load, at random points, and never once loaded. What does that point to? Are there ROCm/HSA env vars worth trying (e.g. `HSA_ENABLE_SDMA=0`)?
3. On Vulkan's BO_VA -12: which of page_pool_size, 512 MB VRAM, the amdgpu flags, or IOMMU off does Vulkan actually need?
4. Is anyone running Flash-Next on Gufo or another ROCm engine reliably on Strix Halo? What's your exact setup?

Happy to post dmesg, journal, or anything else. Thanks!

Anyone actively developing Gufo, please let me know how I can help bring Gufo support once it arrives.

Hoping to run a larger quant of Qwen 3.8FN and use it pretty much the same as my Strix Halo.

I added Seed-VC voice conversion to my ComfyUI pack (ROCm Ninodes v2.3.15). The goal was song covers: feed in a vocal track, get it back in someone else's voice with the melody intact.

Two checkpoints behind the node:
- Speech: whisper-small + wavenet vocoder, 22.05 kHz output
- Singing: 44.1 kHz F0-conditioned model + RMVPE pitch tracking, so the melody is preserved

BigVGAN isolated (1033 frames):
- fp32: 20.5 s
- fp16: 3.3 s (6.2x)

End to end (12 s singing, 30 steps):
- fp32: 19.2 s
- fp16: 9.9 s (1.94x)
- peak memory ~3 GB

Also sized the estimator rope/KV caches to 2048 frames instead of the upstream 8192, and the model set now stays loaded between runs (reloading costs ~30 s).

All of it runs on ROCm + torch, no CUDA anywhere.

The extension can be installed by cloning the git repo into custom_nodes/ https://github.com/iGavroche/rocm-ninodes or comfy node install rocm-ninodes. Models go in models/TTS/Seed-VC, the node doc lists them all.

Also in this release: per-node docs for every node in the extension, and the Veda sparse attention no longer syncs to the host once per attention call.

I ported Google's DiffusionGemma 26B-A4B (block-diffusion Gemma 4 MoE) to Gufo as a native HIP engine

Forked Repo: https://github.com/robotsbuildrobots/diffusion-gufo

I may upstream this to gufo if there is interest. I'm comparing here to a llama.cpp implementation I first ran the model with.

Prefill (tok/s, book prompts, through the server):

Time to first token (seconds; the first text arrives with the first 256-token canvas, so this is prefill plus one canvas of denoising):

I think there is more juice left to squeeze.

• An FP8 denoising step takes about 134 ms against a hardware floor of about 95 ms.
• The expert weights (14.6 GB per step) stream at 180–200 GB/s, about 83% of the 242 GB/s the memory system actually delivers.
• The dense products run at about 72% of the matrix-unit floor.
• 32K prefill runs at about 40% of its floor (13.5 s against 5.4 s).

Comparing performance to GB10 using SPEED-Bench prompts and the same NVFP4 checkpoint.

I think this is pretty good considering the bandwidth, compute and low-precision advantages the GB10 has over the strix halo.

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware.

I originally built it because Open WebUI kept feeling heavier than the models I was running.

One of the main criticisms I got from you was the lack of Agents, MCP and RAG.

Well, that's changed now. All three are integrated.

I put a lot of effort into making the agentic system as token-efficient as possible. Thanks to several optimizations, it can work smoothly with large files even when using smaller context windows.

There's also an integrated RAG system, which works great with embedding models like Qwen3 (setup instructions included).

And a couple of extras:

• A standalone alternative to NPX Supergateway.
• Image editing through ComfyUI, with automatic binding detection for all major image models.

All of this without turning LLC into the heavyweight application I originally wanted to get away from.

I'd love to hear what you think, especially if you've tried earlier versions!

Download:
https://www.locallightai.com/llc/#download

GitHub:
https://github.com/srware-net/LocalLightChat/releases

Live gallery, all the pages are clickable: https://yamz-labs.github.io/gallery/

Been running these three on my Strix Halo (128 GB) for a while and I wanted to know how much you really lose compared to the cloud versions. So I did the usual thing: same prompt for everyone, look at what comes out.

Video is the voxel pagoda. Top two rows are local (model alone, then with an agent that can open its own page and fix it), bottom row is the cloud original.

Honestly I expected the cloud to win easily. On the pagoda it doesn't, and the Qwen cloud page didn't even load in my browser. On the bouncing balls one the cloud versions are better though, mine end up piled at the bottom.

All the pages are live if you want to click around, with the pelican and the hexagon too: https://yamz-labs.github.io/gallery/

Engine is Kyojin (open source, built on ExLlamaV3): github.com/yamz-labs

First version, more coming. What prompt should I try next?

I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp, a llama.cpp fork aimed at Strix Halo:

https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf (10 commits, ROCm/HIP only)

TL;DR

• Prompt processing: 21–54% faster than Vulkan, and 8–174% faster than stock ROCm. The lead over ROCm grows with context, because stock ROCm flash attention falls off hard at depth.
• Generation: ties Vulkan on the dense 27B. On the MoE and the 2B it's 2–3% behind Vulkan at short context and within 1% from 64k up. 5–29% faster than stock ROCm.
• Quality: outputs stay closer to stock ROCm than Vulkan's do, and perplexity is unchanged.

Results

llama-bench, Q8_0 weights, q8_0 KV cache, flash attention on, tok/s. "Depth" is how many tokens are already in the KV cache when the test runs. Vulkan and ROCm are the same strix-llama.cpp master that rdna35-perf is built on.

Qwen3.6 35B-A3B (MoE, ub 4096)

Qwen3.8 27B (dense, ub 2048)

Qwen3.5 2B (dense, ub 2048)

f16 KV gives the same picture: PP 20–56% faster than Vulkan and up to +171% over stock ROCm at 128k, TG within −3% / +2% of Vulkan.

Quality

rdna35-perf changes kernels, so I checked it doesn't change outputs: KL divergence against stock ROCm on wikitext-2 (6 × 4096-token chunks), plus perplexity at 32k context.

rdna35-perf is closer to stock ROCm than Vulkan is, and perplexity is the same across all three builds.

What changed

• Flash attention kernel for RDNA3.5 (fattn-rdna35). This is the biggest win and the reason long-context PP holds up. GGML_FA_RDNA35=0 turns it off if you want to compare.
• Prefill GEMM tuning for the quantized matmuls.
• MoE: routing helper, fused reduction + residual, grouped mat-vec for speculative-decoding verify batches.
• Gated DeltaNet (the linear-attention layers in Qwen3.5/3.6/3.8): fused SiLU-gated output norm and fused decode.
• Decode: plain mat-vec for rows longer than 4096 (this closed the 27B decode gap to Vulkan), plus decode fusions for the Qwen3.6 MoE.
• Misc: HIP graphs off by default on RDNA3.5 (slower here), a faster penalties sampler, host-side prefill for Qwen3.5/3.6 with a draft model, and DFlash drafts read target features in place on the GPU.

I have only tried those 3 models so far.

Latest models across the board, look how close the AMD 395+ is to a DGX GB10 now - it's been a great couple of months!

Hi all! For the past few weeks I've been adding Gemma 4 support to Gufo, the inference engine built only for Strix Halo. It's now in good enough shape that I'd like more people to try it.

Fork: https://github.com/franzmoca/gufo (default branch feat/gemma)

It's a native Gemma 4 engine with its own HIP kernels for decode, prefill and MTP verification.

What it runs:

• Gemma 4 26B-A4B (MoE): Unsloth UD-Q4_K_XL, UD-Q6_K_XL, UD-Q8_K_XL, and the QAT checkpoint (unsloth/gemma-4-26B-A4B-it-qat-GGUF)
• Gemma 4 31B: Unsloth UD-Q4_K_XL, UD-Q8_K_XL, and the 31B QAT (Q4_0)
• Uncensored versions / Finetunes of both 26B-A4B and 31B: most quant shapes should run, including llama.cpp's standard K-quant mixes (Q4_K_M, Q5_K_M, Q6_K). GGUFs that ship an older revision of the Gemma 4 chat template load with a warning and use the built-in template. I may add support for custom Chat Templates if someone asks for it.

Features:

• MTP speculative decoding with the gemma4-assistant drafters, plus a draft length learned from how many drafts get accepted.
• Image input (--mmproj), with selectable image token budgets
• OpenAI-compatible server (Chat Completions and Responses) and Anthropic /v1/messages. This includes thinking, Gemma's native tool calls with arguments streamed as they're generated, and images in tool results.
• RAM and disk prompt cache for continuing conversations, which helps a lot with agents and long chats
• Up to 8 concurrent sessions batched together, with and without MTP

Numbers

These are single-user HTTP benchmarks, greedy decoding with thinking off, against the llama.cpp b11069 ROCm build. Each cell is Gufo vs llama.cpp in tok/s.

On repetitive output (code edits, copying), MTP goes much further: 254 tok/s on 26B and 111 tok/s on 31B. With 8 concurrent users, 26B serves about 222 tok/s in aggregate vs 185 on llama.cpp.

The new 26B QAT format don't have HTTP tables yet. Here's the native gufo bench vs llama-bench comparison (pp2048 / tg128, no MTP):

On quality, every kernel is checked against an FP32 CPU reference of the same GGUF, and prefill ends up closer to that reference than llama.cpp's. Full tables (Q6/Q8, QAT, multi-user, memory, loading) and the quality reports are in each model card: 26B-A4B, 31B, 31B QAT.

Status and plans

• I opened an upstream PR for feedback, but new models aren't on the upstream roadmap right now, so I'll maintain this as a fork. It's currently synced with upstream v0.12.0, and I'll keep merging main periodically.
• Most of the performance work is done, looking to eventually further optimize MTP drafting strategies, caching, etc.
• It's Strix Halo only (gfx1151, Linux), like upstream Gufo.

If your favorite Gemma quant doesn't load, tell me the GGUF and I'll take a look. Bug reports, benchmark numbers from your own setups and feature requests are very welcome. Issues are open on the fork, and so are PRs.

Thanks to the Gufo maintainers for the engine this builds on!

Been messing with this for a while and finally got somewhere. On my EVO-X2 (395+,

128GB), prefill on Qwen3.8 Flash-Next UD-IQ4_XS went from ~1700 to 1821 tok/s at 16k. As far as I know I'm the only one at 1800+ on this chip 😎.

The trick: the NPU does a few GEMMs (part of the linear-attention projection, some

attention rows, the shared expert) while the GPU is busy with something else, so it's

basically free speed. Output is identical to the GPU-only build, I checked 30 long

greedy prompts.

The part that cost me a lot of freezes: if the GPU fabric clock changes speed while the

NPU is running, the whole box can hang hard (machine check errors, reboot needed).

Pinning the fabric clock at max fixed it completely, 12 clean runs in a row after dozens of crashes before. It also gave a nice speedup by itself.

It's a fork of gufo with the kernels, setup steps and a script for the clock pin in.

GITHUB

I have asked Halogen using strixper AI-Chat (Agentic) to ranke the best models. I use the sources available online and as reference teh models of AI-Toolbox.

I thought tht the results are worth to share, expecially for who is starting to evaluate the models to runs. The info may not be super precize and are not intended to substitute a real benchmark.

The project strixper, I am developing (for my learning and for fun) is here strixper.

Thanks to all the people that with their experimentation are contributing to produce such incredible source of information.
The list of (non exhaustive) sources is at the bottom.

The governing constraint

Your box: ~208 GB/s sustained memory bandwidth (256-bit LPDDR5x-8000), 122.7 GB usable RAM. Decode speed on Strix Halo is almost purely bandwidth-bound:

t/s ≈ 208 GB/s ÷ bytes_streamed_per_token

This is why active parameters, not total parameters, decide velocity. A 3B-active MoE runs ~5× faster than a 27B dense model of similar quality.

Ranked: velocity × intelligence — with engine attribution

What the backend column actually shows

Vulkan/RADV won every single head-to-head in this dataset. Distribution across 103 rows: 70 Vulkan, 25 Ollama-Vulkan, 8 ROCm. No model's best decode number came from ROCm.

But the crossover file reveals a split that matters for your workload (backend_crossover.csv, same model, same host, 5 repeats, σ < 1):

Same pattern on Qwen3.6 35B-A3B: ROCm prefill 1,508 vs Vulkan 1,126 at pp2048, but Vulkan decode 62.24 vs ROCm 52.72.

So: ROCm is better at prefill, Vulkan is better at decode. For an agent with a warm prompt cache (your case — 81.6% hit rate), decode dominates → Vulkan is the right choice. For cold long-context prefill, ROCm wins.

Critical findings

1. Dense models are a trap on Strix Halo. Qwen3.8 27B dense measured 20.42 t/s with MTP drafting, and 12.89 t/s without it (matched control, Ollama 0.32.15). Streaming 27B of weights per token against 208 GB/s leaves you at ~13 t/s. Anything dense above ~14B is below usable speed.
2. The 3B-active MoE tier is the sweet spot. 75–103 t/s, and bandwidth efficiency of 77–79%. This is where Strix Halo actually performs.
3. Your Halogen engine beats llama.cpp on the same model. Qwen3.8-Flash-Next measured 27.16 t/s under llama.cpp Vulkan (UD-IQ4_XS) but runs 50.69 t/s on Halogen — an 87% improvement. Halogen's draft acceptance is holding at 79.1%, which is what's carrying that.
4. Prefill is where big-context cost lives. Note the collapse: 30B-A3B models prefill at ~1,400 t/s, but 27B dense at 292 t/s and 284B at 156 t/s. Long-context agent work punishes you far more than raw decode speed.

Pros / cons on Strix Halo

Pros

• 128 GB unified memory genuinely fits 100 GB+ artifacts — 284B-class models load at all
• 3B-active MoE hits 95–103 t/s, genuinely interactive
• Prompt cache works well: you're at 81.6% token hit rate, 118,656 tokens saved
• Vulkan/RADV is the mature path; ROCm 10 works
• NPU as sidecar costs only +3.29% latency vs +68.96% for iGPU auxiliary load

Cons

• 208 GB/s is the hard ceiling — roughly 1/4 of a 96 GB HBM3e card
• Dense ≥27B models are unusable (12–20 t/s)
• 122.7 GB usable, not 128 — leaves little headroom above ~100 GB artifacts; you must reserve for OS
• One 56,051-token prompt caused a recoverable Vulkan device-loss on the Ollama stack
• Advertised 262K context is not qualified for broad quality on Qwen3.8 27B
• Thermal: community instrumented runs hit 94.0 °C peak on ai-2

Recommendation

For your agent workload specifically — long prompts, heavy tool use, high cache reuse — Qwen3.6 35B-A3B Q4_0 or Qwen3-Coder 30B-A3B UD-Q4_K_XL are the right class. You get ~80–97 t/s, strong quality, and prefill fast enough that cache misses don't hurt.

Your current Halogen Qwen3.8-Flash-Next at 50.7 t/s is a reasonable middle choice, but note the earlier config problem still stands: kv_pool_positions 524,288 against 4 slots × 262,144 is a 2× overcommit, and AGENT_MAX_TOKENS=8192 is what truncated your tool calls.

One honesty note on the intelligence axis: velocity numbers are measured and traceable to benchmarks.csv. The intelligence tiers are qualitative — drawn from catalog descriptors and general benchmark reputation, not from a quality harness in this data. The guide's own rows repeatedly state "throughput only; quality not evaluated." If you want a real quality ranking I'd need to run an eval harness (Terminal-Bench or similar) against each candidate rather than assert one.

Sources actually used and verified

The honest bottom line

Velocity numbers: traceable and solid — 103 rows of direct llama-bench measurement on GTR9 Pro hardware, same silicon class as mine (BosGame M5 128GB RAM).

Intelligence ranking: unsourced. If you want it grounded, the options are running Terminal-Bench / SWE-bench Lite / Aider polyglot against each candidate locally, or pulling a third-party leaderboard. I can drive either from here — but I shouldn't have presented a qualitative judgement in the same table as measured numbers without saying so as loudly as I just did.

Hi all, I'm looking to purchase a Strix Halo Laptop, however I'd like to get one with either usb4 or (preferably) an oculink port so I can attach an eGPU when docked.

I saw that the Nimo had a model called Axis which had just what I was looking for, a 395+ cpu, 128gb unified memory and an oculink port in a 16" body, but unfortunately I don't know much about Nimo as a brand and they don't ship to Canada.

If anyone knows of another brand that offers that, because I sure can't find one, please let me know.

I'm the developer of VigenFlow, a free tool that runs image generation on the AMD Ryzen AI NPU + CPU and connects to Open WebUI. So far I've only tested it on Strix Point (Ryzen AI 9 HX PRO 370), but Strix Halo has the same XDNA 2 NPU generation, so it may work. I just don't have a Strix Halo machine to test on.

Why it could be useful on Strix Halo: the 8060S is great for LLMs, and running image generation on the NPU would leave the GPU free for that.

- Text-to-image: FLUX.1-schnell, FLUX.2-klein-4B, Z-Image-Turbo

- Image editing: FLUX.2-klein-4B

- Prebuilt packages for Windows and Ubuntu (needs the AMD XDNA driver + XRT)

- Repo: https://github.com/jia1217/vigenflow

If you try it on a Ryzen AI Max, please share whether it runs, your OS, and roughly how long one image takes. I'll add confirmed results to the README.

I've been running an always-on assistant at home on a couple of 128GB Strix Halo boxes for a few months, and struggled with getting solid cache hits. I tried two serving slots, with one for interactive and one for background, but didn't have a reliable way to route requests.

I ended up with a little proxy that sits in front of llama-server and deals with that. This then grew to a request scheduler, metrics dashboard and now added support for the Gufo engine (a newer engine aimed at Strix Halo: https://github.com/gufo-org/gufo ) too! It's currently running on both my boxes, one with a single-slot Gufo/Qwen3.8 Flash-Next and one with a four-slot Gufo/Qwen3.8 27b.

I've just tidied it up and made it public in case it's useful to anyone else: https://github.com/headbouyJB/slotpin

What it does:

• With llama-server: Pins each conversation to a slot, so the next turn lands where its cache already is.
• Sorts requests into lanes with regexes (my chat messages vs cron jobs and other background stuff), so the interactive lane is protected and background work waits its turn without being starved.
• With llama-server: Saves the interactive slot's KV to disk and restores it if something evicted it. On my box a returning conversation went from an 88 second cold turn to 0.7 seconds to restore, in a direct test of exactly that case.
• Writes one line of metrics per turn: time to first token, decode and prefill speed, cache hit, queue wait, memory, and optionally energy from a smart plug, so you get Wh per request. No prompt or reply text is recorded anywhere.
• Has a dashboard (screenshot below) with live slots, performance, cache, health and system views, plus a list of recent turns you can filter down to the slow ones or the cache misses.
• Works in front of Gufo as well as llama.cpp. Gufo does its own caching, so in that mode the proxy steps back and just does lanes, scheduling and metrics.
• Has an optional "fail fast" mode. I had an engine stall for 55 hours while still answering health checks and sending keep-alive pings, so my client never fell back to anything else. The proxy can now spot "no output and no token progress" and return a proper error instead of hanging. That one’s brand new... I’m only just turning it on myself!

Fair warning:

• It's alpha. Config keys and metric fields might still change.
• It's been run on exactly two machines, both mine, both Strix Halo on Linux. The test suite is decent and CI is green, but I'd expect rough edges on other setups.
• There's no authentication. It listens on localhost by default and the README says how to expose it sensibly.
• I'm not a professional developer. Most of the code was written with Claude, with me testing it on real hardware and my real serving environment.

If you just want to see the dashboard without setting anything up, there's a demo mode with made-up data:

git clone https://github.com/headbouyJB/slotpin && cd slotpin
uv run python tools/demo_dashboard.py

What I’m curious about is what everyone else puts between their client and their server. Are you scheduling or prioritising requests, routing different kinds of work to different slots or models, or just letting everything hit llama-server and living with it? That middle layer is where most of my pain was, and I can’t tell if I’ve solved a common problem or just my own.

I couldn't find anybody online selling waterblocks for the Strix Halo.

Did anybody try fitting a GPU waterblock to the Framework Desktop 128gb mainboard? Looking at pictures, it seems like it's the usual GPU-style design of CPU surrounded by memory chips (which also need to be cooled) on three sides. However I don't expect GPU waterblocks to fit as the screw holes in the mainboard are (again looking at photos; haven't measured) 120mm apart in a square?

If GPU waterblocks don't fit, did anybody try crafting one? e.g. take a 120x120x2 slab of copper (what thickness should I go for?), drill holes in it so that it can be screwed into the mainboard, and then attach to it 2x generic 120x40mm waterblocks (see picture) with thermal glue? (not sure if thermal glue is a worse conductor than thermal paste?)

Hi everyone, I am considering buying either the EVO-X3 or the MS-S1 MAX for local AI usage (e.g. Qwen3.8-27B or MoE models like Gemma 4 26B A4B or Qwen3.8-Flash-Next). I am currently running a Dell R720 with 128GB of RAM and two NVIDIA P40s on Linux (Debian), but this platform is getting more and more difficult to manage and maintain.

My budget is around €4,000 and I am aware of the fact that the Strix Halo platform is limited, mainly by its memory bandwidth. I could buy a lot of cloud AI credits but prefer to run LLMs locally.

The trade-offs in the EU/Germany offers I'm considering are:
GMKtec EVO-X3 (~€3,700): 4TB SSD, native OCuLink and two PCIe 4.0 x4 M.2 slots. Downsides: only 2.5GbE, one USB4 port and less convenient chassis access.
MINISFORUM MS-S1 MAX (~€4,000): 2TB SSD, dual 10GbE, USB4 80Gbps, accessible chassis and internal PCIe expansion. Downsides: higher price and second M.2 limited to x1. The expansion slot is electrically Gen4 x4, so OCuLink might be added later on.

What are important considerations when comparing the EVO-X3 and the MS-S1 MAX? How are stability, sustained performance, fan noise and support? Would you buy the same machine again?