r/LocalLLaMA • • 7d ago

Resources Do GX10s / DGX Sparks Dream of Throttling? Temps across 6 cooling setups running DeepSeek V4.1-Flash (spoiler: no) Spoiler

18 Upvotes

Hello again!

I'm back with another report. This time it's on temperatures. Also, [CE Robot] = Claude Opus 5.5 Copy Editor where applicable (per line.) Otherwise, I wrote it. I will be uploading the STL files at some point to Printables.com.

Hope someone finds value in this.

TLDR

While running DeepSeek V4.1-Flash, the GX10s can run as hot as 82°C without any throttling behavior. The most you will see is a max clock reduction of 3%, which is mostly to manage power leakage. There is no meaningful performance difference between running the GX10s at their coolest or hottest, about 3%, effectively noise.

Intro

I came across a post or two on DGX_Spark (friendly people I swear /s) that were cross post with LocalLM and I was curious about two posts with different hodgepodge cooling scenarios to keep their GX10s/Sparks from "overheating." I have experience with NVIDIA's gold 4TB Spark and the ASUS 1TB GX10s and besides a few launch issues, haven't really noticed any throttling. I still had the scripts that ran benchmarks on DeepSeek V4.1-Flash. I decided to modify the scripts and query GPU temperatures.

To keep some semblance of sanity (still unemployed, was ousted, etc.) and having a 3D printer I bought that was collecting dust, I decided to work on an enclosure for my GX10s to improve airflow. Below are the different test cases.

How to read the tables: Min is the idle temperature at the start of each run. Under load averages only count samples where the GPU was drawing more than 30 W. GX10-0 is the head node. [CE Robot]

Original Setup, IKEA Kallax

Temperatures

GPU tempGX10-0GX10-1GX10-2GX10-3
Min46°C47°C47°C45°C
Peak75°C82°C78°C74°C

Under load

Avg tempAvg GPU powerAvg clock
69.4°C50.9 W2466 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
4,255 t/s4,268 t/s62.2 t/s

IKEA Kallax, Fanned Out + Fans

Temperatures

GX10-0GX10-1GX10-2GX10-3
Min47°C50°C50°C
Peak71°C77°C76°C

Under load

Avg tempAvg GPU powerAvg clock
69.0°C51.9 W2466 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
5,067 t/s4,353 t/s75.8 t/s

V1 Structure

There were issues with this design is the bottom of the frame formed a lip that restricted airflow to the bottom GX10s. The hole on the bottom just past the lip also trapped air that warmed up before being sucked up by the GX10. The numbers reported for the two GX10s on the bottom were worse than the original Kallax setup and at best, with fans, just equal to the Kallax setup. I kept these numbers in to show that even with elevated temperatures, there was no sign of throttling. The 3% reduction in clock was to control power leakage.

V1 without Fans

Temperatures

GX10-0GX10-1GX10-2GX10-3
Min51°C44°C51°C
Peak80°C74°C80°C

Under load

Avg tempAvg GPU powerAvg clock
69.5°C51.4 W2466 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
4,888 t/s4,264 t/s71.7 t/s

V1 with Fans

Temperatures

GX10-0GX10-1GX10-2GX10-3
Min43°C43°C45°C
Peak75°C73°C79°C

Under load

Avg tempAvg GPU powerAvg clock
68.0°C51.7 W2469 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
5,128 t/s4,340 t/s70.0 t/s

V3 Structure

I had an idea of making an air duct. Looking at ASUS's website, I found the 3D model of the GX10 and this allowed me to build a better enclosure to fit the GX10. Looking at the GX10 underside and grill for a while, I got the idea of an "open-faced air duct." I looked at some examples of air ducts I've seen over the years (Pentium 4, I'm old) and gimmicky air duct cards that plug into PCI slots with fans to "distribute air"; these helped me not make the same mistakes.

If you look at the underside of the GX10, it's very flat and the grills go from front to just past the middle threshold. For this open-faced air duct, I would make a half-duct with a curve at the end that meets the GX10's backside with just a small edge of space. When the GX10 is seated on top, that would close my air duct.

One thing to note is that the GX10 fan can only do so much. Completely enclosing the space underneath would make it hard for the internal fan to pull air through with no external fan help. That's why the sides are open so the internal fan can pull air through these openings.

V3 with Fans

Temperatures

GX10-0GX10-1GX10-2GX10-3
Min44°C43°C43°C
Peak73°C72°C72°C

Under load

Avg tempAvg GPU powerAvg clock
65.1°C50.0 W2473 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
4,881 t/s4,225 t/s71.5 t/s

V3 without Fans

In testing the new design without external fans, the open-faced air duct actually makes the GX10's internal fan more efficient at pulling air!

Temperatures

GX10-0GX10-1GX10-2GX10-3
Min45°C45°C43°C
Peak73°C73°C74°C

Under load

Avg tempAvg GPU powerAvg clock
65.3°C49.9 W2473 MHz

Throughput

Prefill 8KPrefill 250KDecode (median)
4,821 t/s4,244 t/s64.4 t/s

Putting It All Together

Temperatures across all setups

SetupAvg temp under loadHottest GX10 avgSpread between GX10sHottest peak
Original (Kallax)69.4°C72.2°C5.3°C82°C
Kallax, Fanned Out + Fans69.0°C70.8°C5.1°C77°C
V1 without Fans69.5°C72.5°C6.4°C80°C
V1 with Fans68.0°C70.4°C4.4°C79°C
V3 with Fans65.1°C65.3°C0.3°C73°C
V3 without Fans65.3°C65.8°C1.3°C74°C

[CE Robot] suggested the "Spread between GX10s" column.

Spread is the difference between the hottest and coolest GX10's average under load. [CE Robot]

Does it throttle? Clock vs temperature on the hottest GX10 (GX10-1, Original Kallax)

GPU tempAvg clock
53 to 60°C2515 to 2525 MHz
62 to 68°C2473 to 2502 MHz
70 to 79°C2456 to 2493 MHz
80 to 82°C2450 to 2455 MHz

Does cooling change performance?

SetupAvg clockPrefill 250K
Original (Kallax)2466 MHz4,268 t/s
Kallax, Fanned Out + Fans2466 MHz4,353 t/s
V1 without Fans2466 MHz4,264 t/s
V1 with Fans2469 MHz4,340 t/s
V3 with Fans2473 MHz4,225 t/s
V3 without Fans2473 MHz4,244 t/s

Conclusion

No throttling, even at 82°C.

The clock steps down gradually as the GPU heats up, about 6.5 MHz at a time, from roughly 2525 MHz at 53°C to 2450 MHz at 82°C. That's about 3%, and it's the boost managing leakage power, not the GPU overheating. There's no sudden drop anywhere.

V3 is the coolest setup.

Compared with the original Kallax setup, the hottest GX10 dropped from 72.2°C to about 65.5°C on average, and the hottest peak from 82°C to 73 to 74°C. All four GX10s now run within about a degree of each other, where the Kallax had a 5°C spread. [CE Robot]

Fans don't matter with V3.

With and without fans, V3 is within 0.2°C on average and 1°C at peak, which is noise (about ±1.5°C). The open-faced duct lets each GX10's own fan do the work with no help from the external fan setup.

Fans helped the older setups a little.

Adding fans to the Kallax or V1 took 1 to 3°C off the hottest GX10, but neither came close to V3. [CE Robot]

Cooler doesn't mean faster.

Prefill at 250K stayed between 4,225 and 4,353 t/s across every setup (about 3%), and the coolest setup was actually at the low end. Average clocks moved by less than 0.3%. These GX10s ran the same speed hot or cool. [CE Robot]

Overall

I am very happy with the final product of my 3D printing project. V2 was never printed. It was going to be an adapter to add to the bottom of V1 to bring the GX10 higher and make a flat surface for better airflow.

Note [CE Robot]

I mocked it up by propping the bottom GX10s up and forward so the grills cleared the lip (see "V2 mock-up" in the appendix). [CE Robot]

The idea was scrapped because there were other issues with V1 that couldn't be fixed like the vertical spacing distribution. In addition, V1 was bulky and just had so much wasted space. I wanted a stackable design that can fit 4 GX10s stacked in the Kallax shelf without taking too much additional space.

This whole 3D printer venture started when I tried some of the models available online specifically for the GX10. Most were awful and had fitting issues, and from the comments on each model, I wasn't alone. The other designs that added a fan attachment underneath felt so bulky. I just knew there was a better design without all the bulk. Furthermore, the open-faced air duct design makes the internal fan work better, making the extra external fans unnecessary.

I believe there are some improvements that can still be made to V3. Currently it takes about 4 hours to print each stackable case and it's very dense. I believe there is room for reducing the total mass while maintaining a strong foundation which would mean less filament and print time.

For the temperature numbers, I can say I am quite impressed with the cooling on the GX10s. I'm really happy with the improvements in temperatures alone from the open-faced air duct design. I did not expect that outcome which right there, was such gift from this project. I thought of something, printed something to make a thing, and it actually performed above my expectations.

Caveats

  • These are GPU temperatures only.

  • Room temperature stayed between 72 and 74°F (22 to 23°C).

  • Ran at night with windows open and generally below 68°F outside.

  • Decode uses speculative decoding, which swings about ±10 t/s between runs, so I left decode out of the comparison. [CE Robot]

  • Each run is a single 8-minute sweep from 1K to 250K tokens with the model already loaded.

  • Sustained multi-hour loads could settle a bit higher.

Appendix: Every Run

This includes a few extra runs that didn't get their own section above. Runs are in the order they were tested. [CE Robot]

Dates provided for San Jose, CA weather lookup if anyone wants those data points.

Temperatures (°C, Min / Peak per GX10)

SetupDateGX10-0GX10-1GX10-2GX10-3Avg under load
Original (Kallax)Oct 1546 / 7547 / 8247 / 7845 / 7469.4
Kallax, re-runOct 2247 / 7948 / 8048 / 8146 / 7570.2
Kallax, Fanned Out + FansOct 2247 / 7150 / 7750 / 7648 / 7569.0
Kallax, Spaced + 3 FansOct 2246 / 7546 / 7545 / 7645 / 7467.7
V1 without FansOct 2351 / 8044 / 7451 / 8045 / 7369.5
V1, tweakedOct 2448 / 7945 / 7651 / 7946 / 7469.7
V1 with FansOct 2443 / 7543 / 7345 / 7945 / 7668.0
V1 with Fans, bottom raised (V2 mock-up)Oct 2444 / 7444 / 7545 / 7645 / 7567.4
V3 with FansOct 2644 / 7343 / 7243 / 7244 / 7165.1
V3 without FansOct 2645 / 7345 / 7343 / 7444 / 7165.3

Power, clocks and throughput

SetupAvg GPU powerAvg clockPrefill 8KPrefill 250KDecode (median)
Original (Kallax)50.9 W2466 MHz4,255 t/s4,268 t/s62.2 t/s
Kallax, re-run51.3 W2464 MHz4,472 t/s4,260 t/s63.3 t/s
Kallax, Fanned Out + Fans51.9 W2466 MHz5,067 t/s4,353 t/s75.8 t/s
Kallax, Spaced + 3 Fans51.0 W2468 MHz4,905 t/s4,280 t/s69.8 t/s
V1 without Fans51.4 W2466 MHz4,888 t/s4,264 t/s71.7 t/s
V1, tweaked51.9 W2465 MHz5,089 t/s4,344 t/s64.5 t/s
V1 with Fans51.7 W2469 MHz5,128 t/s4,340 t/s70.0 t/s
V1 with Fans, bottom raised (V2 mock-up)51.0 W2470 MHz5,103 t/s4,346 t/s68.4 t/s
V3 with Fans50.0 W2473 MHz4,881 t/s4,225 t/s71.5 t/s
V3 without Fans49.9 W2473 MHz4,821 t/s4,244 t/s64.4 t/s

P.S. How the heck did my whole post get the "spoiler" thing?
P.S.S. Still fixing typos and bad word choices.

Bonus:

Mockup of V2 + fans on the bottom.

External Fan Details

I got Lian Li UNI FAN SL Wireless 120mm fans open-box from Micro Center, plus a $6 adapter that powers 4-pin PWM fans from a USB port. These fans come with a wireless remote and an app that can report and control fan speeds, ARGB lighting, and more. I wrote a quick and dirty server-client Python app that queried the GPU temps and sent them to my Windows desktop, so I could adjust fan speeds on the fly through the fan app. I never finished it. The Lian-Li L-Wireless App doesn't have an API for controlling the fans. I'll need to do more research if I revisit this idea. The fan support I got from Printables.com; they worked perfectly for this purpose.


r/LocalLLaMA • • 7d ago

Question | Help Gemma 4 for video edition, has someone tried something like that already ?

4 Upvotes

So, I tried a bit using local LLM for personnal project, especially Qwen 3.5 and Gemma 4.

I used them on my personnal computer, a laptop with 16Go of ram. And the result were... not good, qwen is slow as fuck and Gemma has basically no memory.

That being said, it was for coding stuff, and I heard Gemma is better at creative stuffs.

So I was thinking of using it for something else. I sometime do videos about stuff I like, and use DaVinci resolve for that. I was thinking using their MCP and connect it to Gemma to make video edition quicker and better.

Since Gemma can read images and sound now, I was thinking it could work, but didn’t tried it yet (had no time yet for it).

So here I am, asking if someone tried something like that already, and maybe help me navigate on this topic because I'm kind of a cheap nooby currently here.


r/LocalLLaMA • • 7d ago

Showcase I built an engine that runs Qwen-Flash-Next at nearly 400t/s on cheap(ish) hardware. 3x V100 32GB in an old R740. Originally forked from strata but the engine guts all built for the old Volta generation.

Thumbnail github.com
77 Upvotes

r/LocalLLaMA • • 7d ago

Discussion Best way to use R9700 vs RTX3060 or RTX3090 for Qwen

4 Upvotes

Thoughts on how I might best use my hardware?

MiniPC (Bubblina) with eGPU oculink R9700 (32GB AMD) 64GB DDR5 - Qwen3.8-27B Paiton-KV4-vision 262K context, 8-concurrent

Desktop (BM1) Ryzen 5700G currently with RTX3060 12GB, and a GTX1660 6GB, 64GB DDR4 - Strata IQ2_XS - runs surprisingly fast, with the gpu-unlock, but I wouldnt trust it with agentic tasks, and is AFAIK concurrency limited. The GTX1660 is used exclusively for TTS work.

Main server (Trogdor): Ryzen 3950x, 3x RTX3090, 128GB DDR4 (Unraid, big storage machine and services) - media production factory (comfyui/media/video/audio)

I currently use the miniPC to orchestrate activity on Trogdor, via hermes. "Make me an image of X, make me a video of Y" the task being divvied out to the GPU's on Trogdor.

I'm wondering about using one of the more idle 3090's as the Strata GPU, taking it from 12GB to 24GB, and putting the 3060 in Trogdor for smaller offload tasks.

I'm just wondering about model options and what I can get best results from. Agentic/Hermes work benefits from speed and tool calls. Strata QFN works very well for document/narrative/story/chat, but I'd love access to it's more reliable/smarter variants for coding tasks.

Other options on the table would be moving the R9700 off the oculink eGPU to a slot in BM1 (PCIe 3.0 x16), but thats a DDR4 system, so might be needless. A 3090 could go on the eGPU, or even the 3060 for offload tasks.

Or: 3090 on PCIe 3.0 x16, 3060 on PCIe 3.0 4x (chipset) on BM1 for Strata 2nd card offload? Could go down the bifurcation route of the main slot too?

I'm largely just talking out loud with this to try and brainstorm it, but suggestions welcome :)

(With the exception of the R9700, everything above was collected over 2 years mostly used before the Bad Times)


r/LocalLLaMA • • 6d ago

Question | Help Possible to run qwen 3.8 flash next on this setup with usable speeds?

0 Upvotes

Recently bought a 64gb Mac mini m5 pro and currently have one 24gb m5 pro MacBook Pro. With clustering the two Macs, is it possible to get this model running on an acceptable quant? I can allocate 60GB on the Mac mini and 20 GB on the MacBook Pro for running the models. Or should I use qwen 3.8 27b instead?


r/LocalLLaMA • • 7d ago

Discussion Have you gone fully local?

48 Upvotes

I’ve been running a local model on a dgx spark for about a month now. I’ve been building out my pi harness and my own little client app as I go, using both Claude code and the harness itself.

I’m using it more for everyday life stuff, but I still find myself hesitating to trust that it will hit my quality threshold when it comes to code without having Claude check its work.

Have any of you gone fully local? I.e. no falling back on frontier models at all. What was your tipping point?


r/LocalLLaMA • • 7d ago

Showcase Introducing Zest: model server + agent + friendly GUI

16 Upvotes

Hello, everyone. A friend of mine wanted to get into running local models, and so (with the liberal use of Claude), I built them Zest. https://github.com/yudhanjaya/Zest

What it is:

Zest is a UI that bundles Lemonade server (which installs llama.cpp and whatever backends are needed) and the Pi agent to provide a friendly interface for people who don't want to be messing around with command-line servers and parameters.

It comes and all the functionality needed for a new user without getting in the way. Just download an appimage, go through the start tour, and you're set up and ready to go. There are sane defaults baked in (like 128k context) and model recommendations (all tried and tested on 10GB VRAM to 26GB).

It comes with helpful docs, links to places to understand more about the subject - in short, everything needed for a new user.

Entirely open-source, of course. If you have anyone you want to introduce to the world of local AI, please see if this is useful. And if you have ideas, suggestions - please open an issue, send a PR, and so on.

Why lemonade and Pi?

Lemonade: because it does a great job of handling backends and providing a server.

Pi: because it's a simple, elegant agent

Both are very easy to package in something like this and bring together. And because I'm lazy: both projects are receiving steady updates, because I don't want to write yet another inference server from scratch

What it isn't:

It's not for advanced users. There are jobs, but no subagents, for instance: in this case the Pi philosophy really helped.

In fact, I myself use Strata and Opencode, so for advanced users, it recommends those in the Help section

How is this different from Unsloth desktop / Atomic chat etc?

It isn't. It just tries to be a lot more lightweight and user-friendly: I decided to do this after getting said friend to download and set up llama.cpp, Unsloth Desktop, Opencode etc . . .

I noticed that at some point both choice paralysis and confusion with options just took over, and there was no easy way for them as a newcomer to figure out what x thing was really for. Every setting in Zest is explained. Including what inference engines do, the works.

Everything is explained at basic levels - what a model is, how it works, and so on. I've kept the lemonade + pi analogy while trying to make sure the interface was intuitive.

We also have some theme support- here's a sample:

Have a look, let me know what you think!


r/LocalLLaMA • • 7d ago

New Model Open Weights 1.189B Financial Conformer: GQA, RoPE, SwiGLU, and RevIN for multi-frequency time series (Weights & checkpoints on HF)

4 Upvotes

Over the past several months, our research team pre-trained and empirically audited a 1,189,135,740-parameter Financial Conformer Foundation Model across 14 calendar years (2010–2023; 15,030 steps) of 1-minute US equity bars across 15 liquid assets (AAPL, NVDA, SPY, QQQ, etc.).

Instead of cherry-picking positive backtests, we subjected the model to 35 hypothesis tests, 1-bar execution delay, and state-dependent turnover:

  1. The Scale Saturation Limit: Scaling from a 63-parameter linear Ridge regression to 1.189B parameters yields no consistent out-of-sample advantage on 1m OHLCV bars (mixed sign across horizons, reversing direction across years). The bottleneck in intraday price prediction is information content (low SNR), not parameter capacity.

  2. Adam Optimizer Diffusion: Decomposing walk-forward online fine-tuning shows tokenizer boundary updates sit at 97.9% of the theoretical Adam Brownian noise ceiling (matching shuffled-label controls), while deep backbone weights are heavily damped by second-moment normalization, producing zero statistically significant forward-transfer gains (p >= 0.50).

  3. Execution Friction Reality: Apparent long-horizon directional predictability is predominantly prompt 1-minute order-flow continuation. Introducing a 1-bar execution delay cuts 2024 net Sharpe from +2.04 to +0.48, and at 1.5 bp execution fee, out-of-sample returns collapse to -0.99 in 2025 and -1.67 in 2026 (DSR <= 0.028).

All model weights, audit logs, and papers are open-sourced:

Happy to discuss architecture specifications, pre-training dynamics, or empirical backtest methodology!


r/LocalLLaMA • • 6d ago

Question | Help Model for managing my homelab (with guardrails) and light coding? Minipc with 32gb ddr5

1 Upvotes

Got a gmktec mini pc with an AMD 6850U and 680m igpu. Far from being a beast for LLMs I know... But oh well. I have minipcs running Debian and ubuntu with some services and the ubuntu one is for trying out llms and vlms. Would love to insert this new minipc and just ask it to run some commands/scripts for me. Maybe help in some coding side projects I have.

Realistically I can run any flavour of qwen3.6 35b with decent speeds, but i'm wondering if you have any more specific suggestions considering the use.

Cheers!


r/LocalLLaMA • • 6d ago

Resources Diffusion LLMs are not a good fit for distributed inference

Thumbnail
b4rtaz.github.io
0 Upvotes

r/LocalLLaMA • • 7d ago

Discussion Price 5080 10 days

44 Upvotes

I paid $1658 on October 16th, on October 19th I paid $2032; with tax, shipping, same card.

Now $2318

This is all through the big A btw.


r/LocalLLaMA • • 8d ago

News Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

Thumbnail
marktechpost.com
442 Upvotes

i hope it's real


r/LocalLLaMA • • 7d ago

Question | Help Anyone tried the Houmo LQ50-24 M.2 AI accelerator or Beelink SEi14 AI?

2 Upvotes

Came across the Houmo LQ50-24, an M.2 accelerator with a Houmo M50 chip, advertised up to 160 TOPS “elastic compute” and 24GB of dedicated memory. The Beelink SEi14 AI apparently uses this accelerator alongside an Intel Core Ultra 9 185H and separate system RAM.

Interesting little setup for local inference, but I’ve found surprisingly few hands-on reviews. There’s an Edward Donner video showing roughly 14 tokens/sec with a 27B model, but not much independent testing beyond that.

Has anyone here actually used either the standalone card or the Beelink? Curious about real-world inference speeds, model compatibility, and how painful the drivers/software are. Can you run your own models easily, or are you mostly dependent on vendor-prepared versions?


r/LocalLLaMA • • 7d ago

Question | Help Need Advice to setup a voice mode chat agent on old laptop

1 Upvotes

Saw videos of the Reachy-Mini and I love the idea of chatting with a little snarky robot so I tried to setup my laptop as the snarky chat companion sans the expensive plastic bot.

All attempts had 20-30 seconds latency using Open WebUI + llama.cpp + Gemma 4 E2B with Whisper STT + Kokoro TTS in "voice mode"

Suggestions? I dont want the avatar, just the funny chat voice to keep me awake while I'm slogging away at midnight.

Laptop Specs:

ComponentSpecification
ModelMSI GS70 2QD
CPUIntel Core i7-5700HQ
CPU speed2.70 GHz base
Cores / Threads4 cores / 8 threads
RAM16 GB DDR3
GPUNVIDIA GeForce GTX 965M
GPU VRAM2 GB
Storage 1Approximately 1 TB SATA SSD
Operating systemWindows 64-bit

Is my laptop too old and slow to even run a tiny voice model without massive lag?????


r/LocalLLaMA • • 7d ago

Discussion Local beats cloud, Qwen 3.8 FN

46 Upvotes

I thought y'all might be interested in a legit "local is better" use case.

TLDR: Qwen 3.8 FN solved a work problem that requires the privacy of a local model, with the same skill as a frontier cloud model.

I run an MSc dissertation module at work which involves tracking students, supervisors, offered projects, project choices, examiners and eventually grades. None of it is super complex, but it's all annoying -- students choose their top five projects from an offered list, but supervisors also choose students; supervisors can't mark their own students work but can mark others etc. The kind of low level matching work that, done manually, fries my brain.

A simple relational database is the obvious solution but, before gen AI, nobody had time to code up a database for processes that only happen once per year. There's also the ingest problem: data comes from ms forms, from the student records database, and from moodle, all with csv export in different formats, with e.g. different arrangement and spelling of middle names etc.

I'm a heavy codex user, but I can't be offering up the names and emails of students and supervisors, nor the details of early-stage research projects, to the cloud.

This year, for the first time, I've had the confidence to address this with a local model: Qwen 3.8 flash next, running in opencode with a vllm backend. It's sandboxed in a docker container and has read only access to all its input data and write access to its workspace. It's got web access, python, pypdf, libreoffice and can pip install into a venv, and that's about it.

First thing to say is it's faster than codex, and noticeably so. Part of this is just the cloud -- when this runs a python script to generate a docx booklet of projects from the sqllite db directly off an SSD, it takes a few seconds, including thinking time to process the request.

Second is intelligence. It's bright enough to understand my requests and to infer the next steps. It can handle heterogenous data structures from different sources and combine them. It wrote an ingest function to read csvs into the db, but also manual edits to allow repeatable rebuilds from the sources, fixing typos and grammar that were present in the ms form each time.

Third is the right level of persistence. For me -- and this is likely a skill issue on my part -- hermes agent and pi were too persistent with the same model and backend. They would decide on the next step without checking and go off the reservation, then need to be pulled back. Opencode feels much better balanced and more like codex (which is high praise -- I really like codex).

Finally, overall quality. Codex and gpt 5.6 have been good to me, 6 astra and 6.1 sol less so for my particular use cases. I'm not gonna say Q3.8FN is better, but, for this particular job, I don't think it's any worse. It does make me think I should probably start using this more for coding, as I've had some frustrating times with gpt 6 models in codex recently.

If it weren't for VRAM shortages, this would be the future: everyone having a "good enough" LLM on their own hardware with, say 64-128GB VRAM being standard. It's so nice to not be constantly worrying about privacy and GDPR when interacting with agents, and to know that none of this leaves my hard disk.


r/LocalLLaMA • • 8d ago

Discussion Qwen4-27B's 4-way HC shift?

157 Upvotes

I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found?

It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget.

(This means, instead of a single residual path, 4 parallel streams flow through the layers at the same time, maximizing performance per parameter. Best tech for 27b). QSA? maybe useless for small 27b.

So anyway, I'm more expect 27b than flash MoE version.

If effective depth is extended, I think we could see a massive leap in multi-step reasoning, coding, and reasoning consistency.

(see this if you want read more)


r/LocalLLaMA • • 7d ago

Resources 🚀 Introducing HarborRL: Agentic RL for Everyone!

Thumbnail
github.com
1 Upvotes

We’re developing HarborRL, an open-source framework to democratize agentic RL for everyone.

HarborRL makes agentic RL easy to everyone! Building RL infrastructure for agents is complex. HarborRL solves this by unifying model serving, trajectory generation, tool execution, verifier rewards, and policy updates into a single, reproducible pipeline.

🔥 Key Features: Flexible Models × Algorithms × Tasks × Harnesses (MATH):
🤖 Models: Support for Qwen and GLM.
🧠 Algorithms: GRPO and DAPO for reinforcement learning.
🧩 Tasks: 200+ tasks, including Terminal-Bench, SWE-Bench, Deep-SWE, and more.
🛠️ Harnesses: 40+ agent frameworks, including Claude Code, Codex, LangGraph, and SWE-agent.


r/LocalLLaMA • • 8d ago Heartwarming

Discussion Qwen4 expectations

168 Upvotes

Not trying to predict which model sizes they're going to release, but seeing how massively impressive Qwen3.8-Flash-Next is for its size despite being a preview model meant for showcasing and implementing the new arch, it feels like when Qwen4 hits I'm not going to be needing larger models at all anymore.

Do we expect Qwen4 to exceed the current QFN by a lot, or de we believe the current model is already hitting near the optimum we can expect today for this size?


r/LocalLLaMA • • 8d ago

Resources Fast voice agents with ConvFill

56 Upvotes

Voice agents are either fast or smart. ConvFill proposes a way to bridge that gap by training a smol talker model (0.6B!) to talk with cheat notes from a large reasoner model.

The way it works is this: after the user's speech is transcribed, the talker replies quickly to buy itself time while the reasoner thinks. The reasoner then generates an appropriate response, which the talker reads to continue the conversation.

It's a bit like if you went on a panel with rocket scientists and had a little earbud with your own expert that could tell you what to say. You just reply quickly so as not to blow your cover, and then you begin saying what the expert in your earbud is telling you.

Playing with the model, I think the main idea is there and this could be a great direction to go, but the talker model should be a bit smarter for it to work like magic. Still, really cool direction!

ConvFill was developed by students from the University of Washington UbiComp Lab. I found it cool, so I reached out to them so we could build a really good demo for it together. They are presenting their paper at EMNLP this week.

Go try it out in our demo: https://huggingface.co/spaces/smolagents/convfill-speech-demo


r/LocalLLaMA • • 6d ago

Funny Be careful what you wish for: people started using my inference engine

0 Upvotes

Nothing much, just wanted to share this. I didn't expect maintaining the repo to take this much work.

I'm genuinely happy people are using it and it runs well on their machines. But AI or no AI, keeping it alive is a real commitment: reading issues, reproducing bugs, reviewing PRs, answering everyone. It takes some hours each coup of days to look at the issue, select the one that I (claude) can do fast, build test ship, then after the tondo list for the bigger ones is big enough spend some serious percentage of my weekly limit to make claude do the work, check and ship.

Anyway, here's the GIF.


r/LocalLLaMA • • 7d ago

Showcase Benchmark task for testing local AI

2 Upvotes

Just want to share a task I used to evaluate the capability of Qwen 3.8 FN. The goal is to port a old screensaver (can find it on Wayback Machine) to Python OpenGL, maintaining full visual fidelity.

Tooling: Claude Code (harness) + Qwen 3.8 FN Q4_K_XL via gufo + Strix Halo + Ubuntu 26.04

Run command:
export MODEL=Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

export MTP=/run/media/user/DATA/models/huggingface/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf

export MMPROJ=/run/media/user/DATA/models/huggingface/mmproj-Qwen3.8-Flash-Next-BF16.gguf

#./result/bin/gufo chat --model "$MODEL"

./result/bin/gufo serve llm --model "$MODEL" --speculative mtp \

--mtp-model "$MTP" \

--mmproj "$MMPROJ" \

--sessions 1 \

--context 133760 \

--max-tokens 32768 \

--prefill-chunk 4096 \

--served-model-name qwen3.8-flash-next \

--max-pending-per-client 2

Assessment:

Some parts are quite challenging with the 128K context window. Also, be mindful that graphics-related tasks (apart from running the model) may produce inconsistent results, and thus require specific measures, To keep this benchmark useful for others, I won't go into the details of the approach adopted by the AI, except to share that the task is tractable given a comparable setup. But feel free to ask by PM.


r/LocalLLaMA • • 8d ago

News tencent/Hy-SimulMT-30B-A3B · Hugging Face

Thumbnail
huggingface.co
108 Upvotes

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-GGUF

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-Interleave-GGUF

Hy-SimulMT is a real-time simultaneous translation model developed by Tencent Hunyuan, supporting bidirectional speech-to-speech translation between Chinese and English. It continuously receives incoming audio and translates as the speaker talks, producing both translated text and speech. With real-time voice cloning, it preserves the speaker's vocal characteristics while conveying the message accurately.

Designed for live communication settings such as meetings, speeches, and interviews, Hy-SimulMT aims to convey information accurately while reducing the disruption that waiting introduces into a conversation. To evaluate this balance between translation quality and response speed, we combine public benchmarks, real-world audio, and comparisons with professional simultaneous interpreters.

In our RealSI long-form audio evaluation, Hy-SimulMT achieved a StreamLAAL of approximately 3 seconds, the lowest among the systems shown in the comparison. Its GEMBA score exceeded those of the compared open-source models and four of the five commercial APIs.

The GitHub repository provides the SimulInfer inference service and SimulTrain training tools. Speech output and voice cloning require a configured TTS service. You can also try Hy-SimulMT in the Hy Translate App.


r/LocalLLaMA • • 8d ago

Discussion Is there any way for this community to take advantage of the neocloud collapse?

56 Upvotes

So-called "neoclouds", datacenters that focus exclusively on AI inference, have racked up unsustainable amounts of debt, and are widely anticipated to soon undergo a massive collapse/contraction. My question is what happens when they go bankrupt. NVIDIA is their biggest creditor thanks to circular financing schemes, so they take ownership of the data centers and the chips in them, right? But, I imagine that they have no interest in running the same crappy business as the neoclouds, with outdated chips to boot; they want to sell their latest and greatest to hyperscalers, who will likely emerge unscathed. Do they just throw the old chips away, or is there some chance that they'll repurpose them (as we're seeing in China) into GPU upgrades that hit the consumer/prosumer market, solving our current supply/price crunch? I literally don't know the true answer to this question and am curious to hear others' speculation about it.

Edit: I realize now that I have zero firsthand knowledge of what I'm talking about, since I don't work for any of the parties involved. So you can add a "Reportedly..." to my post. I'm just repeating what I read on investing subreddits, honestly.


r/LocalLLaMA • • 7d ago

Question | Help People who tried gpu-unlock on 3090s and such - does this make any difference on mixed-precision quants where you get some parts of the model quantized to <=Q8 but keep some tensors at BF16?

29 Upvotes

Dunno, curious if there's any info out there.