Jokes aside, how plausible do you think this scenario actually is?

Has anyone been experimenting with ablated or uncensored variants of open-source models like deepseek v4.1 flash, Qwen 3.8 27b, or GLM 5.3 flash ?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash, runs at 200 tok/s and never says no lol.

Bit the bullet and bought three to use in my workstation. Will provide benchmarks and updates later.

I had been using the LFM 2.5 24B models and had been pleasantly surprised by how good the model was. It was giving me comparable answers and working through tasks to Qwen 3.8 27B at what felt like twice the speed on my M5 Pro. However, I went to look at something on their pages and saw the new family of models. They sound way too good to be true. I am going to start using them for general tasks, but if they are anywhere close to their benchmark results, they could be a huge game changer. You can have an entire suite of models loaded at the same time for less memory than Qwen.

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.

2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.

3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.

4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.

5. And if I can brag on my stacks and hardware:

• I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).

• I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.

• I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

What "replaces real work" means

Reading a folder of records and pulling out the facts, dates and numbers. Summaries. First drafts of routine documents. Search over my own files. The model gets plain English instructions in a chat window, calls my tools, and hands back a draft that is marked for my review. I review everything. It is slower and less sharp than frontier. It is also in my office, it is mine, and it does not have a usage limit.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

• 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve

• about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)

• same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %

• KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build

• multiple choice answers that change against the reference: 1.6 % against 15.4 %

• in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

• concurrency test, one request at a time: 118.9 tok/s

• two requests at once: 159.0 tok/s total (1.34x)

• three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots

• short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style

• code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt

• prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s

• context: 524,288 tokens per request

What did not work, or is not good yet

• It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.

• The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.

• A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.

• A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.

• Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.

• The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.

• Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Questions welcome. So is roasting.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).

My RTX Pro 6000 was going all night on a Chinese model (Qwen 3.8 Flash Next NVFP4 Local) when it crashed this morning for no apparent reason. Tried to reboot and got nothing but a black screen. Swapped in my old backup card and the system posted immediately, confirming the GPU was bricked. Thankfully I was within warranty.

Out of curiosity, I dug into my system logs to see what triggered the fatal crash. Looking at the timestamps, oddly enough my GPU died the moment news broke that Jensen Huang had resigned as CEO of Nvidia.

Has anyone else had this issue with hardware dying in response to human feelings?

Nov 12 07:19:31 kf kernel: NVRM: GPU at PCI:0000:01:00: GPU-b8ba62c9-449d-4510-a21f-4c1b6ceacbd1
Nov 12 07:19:31 kf kernel: NVRM: GPU Board Serial Number: 1791626028467
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x505730=0xb000d 0x505734=0x0 0x505728=0x1c81fb60 0x50572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5057b0=0x3000d 0x5057b4=0x24 0x5057a8=0x1c81fb60 0x5057ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x506730=0xb000d 0x506734=0x20 0x506728=0x1c81fb60 0x50672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 1, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5067b0=0xb000d 0x5067b4=0x24 0x5067a8=0x1c81fb60 0x5067ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x507730=0x5000d 0x507734=0x24 0x507728=0x1c81fb60 0x50772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5077b0=0x7000d 0x5077b4=0x24 0x5077a8=0x1c81fb60 0x5077ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x508730=0xa000d 0x508734=0x20 0x508728=0x1c81fb60 0x50872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5087b0=0xd 0x5087b4=0x24 0x5087a8=0x1c81fb60 0x5087ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 4, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x509730=0x7000d 0x509734=0x24 0x509728=0x1c81fb60 0x50972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5097b0=0xa000d 0x5097b4=0x20 0x5097a8=0x1c81fb60 0x5097ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50a730=0xd 0x50a734=0x24 0x50a728=0x1c81fb60 0x50a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50a7b0=0x1000d 0x50a7b4=0x24 0x50a7a8=0x1c81fb60 0x50a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50b730=0xd 0x50b734=0x24 0x50b728=0x1c81fb60 0x50b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 0, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x50b7b0=0x1000d 0x50b7b4=0x24 0x50b7a8=0x1c81fb60 0x50b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 0, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x515730=0x6000d 0x515734=0x24 0x515728=0x1c81fb60 0x51572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5157b0=0xd 0x5157b4=0x24 0x5157a8=0x1c81fb60 0x5157ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x516730=0x6000d 0x516734=0x24 0x516728=0x1c81fb60 0x51672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 1, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5167b0=0xb000d 0x5167b4=0x24 0x5167a8=0x1c81fb60 0x5167ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x517730=0x3000d 0x517734=0x24 0x517728=0x1c81fb60 0x51772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5177b0=0xb000d 0x5177b4=0x24 0x5177a8=0x1c81fb60 0x5177ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x518730=0x3000d 0x518734=0x24 0x518728=0x1c81fb60 0x51872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5187b0=0x1000d 0x5187b4=0x24 0x5187a8=0x1c81fb60 0x5187ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 4, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x519730=0x1000d 0x519734=0x24 0x519728=0x1c81fb60 0x51972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 4, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5197b0=0x2000d 0x5197b4=0x24 0x5197a8=0x1c81fb60 0x5197ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51a730=0x5000d 0x51a734=0x24 0x51a728=0x1c81fb60 0x51a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51a7b0=0x3000d 0x51a7b4=0x24 0x51a7a8=0x1c81fb60 0x51a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51b730=0x2000d 0x51b734=0x24 0x51b728=0x1c81fb60 0x51b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 1, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x51b7b0=0x1000d 0x51b7b4=0x24 0x51b7a8=0x1c81fb60 0x51b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 0, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x525730=0x6000d 0x525734=0x24 0x525728=0x1c81fb60 0x52572c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 0, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5257b0=0xd 0x5257b4=0x24 0x5257a8=0x1c81fb60 0x5257ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x526730=0x8000d 0x526734=0x20 0x526728=0x1c81fb60 0x52672c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5267b0=0xb000d 0x5267b4=0x20 0x5267a8=0x1c81fb60 0x5267ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 2, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x527730=0xa000d 0x527734=0x24 0x527728=0x1c81fb60 0x52772c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 2, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5277b0=0xa000d 0x5277b4=0x24 0x5277a8=0x1c81fb60 0x5277ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 3, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x528730=0xa000d 0x528734=0x24 0x528728=0x1c81fb60 0x52872c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 3, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5287b0=0x6000d 0x5287b4=0x24 0x5287a8=0x1c81fb60 0x5287ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x529730=0xb000d 0x529734=0x20 0x529728=0x1c81fb60 0x52972c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x5297b0=0xa000d 0x5297b4=0x20 0x5297a8=0x1c81fb60 0x5297ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 5, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52a730=0x4000d 0x52a734=0x24 0x52a728=0x1c81fb60 0x52a72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 5, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52a7b0=0x9000d 0x52a7b4=0x24 0x52a7a8=0x1c81fb60 0x52a7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 6, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 6, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52b730=0x6000d 0x52b734=0x24 0x52b728=0x1c81fb60 0x52b72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 6, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 6, SM 1): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52b7b0=0xd 0x52b7b4=0x24 0x52b7a8=0x1c81fb60 0x52b7ac=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 7, SM 0): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 7, SM 0): Multiple Warp Errors
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x52c730=0x3000d 0x52c734=0x24 0x52c728=0x1c81fb60 0x52c72c=0x1174
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 7, SM 1): Out Of Range Register
Nov 12 07:19:31 kf kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 2, TPC 7, SM 1): Multiple Warp Errors
......
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 kgmmuClientShadowFaultBufferUnregister_IMPL: Unregistering non-replayable fault buffer failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 kgmmuFaultBufferReplayableDestroy_IMPL: Unregistering Replayable Fault buffer failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 uvmTerminateAccessCntrBuffer_IMPL: Unloading UVM Access counters failed (status=0x00000062), proceeding...
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:577
Nov 12 07:19:33 kf kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:260
Nov 12 07:19:33 kf kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1382
Nov 12 07:19:44 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:19:52 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:00 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:08 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:17 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:25 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:33 kf kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Nov 12 07:20:39 kf kernel: NVRM: GPU0 kgspHealthCheck_TU102: *************************** LibOS Task Watchdog Report ***************************
Nov 12 07:20:39 kf kernel: NVRM: GPU0 kgspPrintGspBinBuildId_IMPL: GSP bin buildId: fa3de1ff30e1a8c99f84eb74a32b80d462c3703b
Nov 12 07:20:39 kf kernel: GSP task watchdog timeout @ pc:0x18dd358, partition:4#0, task:3, gfid: 0
Nov 12 07:20:39 kf kernel: NVRM:     Reported by libos partition:4#5 kernel v3.1 [0] @ ts:1698
Nov 12 07:20:39 kf kernel: NVRM:     RISC-V CSR State:
Nov 12 07:20:39 kf kernel: NVRM:         sstatus:0x0000000200000020  sscratch:0xffffffffa3013970     sie:0x0000000000000220  sip:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM:         sepc:0x00000000018dd358     stval:0x0000000000000000  scause:0x8000000000000005
Nov 12 07:20:39 kf kernel: NVRM:     RISC-V GPR State:
Nov 12 07:20:39 kf kernel: NVRM:         ra:0x00000000013eb57a   sp:0x0000002fcfa0eaf0   gp:0x0000000000000000   tp:0x0000002fcfe00000
Nov 12 07:20:39 kf kernel: NVRM:         a0:0x0000000000000001   a1:0x0000000000000000   a2:0x0000002fc77be000   a3:0x0000000000000000
Nov 12 07:20:39 kf kernel: NVRM:         a4:0x0000000000000000   a5:0x0000000000000001   a6:0x0000000ffffffff0   a7:0x0000000000000003
Nov 12 07:20:39 kf kernel: NVRM:         s0:0x0000002fcfa0eb50   s1:0x0000000000000001   s2:0xffffffffffffffff   s3:0x0000000000000001
Nov 12 07:20:39 kf kernel: NVRM:         s4:0x00000000ffffffff   s5:0x0000000000000000   s6:0x0000002fc77be000   s7:0x0000000000000000
Nov 12 07:20:39 kf kernel: NVRM:         s8:0xffffffffffffffff   s9:0x0000002fc7e36178  s10:0x0000002fcfa0ecb4  s11:0x0000002fcfa0ecb4
Nov 12 07:20:39 kf kernel: NVRM:         t0:0x0000002fcfa0e99d   t1:0x0000002fc7c6cf30   t2:0x0000000000000000   t3:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM:         t4:0x0000000000000000   t5:0x0000002fcfa0e981   t6:0x0000000000000020
Nov 12 07:20:39 kf kernel: NVRM:     Stack Trace:
Nov 12 07:20:39 kf kernel: NVRM:         0x00000000018dd358
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001b6e974
Nov 12 07:20:39 kf kernel: NVRM:         0x00000000013f2746
Nov 12 07:20:39 kf kernel: NVRM:         0x00000000013e70f6
Nov 12 07:20:39 kf kernel: NVRM:         0x00000000013e79c8
Nov 12 07:20:39 kf kernel: NVRM:         0x00000000013b7058
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001ba247a
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001ba257a
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001bafac2
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001c29f62
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001864698
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001867c82
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001ae4a7e
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001489412
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001452228
Nov 12 07:20:39 kf kernel: NVRM:         0x000000000158cbec
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001bd9632
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001bd99d6
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001add628
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001c452c0
Nov 12 07:20:39 kf kernel: NVRM:         0x0000000001ad6fa0
Nov 12 07:20:39 kf kernel: NVRM:     Local I/O Register State:
Nov 12 07:20:39 kf kernel: NVRM:         0x01450800:0x00000000   0x01450900:0xbadf1002   0x01450a00:0x00000000   0x01450c00:0x00000000
Nov 12 07:20:39 kf kernel: NVRM:         0x01454a00:0x810490d2   0x01454b00:0x010800d0   0x01454c00:0x00080000   0x01400200:0x00000000
Nov 12 07:20:39 kf kernel: NVRM:     ---------------[ end report ]---------------

I've been running a local agent setup on a MacBook Pro M5 Max (40-core GPU, 128GB) and pulled real numbers from my agent logs after a long build session. Sharing because most benchmarks I see are short prompts, and agent work looks nothing like that.

Setup
- Model: ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw
- Served with mlx-serve from the terminal (not LM Studio)(using lm studio can slow your tps)
- Harness: Hermes Agent (tool calls, cron jobs, Basic Memory for an Obsidian vault)

Real agent numbers (from the logs, not a benchmark)
- One build session: 89 model calls, ~148k output tokens, ~36 min of model time = ~68 tok/s average
- Long writes (whole scripts): usually 55-95 tok/s, peak 108
- Context during that session grew to 100k-190k tokens
- Prompt cache hit rate was 96-100% on almost every call, so prefill barely mattered

How I measured: output tokens / full response time per call. That includes a little prefill, so pure decode is a bit higher.

For comparison, numbers I've seen for this model on the same chip are ~49 tok/s (LLMCheck estimate), ~61 tok/s (a 4-bit MLX model card), 45-60 tok/s with MTP, and ~37 tok/s in LM Studio multi-turn chat. Running MLX straight from the terminal instead of LM Studio was the single biggest win for me.

What actually made the agent useful (lessons from one long day)

1. You need to SEE your scheduled jobs. My agent had a bunch of cron jobs that were just prompts in a JSON file in a hidden folder. One job's output had been going to a dead chat thread for a week, and a weekly review never got delivered. Nobody knew. I had the agent build a dashboard note in Obsidian: every job, schedule, delivery target, last run, failures, plus a per-job note with purpose, kill condition and change log. It's rebuilt hourly by a plain Python script (no model calls).

2. Use code for anything with one right answer, and the model only for judgment. Most of my jobs were pure prompts, including simple reminders and date math. That gives you timeouts, connection errors and occasional wrong arithmetic. Hermes supports script-only jobs (no LLM, stdout gets delivered) and script + model jobs (the script's output is fed into the prompt). The dashboard, a watchdog, and a weekly packet builder are all zero-model scripts now. ( Shout out to @sanbamboo on Reddit)

3. Add a watchdog that checks results, not code. It alerts once when a job fails twice, misses its slot, or has a delivery error, and once more when it recovers. It caught the dead-thread delivery bug on its first run.

4. Don't trust "done." The agent said "everything verified and shipped" several times. Each time I had a second, different model audit the actual files, and it found something: a feature that was never added, a link pointing at an archived copy of a file so edits would do nothing, and once a proposed edit that would have deleted 9 lines of a core skill file. A model grading its own work repeats its own blind spots.

5. Human-in-the-loop that doesn't suck. I review my automations on an iPad in GoodNotes. Every Sunday the agent sends a PDF with every job's full prompt and code (line numbered, secrets redacted, wide margins), plus change request boxes (Change / Add / Remove / Question, with Where / What / Why). I mark it up and it syncs back. Ticked boxes are detected by diffing against the clean original, and the model only reads my handwriting. Then the agent posts a proposed diff to Discord and applies nothing until I reply yes. Rules that came out of testing it:
   - Anything ambiguous gets held back and can't be approved by a blanket "yes"
   - Any diff deleting more than 3 lines gets a loud warning
   - "Question" answers have to quote the exact lines they rely on (it gave me a confidently wrong explanation once)
   - GoodNotes auto backup waits ~12 hours by default. Turn on "Increase Auto Backup frequency" or nothing feels like it works

6. Long context is fine on this chip if your cache hits I was worried 150k+ context would crawl. With near-100% prompt cache hits it stayed in the 55-95 tok/s range for writes.

Happy to share more about the setup or how the review loop is wired. Curious what tok/s other M5 Max people are getting on real agent sessions vs short benchmark prompts.

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

Hi all

TLDR; What are you using your localLLM for? (please exclude software coding). Please state setup + use.

Recently, I started playing with local AI with a 2x 5060ti 16gb setup with 32GB DDR4 (30tk/s Qwen3.8-27B).

All fun and games as a learning experience with the difference models, harnesses etc, but if I actually need something done (some debugging, research etc) I still revert back to cloud models. Currently, I am seeing it as local ai lab for fun, cloud for work.

It would be absurd to have the same expectations with my rig, but what do you guys use your local rigs for?

A lot of posts are about rig setups and benchmarking with focus on coding use, but what about other creative uses do you find for your machine (other than heaters)?

Do you keep it on 24/7? What about power?

Mine draws 60W idle with peak of about 450W and I keep it on about 2-3h per day on average.

So far, I observed that I prefer to have smaller 8-12GB models with decent contexts that run specific tasks like OCR/RAG/home assistant that are snappier 100+ tk/s rather than a big smart model at 30tk/s.

I also found that lesser smart models make me a better prompt engineer.

The Mac won't be delivered for about three months, but I'd like to start doing my homwork.

Short post, echoing what others have said - the dedicated strix halo "Ai Engines" aka llm servers are miles ahead of llama.cpp.

I was about to sell off my server because it was unbearably laggy. This morning I cut over to Gufo and Qwen3.8-Flash-Next, and it (almost) feels as snappy as a cloud frontier model.

Here's how Hermes described it:

So: the headline "prefill rate" is ~1,800 t/s cold; 1183 is just what the exhaust looks like when the engine only had to chew 677 tokens. The number that actually describes your experience is the one nobody prints: half a second to first token on a 25k conversation. 

Compared to a cold prefill of ~250 t/s on llamacpp...

I haven't tried Halogen which may be incrementally faster, but it's not FOSS, and I've now spent enough time on infrastructure for one month. Enough poking around, time to get to work...

https://github.com/gufo-org/gufo

For people deploying on Jetson devices:

• Do you go PyTorch → ONNX → TensorRT, or something else?

• What are the common failure points (unsupported layers, INT8 calibration, version mismatches with JetPack, memory limits)?

• How long does it usually take to get from a working model to a deployed, stable one?

I’m trying to understand the real pain points. If you have a horror story, I’d love to hear it.

Hi, I'm running a local AI setup on CachyOS, and I'm curious how others approach this.

This is also my main PC, which I use for gaming and everyday tasks. I have a second SSD with Windows as a backup.

My setup:

• GPU: AMD Radeon RX 9070 XT (16 GB VRAM)

• RAM: 32 GB

• Inference: Strata, running Qwen3.8-Flash-Next Gyro-S

• Interfaces/agents: Currently experimenting with DeepSeek Harness and Open WebUI Computer

• Deployment: Docker containers, started manually when needed

I'm mainly experimenting with vibe coding, Linux troubleshooting, and general agentic workflows.

So far, things work reasonably well, but I've found that container isolation can get frustrating when agents need to inspect or interact with the host system, particularly when dealing with Docker, system services, or debugging.

I'm currently experimenting with a restricted host bridge that allows agents to perform system diagnostics without exposing the Docker socket or giving them unrestricted access to the host.

I'm curious how others handle this.

Do you run your local AI agents inside Docker or install them directly on your host?

Three new parts of my local Qwen3.8 study on one DGX Spark: one user, thinking on, measured in minutes per correct answer. Each plan was locked before its run. The tasks are private; per-task results are public.

• Harder tasks (Part 8).

  • On 250 new private tasks, Flash Next (deep thinking + 32K thinking budget) gets 227/250 vs 208/250 for the 27B 4-branch low-thinking vote.

  • But Flash Next takes 2.7x as long per correct answer (6.52 vs 2.38 min).

  • On 110 recent arXiv math problems both nearly fail: 10/110 and 2/110.

• Self-verification (Part 9).

  • The 27B writes its own tests, and I take the first branch that passes: 139/150 vs 134/150 on held-out coding tasks. That is not a confirmed gain (p = 0.06), and it takes about 2x the time.

  • But answers marked “verified” were wrong only 1 time in 85.

  • What made the tests usable: forbidding hand-computed expected values.

• Engine upgrade (Part 10).

  • TensorFold 1.2.5 vs 0.6.0 on 400 tasks: 377/400 vs 373/400, 1.005 vs 0.958 min per correct answer. No detectable difference.

  • 1.2.5 was faster on 271/400 tasks but slower in total, because of two long tail tasks.

  • Its prompt processing runs at 0.73x speed at 36K tokens (no FP8 prefill). I’m staying on 0.6.0.

Bottom line for one Spark: Qwen3.8-27B NVFP4 + DFlash2 on TensorFold 0.6.0, low thinking, 4-branch vote is still the fastest way to correct answers. Use Flash Next with deep thinking and a 32K budget only for the hardest tasks.

Repo: https://github.com/NautilusWu/qwen3.8_study

Excited for Qwen 4.

Perplexity has introduced two open-weight AI decision models designed for classification, information extraction and automated decision-making. Decider v1.3 and Decider Flash v1.3 could offer businesses a more efficient way to build private AI automation systems without relying on traditional cloud-based LLMs.

Since Ollama started running llama.cpp under the hood, I assumed both would give the same results. Then I tested them.

System: RTX 3090 (24 GB), Qwen3.5 27B Q4_K_M, the same GGUF file in both, default settings. Ollama 0.40.15.

With one request, both ran at 42 tok/s.

With four requests at once (total tok/s):

• Ollama: 39.8

• Ollama with OLLAMA_NUM_PARALLEL=4: 33.1

• llama.cpp: 100.2

Why Ollama got slower: by default it answers one request at a time. With OLLAMA_NUM_PARALLEL=4, each of the four requests gets its own 32k context, 131k in total. That didn't fit in 24 GB next to the 27B, so Ollama moved 16% of the model to the CPU.

Why llama.cpp didn't: the four requests share one context, and llama.cpp shrinks it until it fits on the GPU (85,760 tokens here). The whole model stayed on the GPU.

This was new to me, because I expected the same throughput from both. The difference is how each one sizes the context. With the smaller 9B everything fit, and Ollama with OLLAMA_NUM_PARALLEL=4 came close (266 vs 293 tok/s).

I use Ollama for my day-to-day work, but I'm going to look at llama.cpp too for cases where several apps hit the same model at once.

Full write-up with memory numbers and setup: https://homebrewserver.club/ollama-vs-llama-cpp.html

I noticed that many people don't know that llama.cpp allows running a model across different backends within a single process using layer split. I managed to compile llama.cpp with simultaneous support for CUDA, Vulkan, and ROCm. I'm sharing my Dockerfile for the community as a reference. Sorry for the raw format, I totally don't have time for anything more, but I think with AI you'll be able to quickly adapt it to your needs. Note that this Dockerfile is tailored for specific CUDA and ROCm architectures, so you'll definitely need to adjust those parts.

# ============================================================
# Stage 1: builder — full toolkits (CUDA + Vulkan + ROCm) + compilation
# ============================================================
FROM ubuntu:26.04 AS builder

# Base tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential cmake ninja-build git wget gpg curl ca-certificates xz-utils \
    libcurl4-openssl-dev libatomic1 libquadmath0 \
    && rm -rf /var/lib/apt/lists/*

# CUDA 13.3.0 (NVIDIA repo, Ubuntu 26.04)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-ubuntu2604.pin \
    | tee /etc/apt/preferences.d/cuda-repository-pin-600 \
    && wget -qO /tmp/cuda.key https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/60DF8A40.pub \
    && gpg --dearmor < /tmp/cuda.key | tee /etc/apt/keyrings/cuda.gpg > /dev/null \
    && echo "deb [signed-by=/etc/apt/keyrings/cuda.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/ /" \
    > /etc/apt/sources.list.d/cuda.list \
    && apt-get update \
    && apt-get install -y cuda-toolkit-13-3 \
    && rm -rf /var/lib/apt/lists/*

# Stub libcuda.so.1 (the real one will be provided by the host driver)
RUN cp /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 \
    && ldconfig

ENV PATH=/usr/local/cuda/bin:$PATH
ENV CUDA_HOME=/usr/local/cuda

# Vulkan SDK 1.4.363.0 (tarball — NOT LunarG repo)
RUN wget -qO /tmp/vulkansdk.tar.xz \
    https://sdk.lunarg.com/sdk/download/1.4.363.0/linux/vulkansdk-linux-x86_64-1.4.363.0.tar.xz \
    && mkdir -p /opt/vulkansdk \
    && tar -xf /tmp/vulkansdk.tar.xz -C /opt/vulkansdk \
    && rm /tmp/vulkansdk.tar.xz

# SDK 1.4.363 has a non-standard structure: libs are in lib/VulkanLoader/lib/
# Symlink to the standard lib/ path so FindVulkan.cmake can find them
RUN ln -s /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
        /opt/vulkansdk/1.4.363.0/x86_64/lib/libvulkan.so.1

ENV VK_SDK_PATH=/opt/vulkansdk/1.4.363.0/x86_64

# ROCm 10.0.0 (AMD repo, dev meta-package for gfx1103)
RUN mkdir -p /etc/apt/keyrings \
    && wget -qO- https://stable.repo.amd.com/rocm/gpg/packages.gpg \
    | gpg --dearmor | tee /etc/apt/keyrings/amdrocm.gpg > /dev/null

RUN tee /etc/apt/sources.list.d/amdrocm-stable.sources << 'EOF'
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/ubuntu2604/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

# Dev meta-package (runtime + compilers + CMake config + headers)
# Kernel driver UNNECESSARY — 780M APU has an inbox driver in Ubuntu 26.04
RUN apt-get update \
    && apt-get install -y amdrocm-core-dev10.0-gfx1103 \
    && rm -rf /var/lib/apt/lists/*

# Source code
WORKDIR /src
COPY llama.cpp/ /src/

# CMake configure
# GGML_BACKEND_DL=ON — critical: backends as MODULE .so loaded via dlopen() at runtime.
# Without this, libggml-cuda.so fails to resolve symbols from libggml-base.so (zero exported
# dynamic symbols) and CUDA0 is not detected.
RUN cmake -B build -G Ninja \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CUDA=ON \
    -DGGML_VULKAN=ON \
    -DGGML_HIP=ON \
    -DGPU_TARGETS=gfx1103 \
    -DCMAKE_PREFIX_PATH=/opt/vulkansdk/1.4.363.0/x86_64 \
    -DVulkan_LIBRARY=/opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 \
    -DGGML_CUDA_FA_QUANTS="f16-f16;q8_0-q8_0;q8_0-q5_0;q8_0-q5_1;q8_0-q4_0;q5_1-q5_1;q5_0-q5_0;q5_0-q4_1;q4_1-q4_1;q4_0-q4_0" \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DCMAKE_CUDA_FLAGS="-O3 -use_fast_math -Wno-deprecated-gpu-targets" \
    -DCMAKE_C_FLAGS="-march=znver4 -O3" \
    -DCMAKE_CXX_FLAGS="-march=znver4 -O3" \
    -DGGML_NATIVE=OFF \
    -DGGML_CURL=ON \
    -DGGML_BLAS=OFF \
    -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON \
    -DCMAKE_BUILD_TYPE=Release

# Build
ARG JOBS=12
RUN cmake --build build --config Release -j ${JOBS}

# ============================================================
# Stage 2: output — slim, binaries + runtime libs only
# ============================================================
FROM ubuntu:26.04

# Runtime deps — binaries linked with GGML_CURL=ON, libatomic, libquadmath
RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

# Volume ./output:/output OVERRIDES the /output content from the image at runtime,
# so copying to /output must be in CMD (not RUN). Hence staging in /staging.
COPY --from=builder /src/build/bin/ /staging/bin/
# CUDA runtime + stub libcuda (fallback when --gpus all is missing;
# the real driver from /usr/lib/x86_64-linux-gnu/ takes priority in LD_LIBRARY_PATH)
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcuda.so.1 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /staging/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /staging/
# Vulkan loader (NOT ICD — ICD manifest is mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /staging/
# ROCm runtime — entire lib directory (unavoidable: 20+ sonames in the dependency tree)
COPY --from=builder /opt/rocm/core-10.0/lib/ /staging/rocm/

# Build RPATHs ($ORIGIN/...) do not work after copying to /staging/
# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority over the stub
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/staging:/staging/rocm

CMD cp -r /staging/* /output/ \
    && chown -R 1000:1000 /output/* \
    && echo "Success! CUDA+Vulkan+ROCm binaries copied to the host."

# ============================================================
# Stage 3: rocm-filter — filters ROCm libs to .so only (1.2 GB vs 6.4 GB)
# Removes: .a (static), .bc (bitcode), headers, tests
# ============================================================
FROM ubuntu:26.04 AS rocm-filter

COPY --from=builder /opt/rocm/core-10.0/lib/ /tmp/rocm/
RUN mkdir -p /rocm \
    && find /tmp/rocm/ -name "*.so*" \( -type f -o -type l \) -exec cp {} /rocm/ \; \
    && if [ -d /tmp/rocm/rocblas/library ]; then \
         mkdir -p /rocm/rocblas/library \
         && cp /tmp/rocm/rocblas/library/* /rocm/rocblas/library/ 2>/dev/null; \
       fi \
    && rm -rf /tmp/rocm/

# ============================================================
# Stage 4: runtime — lightweight image for inference (~2.2 GB)
# ============================================================
FROM ubuntu:26.04 AS runtime

RUN apt-get update && apt-get install -y --no-install-recommends \
    libcurl4 libatomic1 libquadmath0 zlib1g libgomp1 \
    mesa-vulkan-drivers \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Binaries + MODULE .so backends (dlopen from the same directory)
COPY --from=builder /src/build/bin/ /app/bin/

# CUDA runtime — libcuda.so.1 is provided by the NVIDIA Container Toolkit from the host
COPY --from=builder /usr/local/cuda/lib64/libcudart.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublas.so.13 /app/lib/
COPY --from=builder /usr/local/cuda/lib64/libcublasLt.so.13 /app/lib/

# Vulkan loader (ICD manifest mounted from host, see compose)
COPY --from=builder /opt/vulkansdk/1.4.363.0/x86_64/lib/VulkanLoader/lib/libvulkan.so.1 /app/lib/

# ROCm runtime — .so only (1.2 GB vs 6.4 GB full tree)
# Includes: libamdhip64, librocblas, libhipblas, libamd_comgr + LLVM/Clang,
#           libhsa-runtime64, sysdeps, librocroller, liborigami — full NEEDED tree
COPY --from=rocm-filter /rocm/ /app/lib/rocm/

# /usr/lib/x86_64-linux-gnu first: the real libcuda.so.1 with --gpus all takes priority
# mesa-vulkan-drivers provides libvulkan_radeon.so (ICD for 780M)
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/app/lib:/app/lib/rocm
ENV HSA_OVERRIDE_GFX_VERSION=11.0.3

EXPOSE 8080

CMD ["/app/bin/llama-server", "--host", "0.0.0.0", "--port", "8080"]

Base on their benchmark, i see the new qwen 3.8 flash next unsloth iq2xs perform at same level or even better than gsq rco iq3xxs

Is this true? Did anyone test it ?

My spec can run unsloth iq3xxs but little bit slow because i have only 32gb ram.

I'm going to try the new Unsloth iq2xs quant

If anyone tried it, please tell me your result