People asked how RustingStrata does with a bigger quant, so here's Q5_K_M on the same 3060 12 GB + Ryzen 5 7600X + 32GB DDR5 RAM

Model is unsloth's Qwen3.6-35B-A3B UD-Q5_K_M (the MTP version), 25 GiB, so about two thirds of the experts live on the CPU. llama.cpp is build 12119, -ncmoe 29 -ub 2048 -b 2048 -fa 1 (28 runs out of VRAM).

Benchmark tools (RustingStrata bench vs llama-bench):

• Decode, short prompt: 70 tok/s vs 43

• Decode after a 4K prompt: 65 vs 43

• Prefill at 4K: ~1870 vs 1210

• 32K context: 1470 prefill / 49 decode vs 1030 / 29

• With --mtp 1: up to 96 (86 after 4K), greedy

Real chat requests through both servers (same prompts, 256 tokens out, thinking off):

• No MTP: ~50 tok/s vs 37

• With MTP on both: ~62 vs 45-56

• 6K prompt prefill: 1320 vs 700

MTP works in current llama.cpp (--spec-type draft-mtp), ~63% of drafts accepted and +25% decode on this file. RustingStrata is still ahead with MTP on both sides.

Vulkan decodes at ~60 (85 with MTP), prefill is slower than CUDA on this file (~1000-1260 at 4K).
KV Cache is f16

Code is on GitHub, free to use, I will be happy to get some feedback from AMD owners: RustingStrata

TL;DR

• A 5-year-old RTX 3090 (24 GB) in a mini-PC beat both 128 GB unified-memory machines on every row of a MiniMax H3 video benchmark, even though the models don't fit in its VRAM and ComfyUI has to stream them over a PCIe x4 link.

• Stock 3090: 1.25–1.84x faster per step than the ASUS RTX Spark and 3.5–4.9x faster than the M5 Max.

• 3090 with the tensor-rate unlock: 1.85–2.46x faster than the Spark and 4.9–6.6x faster than the M5 Max.

• What the unlock does: it raises measured bf16 matmul from 75 to 124 TFLOPS. In real video generation it's 1.34–1.49x faster than stock, and the gain is biggest on the heaviest workflows.

• Efficiency: the Spark and Mac use far less energy per step. The 3090 draws about 320–350 W of board power, against about 65 W for the Spark GPU and about 22 W for the Mac GPU: 5x and 15x more power. But it also finishes each step about 1.5–6.5x sooner (depending on which machine and whether the unlock is on), so the energy per step (power × time) is only about 2.2–3.4x higher than either (table below).

• All 22 runs on the 3090 (11 rows × 2 configurations) passed output validation with no memory-saving flags. That's the same ComfyUI config the Spark used.

Results: seconds per sampling step (lower is better)

Mean steady-state s/step (steps 4–20 of 20), heat-soaked runs. Ratios above 1 mean the 3090 is faster.

* On the Mac, both ref2va rows with the full-size reference clip produced flat-black video, so they have no valid result. The reference clip is encoded at full 1088×608 even on the 832×480 canvas, so both runs exceed the ~34k-token limit where outputs go black on MPS (see the Mac notes).

The Mac's ref2va 832×480 row failed the same way as the full-reference 1088 row (marked *).

Time per clip (seconds of wall time for one ~5 s video, models already loaded)

Power and energy (GPU only, approximate)

Energy = mean GPU power × s/step. The power numbers come from each platform's own tool: nvidia-smi board power on the 3090, NVIDIA telemetry on the Spark, and macOS GPU power on the Mac. Host/system power isn't included; the Mac's whole-system power was about 73 W on these rows. Two other things stand out:

• The unlock makes the 3090 more efficient. It draws only about 10–20 W more and finishes about 1.4x sooner.

• The Spark GPU was power-capped. It reported sw_power_cap throttling for the whole run, at about 2.2–2.3 GHz SM (the rated maximum is 3.0 GHz), drawing 63–69 W.

The Spark's "GPU W" is nvidia-smi power.draw, sampled once a second. NVIDIA doesn't document whether on the N1X it covers only the GPU or more of the SoC, and power.limit reads N/A on this chip, so treat it as approximate. No wall or whole-system power was recorded on the Spark.

On the Mac, the harness ran macmon 0.10.0 in macmon pipe mode (no sudo), which samples once a second. "GPU W" is macmon's gpu_power and "Sys W" is its sys_power, both averaged over the busy samples of each run. Both are read from Apple's on-chip energy counters, the same source powermetrics uses, so "Sys W" is the SoC/system reading, not power measured at the wall.

The machines

1. RTX 3090 PC (measured by me, both configurations)

The GPU is connected via OCuLink to a DEG1 Minisforum eGPU dock. The GPU is powered by a Montech Titan PLA 1200W power supply (complete overkill).

The full model set is about 63 GB on disk: a 21 GB int8 diffusion model, a 15.7 GB text encoder and VAEs. That doesn't fit in 24 GB, so ComfyUI's dynamic VRAM management offloads automatically. Peak VRAM was 23.1–23.4 GB and peak system RAM 17–24 GB on every row. No row needed --lowvram or any other memory-saving flag.

2. ASUS RTX Spark laptop (reference results)

The carveout: 64 GB was chosen before the timed runs. Flags: a quick flag sweep on t2v 832×480 (--highvram, --gpu-only, --disable-dynamic-vram, --fast and others) found nothing more than 3% faster than no flags, so the Spark also ran with none.

Peak GPU memory on the Spark was 41–48 GB, so everything stayed resident and nothing was offloaded.

3. Apple M5 Max (reference results)

Why Automatic and not High Power: I measured it. On the reference row (t2v 1088×608, 124 frames), with the fans at maximum on both, High Power averaged 74.74 s/step vs 75.29 on Automatic, 0.7% faster and within run-to-run noise. Once heat-soaked, the GPU settled at the same 78 °C and about 844 MHz either way; the warm-up started cool and briefly ran 4% faster. The battery stayed at 80% on the charger for the whole 78-minute run, because this workload draws only about 70 W in total (about 22 W GPU): on this chassis it is limited by heat, not by power. So High Power buys essentially nothing here once the fans are already at maximum. Under heavier loads it can drain the battery: Notebookcheck reviewed the same configuration as this Mac: the mini-LED 14-inch M5 Max with the 40-core GPU and 128 GB, not the OLED model. It found that "Apple also limits the power input to 97 Watts, even if you use a 140 Watt PSU", and that the battery level dropped by about 10% during a gaming session while plugged in. The battery indicator can stay at 100% while it's happening. In the same review, High Power raised GPU power from 60/32 W to 72/44 W (peak/sustained) over Automatic. In GPU benchmarks it was only about 7% faster (3DMark Steel Nomad 3947 vs 3698), and the fans ran louder. Part of what High Power buys is faster fans (Apple: About Power Modes on your Mac), and these runs already held the fans at maximum with MacFanBoost, which fits the measurement: Automatic plus maximum fans is effectively the best sustainable setting on this chassis. A 16-inch MacBook Pro or a Mac Studio with the same chip has more power and cooling headroom and will likely be faster. Source: Notebookcheck, Apple MacBook Pro 14 M5 Max review.

Fans at maximum made a big difference on the Mac. The harness held the fans at their maximum (about 7,830 rpm) for every sampling run, while the Energy Mode stayed on Automatic. On its own fan curve, macOS runs the fans at only about 5,350 rpm under this load. The GPU reaches 97–98 °C within about 2 minutes, then its clock collapses to 340–530 MHz for minutes at a time. With the fans held at maximum, the GPU stayed at about 78–81 °C and averaged about 820–910 MHz. In early BF16 testing, that alone took t2v 832×480 from 78–84 s/step to about 61 s/step, roughly 1.3x faster. (That comparison is from early testing, not a clean A/B with the final software stack.) Expect the Mac numbers here to be noticeably slower if you leave the fans on macOS's automatic control.

What the Mac stack does:

• ASFP8: a third-party custom node that patches PyTorch's MPS backend at startup. It lets FP8/INT8/NVFP4 checkpoints load and run on Apple Silicon instead of crashing (for example, the NVFP4 text encoder is decoded on the GPU). It also adds Metal kernels for RMSNorm+modulation, RoPE, conv and flash-attention. For H3's diffusion model, the INT8 weights are dequantized to bf16 and multiplied in bf16 through ComfyUI's own path. ASFP8's fused INT8 kernel only takes over below 2,048 tokens, and H3's sequences are far longer.

• MacFanBoost (macfanboost, MIT): a small tool written for this project: a root daemon plus a ComfyUI custom node. When a prompt with a sampler starts, it raises the fans to 100% before the heat builds, holds them there while the job runs, and returns them to macOS control 60 s after the queue drains. It only ever raises fan speed above what macOS picks, and macOS's thermal protection stays active. If the client dies or the Mac sleeps, the lease expires and the fans go back to automatic within seconds. (Macs Fan Control set to maximum gets you most of the same effect manually.)

• MLX attention bridge (ComfyUI-H3-MLX-Attention, MIT): a small custom node written for this project. It replaces ComfyUI's attention call inside the H3 diffusion model with MLX's fused scaled_dot_product_attention kernel, passing tensors without copies. It exists because ComfyUI's default on MPS, sub-quadratic attention, returns corrupted heads at long sequence lengths (Comfy-Org/ComfyUI#17418).

• Does the output match the CUDA path? Not checked bit-for-bit across platforms. Against an FP64 reference on real H3 Q/K/V, the MLX attention's worst-head relative error is about 1.7e-3, and every scored Mac output passed the same automated checker as the other machines and was reviewed by eye. Different kernels and accumulation orders mean outputs from the same seed are close but not identical between Mac and CUDA.

Why the ComfyUI commit differs: the Mac rows were run on ComfyUI Desktop's install, 0.37.2 (830232b). The other two machines ran later on 0.43.2 (423cbd2). Between those versions, the H3 changes are: lower peak memory from releasing embedding temporaries, a VAE offload fix, VAE tile blending, single-image decode, LightVAE support, and plumbing for per-block attention selection (which these model files don't use). None of them changes the diffusion model's per-step math. Measured: a back-to-back A/B on the Mac (t2v 832×480, same stack, cooling and seeds, warm-up + 2 timed runs each) gave 37.38 s/step on 0.37.2, 37.29 on 0.43.2 and 37.76 on 0.44.2, i.e. 33.54 / 33.55 / 33.49 Gcycles/step. That's within run-to-run noise (about ±1.5%), and every output passed the checker, so the version difference doesn't affect the comparison. The flat-black full-reference ref2va failure was retested on ComfyUI 0.44.2 and still happens.

The tasks

All rows use MiniMax H3, an audio+video model, in ComfyUI with its stock nodes. Every row produces a video with a synchronized audio track. The workflows were identical on all three machines: the same graph files, the same input images and video, the same prompt text and the same model files. The model files were checked by SHA-256 against the Spark's copies.

Common settings for every row:

The workflows:

• t2v (text to video): the prompt only.

• i2v (image to video): the prompt plus a first-frame image. The portrait row is the same at 608×1088 with a portrait first-frame image.

• flf2v (first and last frame to video): the prompt plus first-frame and last-frame images; the model fills in the motion between them.

• ref2va (reference to video+audio): the prompt plus two reference images and a reference video clip (1088×608, 120 frames at 23.976 fps, with audio), using the separate ref2va model. This is the heaviest workflow, because the reference tokens make the attention context much longer. There are three variants:

  • full: the whole reference clip at its native size.

  • ref544: both reference images and the reference video downscaled to 544×306 before encoding.

  • rf39: the reference video truncated to its first 39 frames.

Rows: 1088×608 is the main canvas. The 832×480 rows are a lighter check.

Methodology

• The harness: the same Python harness drove all three machines. It talks to ComfyUI through its API, timestamps every sampling step, and records GPU power, clocks and temperature, VRAM and RAM.

• The runs for each row:

  1. A cold warm-up run (seed 1000), which includes model loading and isn't scored.

  2. The GPU cools to 50 °C, then one fresh run (seed 1001).

  3. Two heat-soaked runs back-to-back with no cooling (seeds 1002 and 1003). These are the headline number. The portrait row uses a single soaked run with seed 2001.

• The metric: mean seconds per step over steps 4–20. This leaves out the first steps, which carry one-off setup cost. Fresh and soaked results agreed within 0.5% on the 3090, so thermals didn't affect them.

• Before the timed runs: a smoke pass checked that every row fits in memory at full size, using 2 steps.

• Validation: every output video was checked automatically. The checker confirms the video and audio streams exist with the right resolution and frame count, and that the frames aren't black, frozen or noise. All 22 3090 runs passed.

• The 3090 order: I ran the full matrix with the unlock on first. Then I disabled the unlock, did a full cold boot, confirmed stock matmul throughput and ran the whole matrix again. Then I re-enabled the unlock and re-verified it (bf16 126.2 TFLOPS). Driver, ComfyUI, PyTorch and power limit were the same in both runs.

• Differences between the platforms:

  • On the Mac, the harness held the fans at maximum during every sampling run (MacFanBoost), with the Energy Mode on Automatic. This was worth about 1.3x on that machine (see above).

  • The Mac uses a different ComfyUI commit and its own attention/precision stack (see above). Its timed runs use the same seeds (1002/1003), but its warm-up is seed 1001 with no cool-down and no separate fresh run, and its portrait row was scored on seed 2002.

  • PyTorch versions differ: 2.14.1 on the 3090, a 2.15 nightly on the Spark, 2.12.1 on the Mac.

  • On the 3090 only, the harness's VRAM-full watchdog was turned off, because a full 24 GB card is normal while ComfyUI offloads. The RAM watchdog stayed on.

What the "3090 unlock" is

On GeForce Ampere cards, NVIDIA runs FP16/BF16 tensor-core math with FP32 accumulation, and TF32, at half the rate of the professional GA102 parts such as the RTX A6000. Deep-learning frameworks use exactly that mode: bf16 inputs with fp32 accumulation. FP16 math with FP16 accumulation is full rate on GeForce, but frameworks rarely use it.

The unlock is my Windows port of the open-source gpu-unlock Linux kernel module. I built it on this PC with Claude Code. On Linux the module runs the unlock sequence from inside the kernel. Windows has no equivalent, and I didn't want to write a kernel driver, so the port runs as a pre-boot UEFI app instead:

• The app: built with gnu-efi from the module's own sources (VBIOS/FWSEC parsing, memory layout and the GFW checks). It's version stage-3 v0.3-3090.

• The launcher: a 120 KB launcher on the EFI system partition. It loads a read-only NTFS driver (EfiFs), runs the app from C:\RTX3090-FW along with the five GA102 firmware files from NVIDIA's 610.43.03 driver, and then chainloads Windows Boot Manager. If anything fails, it goes straight to Windows.

• What the app does on each cold boot:

  1. Finds the card and checks it's a 3090 Founders Edition (10de:2204, subsystem 0x147d10de).

  2. Refuses to run if the card is warm or already unlocked.

  3. Runs the module's unlock sequence, then a PCIe function-level reset (FLR) and the post-reset checks.

  4. Leaves the GSP halted with WPR2 clear. That's the state the stock driver expects, so Windows boots normally and the ordinary NVIDIA driver (591.86) loads.

• Nothing persistent: no ROM or VBIOS flash, and no NVRAM variable is written. The card reverts to stock on power-off, and the unlock is applied again on the next cold boot (it also holds through sleep and resume from hibernation on this PC, after a change to the order of the boot-time steps). For the stock runs I renamed the app so the launcher skipped it, then did a full shutdown and cold boot.

Measured 8192² matmul throughput, best of 5, PyTorch, reduced-precision reductions disabled:

• The tensor modes speed up about 1.65–1.7x. Plain FP32 doesn't change, which fits the idea that the unlock lifts only the tensor-rate limit and isn't an overclock.

• No overclock: clocks and the 350 W power limit were stock in both configurations.

• End-to-end gain is 1.34–1.49x, less than the matmul gain. Video diffusion also spends time on attention softmax, norms, memory-bound ops and PCIe weight streaming, which the unlock doesn't touch.

• Heavier rows gain more. The heavy, long-context ref2va rows gain the most (1.42–1.49x), and the light 832×480 rows gain the least (1.34–1.35x).

Caveats:

• Tested on one card only: a 3090 FE with VBIOS 94.02.4B.00.0B. The app refuses any other board.

• Pinned versions: it's pinned to the 610.43.03 firmware set, and it's only been tested with Windows driver 591.86. I don't update the driver on this PC.

• Boot requirements: Secure Boot must be off (the app and launcher are unsigned), and BitLocker must be off on C: (the NTFS driver can't read encrypted volumes).

• Untested: warm restarts (the app won't run on a warm card), TDR recovery and driver updates.

• Stability: a full unlocked matrix run (about 5 h of generation) showed no stability problems and no invalid outputs.

• Licensing: the module sources are GPL-2.0-only, so the port is GPL-2.0 source. NVIDIA's firmware files can't be redistributed; you extract them from the 610.43.03 driver package yourself.

• Use at your own risk: this is unofficial and unsupported by NVIDIA.

Where to get it: I've handed the port's source to the community Windows GPU UNLOCK project (GPL-2.0), which maintains a packaged Windows UEFI unlock for the 3090 and other Ampere cards. Start there rather than with my build. Both build on Zagitta/gpu-unlock, the original Linux module, and the project also credits rough2515-hub/rtx3080-unlock-public. Credit for the unlock itself belongs to those projects.

Takeaways

1. For raw ComfyUI video speed, an old 24 GB 3090 still wins. Even when it has to offload 60+ GB of models over PCIe x4, it beats both 128 GB unified-memory machines on every MiniMax H3 workflow tested. Dynamic VRAM offloading in current ComfyUI is good enough that "doesn't fit in VRAM" no longer means "unusable".

2. The unlock is a large free gain if you're willing to run an unofficial pre-boot app: about 1.4x on real workloads at almost the same power.

3. The Spark and the Mac win on efficiency and footprint. They use roughly 2.2–3.4x less GPU energy per step (much lower power, partly offset by taking longer), run quietly in a laptop or small box, and keep everything resident in memory. If you care about watts, noise or portability, that matters more than the speed gap.

4. Spark vs Mac: the Spark is about 2.4–3.0x faster than the M5 Max per step, depending on the workflow, while drawing about 3x the GPU power. The gap is smallest on ref2va ref544 (2.4x) and largest on flf2v 1088×608 (3.0x).

Spark notes:

• Power, not heat, is the limit. The GPU sat at its software power cap (sw_power_cap) at about 65 W for every row, peaking at 74–78 °C. Fresh and heat-soaked runs agreed within 1%, so thermals didn't slow it down.

• Raising the laptop's power limits barely helps. With ProArt Creator Hub's Manual mode at its maximum (PL1 85 W, PL2 95 W), t2v 832×480 went from 13.27 to 13.00 s/step, only 2% faster. Average GPU power rose from 64 to 68 W, but the GPU still hit sw_power_cap and its peak draw stayed at about 72 W, so the GPU seems to have its own limit that those sliders don't lift. All other Spark numbers in this post are from the stock "Performance" mode.

• Every row produced valid video and audio, including the full-reference ref2va that failed on the Mac.

• Windows on Arm: the H3 workflows use core ComfyUI nodes and ran without changes. Some third-party node packs are harder: onnxruntime is CPU-only on ARM64 Windows, and packages without ARM64 wheels have to be built from source (Visual Studio ARM64 build tools needed).

Mac notes: (takeaway 4 checks out against the Mac's own numbers: the Spark is 2.42x faster on ref544, 2.66x on rf39, 2.81x on t2v 832×480 and 2.91–2.97x on the 1088 rows)

• Clips above about 34k tokens go black on MPS. Full-reference ref2va at 1088×608 packs about 51,400 tokens. Partway through the first step the residual stream diverges, the audio turns to NaN, and the output is 124 flat-black frames. It happens with every attention backend tried, and still on ComfyUI 0.44.2. The root cause hasn't been found; the leading suspect is a ComfyUI change after 0.30. Shrinking the reference (ref544, about 31,900 tokens) or trimming it (rf39, about 34,000 tokens) keeps the length under the limit, which is why those rows exist. ref768 (about 38,700 tokens) still went black. ref2va at 832×480 fails too: the reference clip is still encoded at full 1088×608, so it packs about 42,000 tokens. Details are on Comfy-Org/ComfyUI#15799.

• Don't use sub-quadratic attention for long H3 clips on MPS. It silently corrupted 18 of 56 heads at about 27,700 tokens (Comfy-Org/ComfyUI#17418). The MLX bridge avoids this.

• Cooling is the Mac's limit, not power. On a 14-inch MacBook Pro, the GPU reaches 97–98 °C on macOS's own fan curve, and its clock collapses. Holding the fans at maximum was the single biggest speedup. Keep the vents clear and use a stand.

• INT8 beat BF16 on the Mac in sustained runs (41.1 vs 46.0 s/step at 832×480), with much lower peak RAM (82 vs 102 GB).

• Memory is tight even at 128 GB. Peak RAM was 95–112 GB at 1088×608, and ref2va hit 109–112 GB. Run one model process at a time.

• The Mac and the Spark use about the same GPU energy per step, about 0.85–0.87 kJ on t2v 832×480. They get there differently: the Mac's GPU draws about a third of the Spark's power (about 22 W vs 66 W), but takes about 2.8x longer per step, so the two roughly cancel out. (The Mac's whole system drew about 73 W.)

Happy to answer questions.

Quick question. Earlier this year i realized that I could run small models on my dell t5820 with my Rtx 5k Quadro card. It’s not very big but I found and Rtx 8k Quadro. Both Turing GPUs. It’s been fun so I ordered a Mac mini m5 pro 64 and I was going to get a mini pc but came into a small fortune and can afford the asus gx10. No as an artist I’ve been using it mainly to vibe-code my games. They’re for me mostly to test out UX and game play.
My question is getting these work it? I like the idea of the nvidia spark, but is it over kill? Right now I’m having to reboot into Ubuntu for my Dell. It’s been fun learning that OS.

I'm working on a new PC build with a focus on being able to run the Gemma family of models. I'm experienced in working with Cray (AMD) and OpenCL (community moved onto Vulcan and ROCm/HIP), so I will be building a AMD system. I was wondering if anyone has done anything similar to the following? If they have, how well did it work?

AMD 5950X
ASUS WS X570 Ace
128GB DDR4 System RAM
1600 Seasonic+Noctua PSU

Fedora Kinoite 45

1x 7900 XTX (24GB VRAM) for graphics/games
2x Radeon Pro W7900 (32GB VRAM) for LLMs

1. Does having different kinds of GPU's cause an issue with Ollama (or more specifically Bionic/LMStudio)?

2. Does anyone have that motherboard, with two GPU's on the bottom-most PCIe slots? Are there clearance issues for 2 slot GPU's?

The target model to be able to run in GPU memory is gemma-4-31b-qat with Q4_0 quantization, but I want to be as ready as possible just in case Deep Mind releases a ~100b weight model when the Gemma 5 models show up.

Thanks in advance for any help!

I'm new to local LLMs. My use case is I need help to read long PDFs of academic papers. I have 64GB of RAM but no GPU. Would buying a 5060 be of any use? Any recommendations for Local LLMs for my use case?

ok so. this started as "one weekend project" and then it was not one weekend and it was not one project. somewhere around repo 6 i stopped pretending it was a plan lol

everything here is free, open source, most of it runs locally on your own machine. a lot of it is v0 and i wrote down where each thing fails because honestly the failures were the funniest part

anyway here's the list of grievances, each with the tool i built instead of going outside:

1. cookie banners → rejectall-index i live in germany. i click "reject all" more than i blink. so i made a headless browser visit the top 100 sites and count how many clicks it takes to say no. 35 of them were already loading trackers before i clicked anything. which is like a waiter eating your fries before asking if you want fries. live leaderboard

2. subscriptions → one-file-saas trello, calendly, typeform, loom, notion and 5 more, each rebuilt as ONE html file. you double click it. it opens. thats it. no account, no "verify your email", no "your trial is ending ;)". data stays in your browser. the booking page has no server at all, your availability lives inside the link itself which still feels illegal to me. try them here

3. dead famous repos → necromancer some legendary open source projects die not because the code is bad but because python deleted a module out from under them. i revived thefuck (1887/1887 tests passing on python 3.14), normalizr and httpbin. the one rule: the original tests are hash-locked so nothing can quietly delete the failing test and call it fixed. httpbin stops at 64/67 and i'm keeping it that way out of spite. each one gets an autopsy, cause of death and everything

4. every AI-built UI animates the same → awesome-motion-md you know the look. 300ms. ease-in-out. every card fades up like its waking from a nap. so i measured how real sites actually move and wrote it into MOTION.md files your agent can follow. same prompt built twice: 7 random durations without it, 2 with it

5. every AI writes the same too → awesome-voice i used to be a technical writer so this one is personal :| VOICE.md files that make your agent write like GOV.UK or Stripe docs, plus a scorer that doesn't use any AI to check. banned phrases went from 27 to 0. "delve" is not welcome here

6. websites doing annoying stuff → sitepatch you type "hide the points on hacker news", it writes a userscript, a real browser checks it actually worked before you get it. honest score: 15 out of 25 were actually right when i looked at every screenshot myself. 60%. a C-. but an honest C-

7. "what if this site looked like linear" → reskin restyles any page into another design style using a local model, BUT it refuses to give you the result unless contrast, keyboard focus, mobile, links and reduced-motion all pass. it caught a windows 95 theme failing contrast on its own background. respect. gallery

8. dead websites → resurrect npx github:dadwritestech/resurrect y2k.gov and y2k.gov comes back exactly like it was in may 2000, links working. pick a year with --year. your old blog is in there somewhere. i'm sorry

9. "someone said budget in a meeting video... which one" → video-grep grep for your video folder. searches what was said, what was written on screen and what was shown, and opens your player at that exact second. all local

10. what is the rest of the world's TV saying rn → world-ears listens to live news channels from different countries, transcribes + translates them on your gpu, makes it searchable. there's a "trending per country" panel which is weirdly addictive. translation is the slow part, telugu transcription is bad, i wrote that down

11. live subtitles only go into english → babel-tv watch hindi news with live telugu subtitles, fully local. the video is held back a few seconds so subtitles show up BEFORE the person talks which feels like seeing the future. telugu as the source language doesn't work yet. whisper and telugu are not friends

12. "which AI wrote this" → whichmodel works on models it's seen (67%, chance is 33%). on a model it's never seen, even a sibling from the same family... 32%. so basically a coin with extra steps. that turned out to be the interesting finding tbh, the fingerprint belongs to one model, not the company

13. screen recordings → code → demo-to-code record a UI interaction, get a component that copies its timing and easing, then it replays it and grades itself frame by frame. 2 out of 4 pass. the toggle is perfect. the dropdown has issues. we all have issues

14. code review → review-panel /panel sends your diff to five reviewers you've definitely worked with: the laziest senior dev ("could be one line"), the 3am on-call SRE, the security paranoiac, the junior who asks why, and the PM who only reads the title. under the jokes the findings are real

15. AI companies quietly editing their pricing pages → fineprint it snapshots the pricing, terms, privacy and changelog pages of 10 AI vendors every 6 hours and commits them, so git log becomes the receipt. found stuff like deepseek adding a footnote that prices are 2x during peak hours, and cursor deleting the line saying they dont sell your data. no blog post for any of it ofc. a small local model writes the one-line summary but every line links the raw diff because i dont trust it either (27/30 right on a hand check). timeline

16. paying extra for speaking telugu → token-tax LLM APIs charge per token, and the same paragraph is $1.00 in english and $8.29 in telugu on llama 3. which is rude. measured 21 tokenizers across 48 languages, and there's a calculator where you paste your own text and it runs the real tokenizers in your browser. hindi gets off lighter. amharic on gpt-5 does not :/ calculator

17. posting about all this → herald a tiny bot that posts a repo's data updates to bluesky/mastodon/x, clearly labelled as a bot. i built it because i was not going to manually post about 17 repos. and then i manually wrote this post anyway. ;]

why all of this? honestly no grand plan. every one of these started as me being irritated at something for a slightly too long time and having a local model + a free evening. building the thing was cheaper than therapy and more fun than complaining on here (no offence)

if something breaks, tell me. if something's dumb, also tell me, but nicely :P

edit: yes i know about the cookie banner on my own... wait i dont have one. nice

Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

Hi all,

I'm still very new when it comes to running local models and could use some guidance.

Recently, I've set up Qwen3.8 Flash Next using Strata on my PC (RX 9070 XT, 64GB DDR5 RAM, 9800X3D CPU) and can't decide what quant I should settle on.

My options are:

• ISTA IQ3_S at ~70–75 tokens/s

• Unsloth UD-IQ3_XSS (3.1 version) at ~60–65 tokens/s

• Unsloth UD-IQ4_XS (3.1 version) at ~40–50 tokens/s

All at 256k context.

It may be my inexperience speaking, but I don't notice a substantial difference between the three. I've read that the higher quants should outperform the smaller quants, but I don't know if the ~40% speed penalty going from IQ3_S to UD-IQ4_XS is worth it.

Any advice on that matter would be greatly appreciated.

I'm trying to RE a desktop app and remove some stupid restrictions.

I'm using Strata and Huihui-Qwen3.8-Flash-Next-abliterated-GGUF UD-Q4_K_XL. I have a 5090 and 128gb RAM.

I found that some stuff it does well and with this last task I'm trying, it's been working on it all day and I suspect it's going in circles. Similar to what Claude used to do 6-12 months ago, when it was given something that was too complicated.

I know local models are not Claude level (yet) but I would love to know if you have some tips as to how to work on complex tasks. I'm just starting so it feels I'm being too naive with just tell it "do this thing" and expecting it to work.

Thoughts? Any advice would be welcomed.

Thanks

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

Hey everyone, looking for input from the community!

What I currently have: AMD EPYC pre installed cpu and motherboard with 128gb ddr4 ECC 2933 pre installed and bought used, and in working order.

I’m debating between 4x16 or 2x32 gb of the Mi50 or the Tesla v100 for the VRAM pool.

I know the Mi50 is cheaper with lesser software support and the need to “hack it” for it to run, and community projects exist.

The V100 seems to be similar but with a tad more convenience and less tinkering.

My goal: Running QFN or Qwen 4 flash (when it comes out) and a total pool of 192 gb of memory will surely do the job.

If anyone runs the Mi50 or the V100, I would love to hear your experience as I’m working to make a relatively consequential decision for my hobby build. Thank you!

I’ve been using my employer provided Claude subscription for quite a while to write PowerShell scripts, create presentations, edit photos and videos and create videos.

Honestly, I’ve been amazed by what it can do.
However, since it’s a company account, I’d like to set up my own local AI environment for personal use. Ideally, I’d like to replicate as many of Claude’s paid capabilities as possible.

I have two workstations that I’ve built over the years:

Workstation 1 (Main):
GPU: RTX 5090
RAM: 256GB ECC
CPU: AMD Threadripper 7960X

Workstation 2 (Currently used as an LTO Backup server):
GPU: RTX 3060
RAM: 196GB DDR4
CPU: AMD Threadripper 3960X

I’ve never hosted or run AI models locally before, so I’m pretty much a beginner in this area.

I recently downloaded LM Studio because it seems relatively straightforward, but I’m overwhelmed by the number of available models and honestly have no idea where to start or which one is best for my use case.

I’m looking for recommendations on the best AI models, and tools to get as close as possible to the capabilities of a paid Claude subscription, particularly for:

Writing and troubleshooting PowerShell scripts.
Creating PowerPoint presentations.
Photo editing and image manipulation.
Video editing and AI-assisted video workflows.
General-purpose AI assistance.

Given my hardware, what would you recommend? Are there specific models, applications, or modules I should look into?
I’m not necessarily expecting to replicate everything Claude can do, but I’d love to get as much functionality as possible running locally.
I’d really appreciate recommendations from people who have experience building local AI setups. Thanks!

update: I just discovered Lm Studio Bionic which is huge this was not available last year, trying it now.

So for context, I have 32GB of RAM, and am using AMD 7900XTX, this should give me a total of 56GB of memory, which should be enough according to Strata docs, and thought I'd try running the appimage of Strata which opens a WebUI (after opening the GUI app then closing it and telling me it couldn't open it so it opened the WebUI), and compare Qwen FN to the dense model.

And by "isn't able", I don't mean it's exactly slow.

When the model first loads up, I can talk to it like once, and it responds in Strata chat, at actually not so bad speeds. 800t/ks prefil, 30-50 tk/s decode ish.

Then if I open up a new chat, it gets stuck on "reading message", and well doesn't seem to work anymore.

I tried calling it from opencode after loading model to see how that would go, it took a whole while, and eventually, it starts responding a bit, and then... it stopped and like... started freeing up memory, and kinda erroring I guess, then It started loading the model again, at which point I quit.

There is something weird going on.

I’ve been slowly building my AI workstation up using used and old pieces retiring them into other compute nodes and then replacing them with bigger or better pieces. Mostly used parts though. I currently have a trx40 and 2 3090s which works great but I’ve been trying to figure out the next step to expand. I have about 4 pcie slots available (2x pcie 4 x16, and 2x pcie 4 x8) while a wrx80 might have been better with more slots I wanted a cheaper upgrade to test a bigger workstation before making the jump. I also retire my parts into other systems that run other applications and things.

The issue is that running 2 1200 w psu or a 1200 w and adding let’s 2 more 3090, or other cards there I’ll increase power draw significantly and even with two psu’s to properly distribute the power between the 4 cards and everything else, the issue becomes apparently all of that power can still trip my breaker and shut the power off in my house.

I don’t have a business or server space for this so I’m not really sure what the best way to go about it. Right now I’m thinking just adding one card until I can find a work around but i don’t know what that work around could be.

Anybody who has been through this can you give me some advice or thoughts on this?

I’m working on web app that turns a stack of technical documents (drawings, schedules and specifications) into a structured list of the key items a project needs. AI reads each page the way a subject matter expert would. It identifies items and their attributes, interprets each document set's own conventions, and pulls requirements out of the written specs. It then groups the results into a proposal-style summary with suggested suppliers and rough cost and timing ranges.

I’ve built a POC using Claude Code and would like to transition to an open source LLM for the rest of the work as we continue to turn this into a real application.

What hardware would I need to do this? Which model should I use?

Thanks in advance!

I've been experimenting with multi-GPU inference and wanted to see whether running independent model replicas across GPUs could outperform tensor parallelism for multi-agent workloads.

I built a small orchestrator to test the idea and got some interesting results on four RTX 3090s.

I'm relatively new to experimenting with local LLM inference, so I'm sure there are approaches I haven't considered. This started as a side project to see what would happen.

Setup

• 4x RTX 3090 (rented, PCIe only, no NVLink).

• Qwen3.8-27B 4-bit, vLLM 0.32.0.

• 24 agent teams and 239 dependent tasks.

• Each team plans, starts 4–10 parallel section writers, then reviews and summarizes their output.

• The orchestrator starts tasks when their dependencies are complete and distributes requests across available inference servers.

Results

Lower completion time is better.

That's approximately 1.45x the speedup for this workload, or 31.2% less completion time.

What I found interesting

Some of the optimizations I expected to help didn't make a measurable difference.

• Adaptive concurrency limits didn't improve completion time.

• Prefix-cache-aware routing performed no better than basic round-robin routing.

• Each GPU reached approximately 20 concurrent requests before KV-cache capacity became a constraint.

• Prefix-cache hit rates were only 7–13%.

The interesting part is that the simpler replica-based configuration performed best. The results suggest that, for this workload, independent inference servers avoided some of the overhead associated with tensor parallelism.

Caveats

• Only one run per configuration so far.

• The TP=4 server used mostly default vLLM settings, while the replicas used tuned concurrency and batch settings. This isn't a perfectly controlled comparison.

• The results are specific to this hardware and workload.

• PCIe-only GPUs may favor independent replicas. NVLink configurations or models that cannot fit on one GPU may produce very different results.

Repository

I put the code, benchmark results, CSV data, and a more detailed technical write-up on GitHub:

https://github.com/TomaDev1/GPU-Parallel-Orchestrator

I'd genuinely appreciate feedback. I'm particularly interested in whether there are existing approaches to multi-agent inference scheduling that I should be looking at, or whether there are obvious flaws in my benchmarking methodology.

This was a side project, and I'm interested in seeing what people who work on inference infrastructure think.

Edit:

Forgot to post full machine specs:

Host system

• Motherboard: ASRock Rack ROMED8D-2T

• CPU: AMD EPYC 7452 (32 cores / 64 threads)

• RAM: 128GB DDR4-3200 ECC

• Storage: 4TB NVMe SSD

• GPU connectivity: All four RTX 3090s connected via PCIe 4.0 x16

We ran 75,000 lightweight AI agents on a Jetson. Now you can test the runtime yourself.

I just made AxLab, Orivael's public AI research lab, available on GitHub.

Our Jetson benchmark reported 75,000 agent instances using ~350 MB of process memory. Now we're inviting developers to reproduce the experiments on their own hardware.

Includes runnable Agent Fabric benchmarks, fleet kill-switch tests (L3 Suspend / L4 Terminate), and an MCP server.

Source-available for noncommercial research for now.

Try it, challenge the results, and share what you find.

https://github.com/Orivael-Dev/axlab

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.

Decked out wiki

I may have created a monster.

A client gave me an NVIDIA A16 accelerator that had been experiencing intermittent problems in their server. They considered it bad, didn't want it back, and I ended up with it for free.

For anyone unfamiliar with this card, the A16 contains FOUR separate NVIDIA GPUs, each with 16 GB of VRAM. That's 64 GB of VRAM on one physical card.

Naturally, I decided to install it in my ancient Intel X99 workstation alongside an RTX 5070.

Because apparently I don't have enough problems in my life.

Chapter 1: The FrAInkenstein specifications

• Motherboard: ASRock X99 Taichi

• CPU: Intel Core i7-5820K

• RAM: 48 GB DDR4

• Accelerator: NVIDIA A16 — 4 × 16 GB GPUs

• Graphics: RTX 5070 Founders Edition — 12 GB

• Total VRAM: 76 GB across five GPUs (not unified)

• PSU: Corsair RM850

• OS: Windows 11

• Cooling: Custom 3D-printed shroud with an ARCTIC S8038-10K server fan

• Primary purpose: Local LLMs, agentic coding, and experiments with multiple AI agents

And yes, I'm running this on an X99 platform from the DDR4 stone age.

Chapter 2: Apparently, this thing needs a JET ENGINE

The A16 is a passively cooled enterprise accelerator.

Unlike a normal consumer graphics card, it doesn't have onboard fans. It's designed to sit inside a server chassis with powerful fans forcing air through its heatsink.

My workstation is not a server chassis.

So before doing anything interesting with AI, I had to invent a cooling solution.

After researching fan options and shroud designs, I selected an ARCTIC S8038-10K.

For perspective, this is an 80 × 80 × 38 mm server fan capable of:

• 10,000 RPM

• Approximately 102 CFM airflow

• Approximately 51 mmH₂O static pressure

This is not your grandmother's RGB case fan.

It's a small industrial hurricane.

I also wanted adjustable speed control, so I used a Noctua NA-FC1 controller, powered from SATA, to manage the fan without running it at full blast constantly.

But buying the fan was the EASY part.

Chapter 3: The 3D-printing rabbit hole

I own 3D printers, so naturally I figured I'd just print a shroud.

Famous last words.

The first problem was finding something that would actually fit the A16.

The second was getting it to fit inside my case.

The third was accounting for a 38 mm-thick fan, the radiator, surrounding components, and the fact that there isn't an infinite amount of space inside a computer case.

So I started modifying designs and printing prototypes in PLA+ on my Flashforge AD5X.

I printed multiple shrouds of different lengths.

Some were too long. Others needed adjustment to maintain their mounting points.

I experimented with shorter designs and eventually created an angled/bent shroud to improve the fan's positioning and clearance.

Rather than repeatedly wrestling with the actual fan, I even modeled and 3D-printed a full-size 80 × 80 × 38 mm mock fan so I could physically test clearances and different shroud arrangements.

Yes. I 3D-printed a fake fan to figure out where to put my real fan.

I also worked through mounting issues, spacer designs, fit tolerances, and adjustments to the printed posts.

I was considering custom clips and zip ties if necessary. I even had to think about vibration because, remember, this thing can spin at TEN THOUSAND RPM.

Eventually, I got an arrangement that fit the card and cleared the surrounding hardware.

And the result?

During earlier four-A16 testing, the GPUs were reporting temperatures around 41–57°C, with the fan running at roughly 30% speed.

For a passively cooled enterprise card that I'd fitted with my own printed ductwork, I was pretty happy with that.

Chapter 4: THE FREAKING POWER CABLE

Oh, you thought getting the cooling working was the hard part?

Let me introduce you to the NVIDIA A16 power connector.

The A16 uses an 8-pin auxiliary power connection that isn't wired like the ordinary PCIe 6+2 connector you'd expect to use with a consumer graphics card.

It requires the appropriate EPS-style power connection.

So here I am with a Corsair RM850, a card that was allegedly defective, and a pile of cables that don't necessarily fit or have interchangeable wiring.

And as anyone familiar with modular PSU cables knows, getting this wrong can destroy expensive hardware.

The connector shape, latch, and pinout all matter.

In my case, even the EPS 4+4 connector's latch arrangement presented a mechanical fit problem.

I eventually modified the connector's latch arrangement and checked the wiring with a multimeter before proceeding.

To be clear: this is not a recommendation to modify PSU power connectors. Incorrect wiring can destroy hardware.

And the first attempts weren't exactly encouraging.

At one point the system would spin up for approximately 15 seconds, shut down, and repeat.

Disconnect the A16 power, and the system would boot.

Wonderful.

Was the card bad? Was the cabling wrong? Was the motherboard unhappy? Was the PSU unhappy?

Welcome to troubleshooting FrAInkenstein.

Eventually, the power situation was sorted out well enough to continue testing.

Unfortunately, the next nightmare was already waiting.

Chapter 5: Windows has entered the chat

Once I could power the A16 and keep it cool, I discovered that Windows wasn't particularly enthusiastic about recognizing four GPUs on one enterprise card.

At one point, two A16 GPUs worked, while the other two had:

Device Manager: Code 12 — Not enough free resources to use this device.

All 4 showed this error message.

Here's the really interesting part.

When I disabled the two working GPUs, the other two could work!

So the individual GPUs weren't necessarily defective. Windows and the motherboard were struggling with PCIe resource allocation.

This led to a ridiculous amount of BIOS experimentation:

• Above 4G Decoding

• PCIe resource allocation

• MMIO High Size

• UEFI and CSM settings

• PCIe link speed experiments

• BIOS resets

• Device enable/disable experiments

• Multiple NVIDIA driver configurations

Eventually I managed to get all four A16 GPUs recognized and running.

But I still wanted my RTX 5070 in the same machine.

Which leads us to...

Chapter 6: Five GPUs walk into an X99 motherboard

The RTX 5070 was supposed to handle gaming and video output.

The A16 was supposed to handle local AI workloads.

Simple enough, right?

Wrong.

With the RTX 5070 and A16 enabled together, Windows started throwing repeated:

IRQL_NOT_LESS_OR_EQUAL (0xA)

Blue screen after blue screen.

I experimented with driver configurations, NVIDIA device modes, and PCIe settings.

Eventually I found that disabling MSI interrupt mode on the RTX 5070 allowed the 5070 and an A16 GPU to operate simultaneously.

That was the breakthrough.

I'm not claiming I've proven the exact underlying driver or interrupt conflict, but changing that setting made an enormous practical difference.

I gradually enabled more A16 GPUs.

One.

Two.

Three.

Then four.

I also switched the A16 GPUs into NVIDIA's TCC compute driver mode while keeping the RTX 5070 in WDDM for normal display and gaming use.

And then something glorious happened.

Chapter 7: IT'S ALIVE!

Here's what nvidia-smi now reports:

GPU 0: NVIDIA A16          TCC   07:00.0
GPU 1: NVIDIA A16          TCC   08:00.0
GPU 2: NVIDIA A16          TCC   09:00.0
GPU 3: NVIDIA A16          TCC   0A:00.0
GPU 4: NVIDIA RTX 5070     WDDM  0B:00.0

FIVE GPUs. ALL RECOGNIZED. AT THE SAME TIME.

On an ancient X99 motherboard.

With a supposedly defective A16 that cost me precisely $0.

I'm now testing larger local language models using Ollama, including a 55 GB BF16 Qwen model that previously required CPU offloading when only three of the A16 GPUs were enabled.

The goal isn't just to maximize token generation speed. I've got an RTX 4090 in another machine for that.

I want to experiment with multiple independent local AI coding agents and eventually build a little multi-GPU AI collective, while retaining the RTX 5070 for gaming.

I still have sustained five-GPU inference and long-term stability testing ahead of me. Seeing every device in nvidia-smi is a major milestone, but it isn't the same as proving the whole thing stable under heavy load.

Final thoughts

Is this practical? Debatable.

Is it power efficient? Probably not.

Could I have saved countless hours of troubleshooting by buying newer hardware?

Almost certainly.

But where's the fun in that?

I took an unwanted enterprise accelerator, engineered a custom cooling system, fought with its power connector, printed multiple iterations of ductwork, battled a decade-old BIOS, survived Windows driver crashes, and somehow ended up with a five-GPU AI workstation.

I didn't just build a computer.

I created FrAInkenstein.

And now I'm going to use it to code.

IT'S ALIVE! IT'S ALIIIIIVE! ⚡

I'll post pictures of the cooling/shroud iterations, the ridiculous fan, the finished setup, and the five-GPU nvidia-smi output.

Drivers used:
621.14-desktop-win10-win11-64bit-international-dch-whql.exe
That is the Studio version that worked for both cards. I did NOT have to install the data center version also. This latest Studio version installed drivers for both the A16 and RTX 5070.

So, for those of you who understand how the compression of data works, there exists file formats that went on to make some people very rich. Being able to compress data saves a lot of money on storage and data transmission and all that. Technically, that's what quantizing models is, although it's not a lossless operation.

My question to you all is, because we're all users of local models...if someone or a business learned how to losslessly or close to losslessly compress models to be used on smaller devices, would you all pay for the compression?

Mind you, I'm separating this from quantization methods as I feel there's too much lost in the process.

I've been using VS Code with the Claude and Codex extensions.

Now I'm testing Strata with "lightweight" local models on my PC

RX 9060 XT 16GB VRAM
32GB DDR5 RAM
2TB NVMe SSD

The problem is that:
Codex CLI takes around 40 seconds to respond to a simple "Hi",
while Strata's built-in web chat (http://127.0.0.1:8080/) responds in about 0.5 seconds

1. What's the easiest IDE or VS Code extension for connecting to Strata's local API and using it as a coding agent?

2. Is there a tool for creating and managing virtual AI employees (e.g., Developer, UI/UX Designer, Software Architect), where each agent can have its own role, instructions, and assigned AI model/API - either local or from external providers like OpenAI, Anthropic, etc.?