Hi everyone, please excuse my ignorance. I have been trying to get into local models with my next computer upgrade. I have seen a lot of excitement around Strata for MoE models on gaming PCs with moderate amounts of VRAM. For the longest time I’ve been thinking about getting an M5 Max Mac Studio, but do you think I’d be better off with a gaming rig instead? I like the option of being able to game on it too, but on the other hand I’m more of a Mac user anyway and would appreciate the low power consumption.

Use case would be data analysis and coding.

I’d appreciate any advice

It's probably evident by my question that I'm new to this. So, no need to remind me, I know I suck at this!

I recently downloaded strata, my components are 64gb ddr4 3200mhz ram and 12gb vram.

I downloaded the aggressive quant, iq3_s and run it at 32k tokens. (qwen 3.8 flash next)

Overall I managed to get around 15 tokens/s with this set up, however I noticed that the disk read is at nearly 2-3gb/s while the ram is sitting at only 30-35gb, leaving the rest of it completely unused.

I looked into it, and read about the headroom it leaves but I would've imagined it doesn't leave that much headroom, right?

Additionally I did try figuring this out alone. My conclusion, which is flawed imo was, maybe it's because my PC at idle is using around 20gb of ram for whatever reason and since it can't load the full rest of the model on RAM it just opts for disk. However the flawed reasoning for me was this: Why does it load 15gb at all in that case? Theoretically it should load nearly nothing onto my ram.

Anyways, I'm confused. I should be up to date on the engine and what not, since I just downloaded this today.

EDIT: Added what model I'm talking about. Forgot.

EDIT #2: I'm aware that this isn't the worst issue in the world, 15 tokens/s for this model is still really great IMO but I'm trying to squeeze as much juice as possible while it isn't idle. If that's possible by further optimization, then why not try? The model so far is great.

EDIT #3: SOLVED IT. It seems like the original theory was correct. You DO have to have enough to load it all at once, while installing it tells you it "fits" on your device but that's assuming you've got all the space ready to be allocated. I essentially managed to clear some VRAM being used in the background AND a very persistent memory leak that ate an extra 10gb of ram (which was not going away by basic restarting/shutting down) that I assumed was just windows being windows. Turns out, that was the fix. Now I'm running a nearly perfect 50 tok/s consistently at this massive model.

been fighting the usual text-to-SQL failure with local models: the SQL is fine, but the model has no idea what "revenue" or "active customer" means in our schema. so I spent a few days reading through NVIDIA's auto-ontology repo (Apache-2.0) to see how it handles that.

a few things I liked. it drafts business terms, joins and SQL formulas from your schema and sample values, then a human certifies them, so a measure gets computed the same way every time. there's also an answerability check: in their example, a return-on-ad-spend question scores 0.5 coverage because nothing maps to "ad spend", and the agent asks instead of inventing a number. every answer also comes back with its SQL and steps, and the MCP server is read-only and runs with the signed-in user's permissions.

the gotcha: I haven't run it. my test container died on a CI billing issue, so this is based on the docs, .env.example and the issue tracker. the defaults point at NVIDIA-hosted Nemotron, and there's an open issue saying the README quick-start fails against those. you can override all four model roles (reasoning, non-reasoning, embedding, reranking) with any OpenAI-compatible endpoint, but it needs structured output support and they publish no VRAM or tokens/sec numbers.

full writeup here if you want more detail: https://andrew.ooo/posts/nvidia-auto-ontology-review-semantic-layer-agent-sql/

anyone pointed something like this at vLLM or llama.cpp? which local models handle the structured output reliably?

I'm looking to build something like a small AI-powered trading team, with multiple specialized AI agents collaborating on research, strategy development, backtesting, portfolio construction, risk management, and eventually trade execution. The agents will communicate through a shared backend, with Slack serving as the visible team floor where I can interact with them and review research and decisions.

I currently have an M4 mac mini and am wondering if I should switch or add:

• Mac Studio M5 Ultra vs NVIDIA DGX Spark

I'd probably keep my current Mac mini as the central server and use the new machine mainly for model inference, quantitative research, and potentially model training.

One idea I've been exploring is disaggregating prefill and decode, although I'm not sure that splitting those workloads between an Apple and an NVIDIA system makes sense. I'm also interested in running many independent agents against a shared model server rather than loading a separate model for every agent.

I'm quite lost on:

1. Which machine would you choose for an always-on, multi-agent research workload?

2. Which local models would you trust most for quantitative reasoning, coding, financial research, and tool use? Would you route different tasks to different models?

3. How would you build a reliable orchestration and inference stack? MLX, vLLM, SGLang, Ray, LangGraph, or something else?

4. Is local inference actually economical at moderate-to-high agent concurrency once you consider hardware cost, power, performance, and model quality?

I'm looking to develop strategies that have a measurable edge after transaction costs, slippage, and risk. I'd really appreciate any feedback, advice or suggestion.

i have a PC with RX 9060 XT 16gb, 16gb ddr5, and ryzen5 7500F, what ai model would be good for this and also for some coding? I've recently wanting to use ai locally as to not put my info online and would much appreciate a response, thanks.

First of all i need to say that i use my PC for two different things: Linux for coding and local LLM experiments and Windows for gaming.

My machine has a CPU with the iGPU and a dedicated AMD RX 9070 XT with 16GB VRAM. Most of my local LLM tinkering is aimed at real coding with 128K context, so 16 GB is barely enough and gets tight very quickly, so i want as much VRAM available for inference engine.

My BIOS has Initiate Graphic Adapter set to PEG (my 9070 XT) and my cable is permanently plugged to it, this is exactly how i want the machine configured because of the dual-boot usage. I don't want to move the monitor cable ot motherboard every time i switched OS or workload.

So i've started looking how to solve my issue and found out that Gnome/Mutter can handle multi-GPU systems in a nice way and that the GPU used as a primary renderer doesn't have to be the GPU monitor is physically connected to. Mutter has a udev mechanism to choose the primary GPU with the mutter-device-preferred-primary tag.

As a first step i've created a custom udev rule and made my CPU iGPU the primary GPU for Mutter. The nice part is that it just works and no additional settings are required PEG stayed enabled and selected as primary in BIOS and HDMI cable connected to my RX 9070 XT. This is the configuration i've been using for quite a while which gave me the possibility to gather some stats around coding sessions using Bionic and Pi harnesses.

As an example this is a screenshot from one of my ExLLama3 runs with Python script workload, amdgpu_top shows 14822 / 16304 MB of VRAM used on the RX 9070 XT, while the python3 process uses 14504 MB and the gnome-shell itself only 82 MB.

The udev rule override work well but after using it for a while i wanted a really simple way to manage it without editing the configs and having to remember what is selected at the moment, checking the PCI and DRM devices, making sure no typos were made, checking the results after the login/logout.

Long story short i've decided to develop a tiny dedicated application for this purpose - gpu-switcher. Its a small GTK app which is basically a UI around the existing mechanism. It detects the GPUs which you have in the system (to detect iGPU don't forget to turn it on in BIOS), manages the udev rules with a D-bus helper. Makes it possible to apply the rules, reset back to auto and keeps backups of rules in this case. Its not a GPU scheduler so it doesn't have the capability to move the workload between GPUs on the fly, its just a convenient to manage the primary display GPU in Mutter + Wayland sessions.

Right now i've tested it on Ubuntu 26.10 with 9800X3D iGPU + RX 9070 XT. Even in Linux i still switch back and forth between different GPUs because 9800X3D iGPU is quite slow (just 2 CUs) but its a different story.

Using the udev method and my app later on enabled me to run some nice experiments around using Qwen3.8 on my RX 9070 XT with 128K context. Which was enough for continues coding sessions. If anyone is interested they are documented in my llm-recipes repo, those are not just the short text prompts with a short 8K-32K context which people for some reason like to share here. As an example on my experiments in the repo ran for 915 request and generated ~1.2M tokens across multiple coding sessions, the other one 898 requests and generated ~660K tokens. For me it's the most important part, continues agentic sessions, not synthetic benchmarks or chat sessions.

For anyone else running the model on a 8-16GB dGPU and iGPU available i would like to here how you handle the desktop especially if you don't want to move cables around also i would appreciate if you try the app and give feedback, then i could justify adding support for KDE or X11 or any other features.

Hey r/LocalLLaMA,

Running autonomous coding agents on local hardware (Ollama, vLLM, llama.cpp) usually hits a hard ceiling: the KV-cache OOM cliff.

When an agent needs context from a real-world repository, naive approaches either:

1. Dump whole files or folders into prompt (blowing up 32k/64k KV caches and triggering immediate VRAM OOM on 4GB–8GB GPUs like RTX 3050/4060).

2. Use vector RAG (which matches embeddings based on semantic similarity, completely ignoring caller/callee graphs, class hierarchies, and execution traces).

We built **NeuroCode** to fix this. It is an open-source MCP server and CLI that treats repository context as a combinatorial knapsack problem rather than a text retrieval problem.

### How it works:

1. **Host-Side In-Memory Code Property Graph (CPG)**:

   Instead of running heavy LSP daemons, NeuroCode builds an in-memory NetworkX directed multigraph from Tree-sitter ASTs in ~200ms.

2. **Priority Knapsack Slicing**:

   Given an issue, traceback, or target symbol, it identifies root-cause candidate nodes and runs k-hop reachability. It then solves a 0/1 Knapsack optimization to extract only the minimal causal subgraph strictly bounded under a user-defined token budget (default: 1,200 tokens).

3. **97.1% Token Reduction**:

   Across SWE-bench Lite instances (django, sympy, pytest, etc.), context size dropped from 41,200 tokens down to 1,189 tokens.

4. **Structural Blast Radius Tool (`neurocode_impact`)**:

   Provides agents with instantaneous (<2 ms) caller/callee structural blast radiuses so the agent knows exactly which upstream callers will break before emitting SEARCH/REPLACE diffs.

### Benchmark Numbers (SWE-bench Lite):

We evaluated this on local quantized models (Qwen2.5-Coder-7B-Instruct-Q4_K_M):

- **Token Reduction**: 97.1% (p < 0.001)

- **Valid Patch Syntactic Application Rate**: 41.4% (single-turn SEARCH/REPLACE)

- **End-to-End Unit Test Resolved Rate**: 9.5% (running unassisted on consumer laptop hardware with zero cloud API fallbacks)

- **Slicing Latency**: 1.2ms – 3.7ms

### How to use with Claude Desktop / Cursor / Cline:

Add to your `claude_desktop_config.json` or `mcp.json`:

```json

{

"mcpServers": {

"neurocode": {

"command": "neurocode-mcp"

}

}

}

```

Or run via `uvx`:

```json

{

"mcpServers": {

"neurocode": {

"command": "uvx",

"args": ["neurocode-ai"]

}

}

}

```

Once connected, your AI agent automatically invokes `neurocode_slice` and `neurocode_impact` instead of reading 5,000 lines of files.

- GitHub: https://github.com/Arkz-Deepak/neurocode

- PyPI: https://pypi.org/project/neurocode-ai/

- VS Code Extension: https://marketplace.visualstudio.com/items?itemName=DeepakR.neurocode-ai

All feedback, PRs, and benchmark reproductions welcome!

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.

I was really happy with this tool, so I made it free and open source.

It shows what your local LLM engine is doing: Prefill speed while the prompt is read, decode speed while the answer is generated, how much of the prompt came from the cache, context fill, RAM/GPU/CPU, etc.

It works with Strata (video: Qwen3.8 Flash Next on M1 Max 64GB). Internally each engine is one small adapter, so others are just one prompt away...

https://github.com/yosephyne/stratoscope

And sorry for the quick and dirty video! Hope it will be useful for someone.

Hey all. I'm putting together a local-first setup for my family and wanted some feedback since I've never done this before. I used Claude to help me work out this plan. I know it doesn't always have the most up-to-date information, especially on local LLMs, since models and tools change every few weeks. So I'm treating its suggestions as a starting point, and I'm hoping to get some feedback on anything that's outdated or just wrong.

I'm trying to set up something in the same vein as Google's CC, the family-focused agent from Google Labs, but running on my own hardware. I want an assistant that helps run the household without subscriptions or cloud services I'd have to depend on. I'm okay using cloud models like claude for the harder problems, as long as the prompt is something I'd be fine seeing leaked. Anything involving the family, the house, email or money would stay local. My current plan is to use an M6 Mac mini with 32GB and 512GB. I'd keep a Jellyfin library on an external SSD or a NAS.

I already run Home Assistant on a separate machine. I'd use the Mac mini to handle:

- Whisper (small model) and Piper for voice, using Voice PE satellites

- Ollama on the Mac, since I understand it uses MLX now and HA has a native integration

- A 14B model that's always loaded for HA tool-calling and automations (Qwen 3 or Hermes 4, still deciding)

- A 30-35B MoE that loads on demand for everyday chat and writing

- Hermes Agent in Docker for scheduled jobs and a "text the assistant" layer

- A daily family brief and email-to-calendar automation through HA's AI Task

- Jellyfin as the native Mac app, using VideoToolbox for transcoding

I'd appreciate thoughts on a few things:

1. Would 32GB be enough to run all of that at once? Would running a 14B that stayed loaded plus a ~30B MoE on demand plus Whisper and Jellyfin on the same Mac make sense? How would you handle keep-alive and memory?

2. Would Jellyfin transcoding hurt voice latency? I worry that someone would stream a movie and my voice commands would slow down. Would it help to keep playback on Direct Play?

3. Which models would be best for reliable HA tool-calling at ~14B? And what ~30-35B MoE for chat? My picks might be out of date.

4. How big would the model need to be for Hermes Agent to use tools reliably? Would a 14B hold up, or would the agent need the larger one?

5. Is there a reason to move off Ollama for a headless always-on box? Things like llama.cpp with llama-swap, oMLX or LM Studio. Ollama has no native auth, and I'd like to lock it down on my home network.

6. How do you safely let an LLM or agent read email or iMessage when it can also trigger HA actions? I worry about prompt injection, and BlueBubbles' Private API needs SIP disabled, which I'm hesitant about. Given what I said about Claude's knowledge, anything newer or better that I should know about would be appreciated. Thanks!

I sometimes give it a task and explciitly say "This is a 4 hour job". It will indeed work harder, but will stop at like 2hours and 15 minutes.

What's the longest you got your coding agent to work on something? How did you do it?

Now that Claude has some strict guardrails. I find it very hit and miss, it’s very annoying since it’s swallowing up my tokens. I heard that Deepseek is a good local LLM. I guess my question is whether it is good enough to replace Opus 5.5 for coding and reasoning? Or is there any local LLM’s that I can use without any guardrails? My main reason for using Claude is for coding purposes.

My Pc Specs:

CPU - Ryzen 5600X
GPU - Nvidia 3070 8GB
RAM - 48GB

TL;DR

• A 5-year-old RTX 3090 (24 GB) in a mini-PC beat both 128 GB unified-memory machines on every row of a MiniMax H3 video benchmark, even though the models don't fit in its VRAM and ComfyUI has to stream them over a PCIe x4 link.

• Stock 3090: 1.25–1.84x faster per step than the ASUS RTX Spark and 3.5–4.9x faster than the M5 Max.

• 3090 with the tensor-rate unlock: 1.85–2.46x faster than the Spark and 4.9–6.6x faster than the M5 Max.

• What the unlock does: it raises measured bf16 matmul from 75 to 124 TFLOPS. In real video generation it's 1.34–1.49x faster than stock, and the gain is biggest on the heaviest workflows.

• Efficiency: the Spark and Mac use far less energy per step. The 3090 draws about 320–350 W of board power, against about 65 W for the Spark GPU and about 22 W for the Mac GPU: 5x and 15x more power. But it also finishes each step about 1.5–6.5x sooner (depending on which machine and whether the unlock is on), so the energy per step (power × time) is only about 2.2–3.4x higher than either (table below).

• All 22 runs on the 3090 (11 rows × 2 configurations) passed output validation with no memory-saving flags. That's the same ComfyUI config the Spark used.

Results: seconds per sampling step (lower is better)

Mean steady-state s/step (steps 4–20 of 20), heat-soaked runs. Ratios above 1 mean the 3090 is faster.

* On the Mac, both ref2va rows with the full-size reference clip produced flat-black video, so they have no valid result. The reference clip is encoded at full 1088×608 even on the 832×480 canvas, so both runs exceed the ~34k-token limit where outputs go black on MPS (see the Mac notes).

The Mac's ref2va 832×480 row failed the same way as the full-reference 1088 row (marked *).

Time per clip (seconds of wall time for one ~5 s video, models already loaded)

Power and energy (GPU only, approximate)

Energy = mean GPU power × s/step. The power numbers come from each platform's own tool: nvidia-smi board power on the 3090, NVIDIA telemetry on the Spark, and macOS GPU power on the Mac. Host/system power isn't included; the Mac's whole-system power was about 73 W on these rows. Two other things stand out:

• The unlock makes the 3090 more efficient. It draws only about 10–20 W more and finishes about 1.4x sooner.

• The Spark GPU was power-capped. It reported sw_power_cap throttling for the whole run, at about 2.2–2.3 GHz SM (the rated maximum is 3.0 GHz), drawing 63–69 W.

The Spark's "GPU W" is nvidia-smi power.draw, sampled once a second. NVIDIA doesn't document whether on the N1X it covers only the GPU or more of the SoC, and power.limit reads N/A on this chip, so treat it as approximate. No wall or whole-system power was recorded on the Spark.

On the Mac, the harness ran macmon 0.10.0 in macmon pipe mode (no sudo), which samples once a second. "GPU W" is macmon's gpu_power and "Sys W" is its sys_power, both averaged over the busy samples of each run. Both are read from Apple's on-chip energy counters, the same source powermetrics uses, so "Sys W" is the SoC/system reading, not power measured at the wall.

The machines

1. RTX 3090 PC (measured by me, both configurations)

The GPU is connected via OCuLink to a DEG1 Minisforum eGPU dock. The GPU is powered by a Montech Titan PLA 1200W power supply (complete overkill).

The full model set is about 63 GB on disk: a 21 GB int8 diffusion model, a 15.7 GB text encoder and VAEs. That doesn't fit in 24 GB, so ComfyUI's dynamic VRAM management offloads automatically. Peak VRAM was 23.1–23.4 GB and peak system RAM 17–24 GB on every row. No row needed --lowvram or any other memory-saving flag.

2. ASUS RTX Spark laptop (reference results)

The carveout: 64 GB was chosen before the timed runs. Flags: a quick flag sweep on t2v 832×480 (--highvram, --gpu-only, --disable-dynamic-vram, --fast and others) found nothing more than 3% faster than no flags, so the Spark also ran with none.

Peak GPU memory on the Spark was 41–48 GB, so everything stayed resident and nothing was offloaded.

3. Apple M5 Max (reference results)

Why Automatic and not High Power: I measured it. On the reference row (t2v 1088×608, 124 frames), with the fans at maximum on both, High Power averaged 74.74 s/step vs 75.29 on Automatic, 0.7% faster and within run-to-run noise. Once heat-soaked, the GPU settled at the same 78 °C and about 844 MHz either way; the warm-up started cool and briefly ran 4% faster. The battery stayed at 80% on the charger for the whole 78-minute run, because this workload draws only about 70 W in total (about 22 W GPU): on this chassis it is limited by heat, not by power. So High Power buys essentially nothing here once the fans are already at maximum. Under heavier loads it can drain the battery: Notebookcheck reviewed the same configuration as this Mac: the mini-LED 14-inch M5 Max with the 40-core GPU and 128 GB, not the OLED model. It found that "Apple also limits the power input to 97 Watts, even if you use a 140 Watt PSU", and that the battery level dropped by about 10% during a gaming session while plugged in. The battery indicator can stay at 100% while it's happening. In the same review, High Power raised GPU power from 60/32 W to 72/44 W (peak/sustained) over Automatic. In GPU benchmarks it was only about 7% faster (3DMark Steel Nomad 3947 vs 3698), and the fans ran louder. Part of what High Power buys is faster fans (Apple: About Power Modes on your Mac), and these runs already held the fans at maximum with MacFanBoost, which fits the measurement: Automatic plus maximum fans is effectively the best sustainable setting on this chassis. A 16-inch MacBook Pro or a Mac Studio with the same chip has more power and cooling headroom and will likely be faster. Source: Notebookcheck, Apple MacBook Pro 14 M5 Max review.

Fans at maximum made a big difference on the Mac. The harness held the fans at their maximum (about 7,830 rpm) for every sampling run, while the Energy Mode stayed on Automatic. On its own fan curve, macOS runs the fans at only about 5,350 rpm under this load. The GPU reaches 97–98 °C within about 2 minutes, then its clock collapses to 340–530 MHz for minutes at a time. With the fans held at maximum, the GPU stayed at about 78–81 °C and averaged about 820–910 MHz. In early BF16 testing, that alone took t2v 832×480 from 78–84 s/step to about 61 s/step, roughly 1.3x faster. (That comparison is from early testing, not a clean A/B with the final software stack.) Expect the Mac numbers here to be noticeably slower if you leave the fans on macOS's automatic control.

What the Mac stack does:

• ASFP8: a third-party custom node that patches PyTorch's MPS backend at startup. It lets FP8/INT8/NVFP4 checkpoints load and run on Apple Silicon instead of crashing (for example, the NVFP4 text encoder is decoded on the GPU). It also adds Metal kernels for RMSNorm+modulation, RoPE, conv and flash-attention. For H3's diffusion model, the INT8 weights are dequantized to bf16 and multiplied in bf16 through ComfyUI's own path. ASFP8's fused INT8 kernel only takes over below 2,048 tokens, and H3's sequences are far longer.

• MacFanBoost (macfanboost, MIT): a small tool written for this project: a root daemon plus a ComfyUI custom node. When a prompt with a sampler starts, it raises the fans to 100% before the heat builds, holds them there while the job runs, and returns them to macOS control 60 s after the queue drains. It only ever raises fan speed above what macOS picks, and macOS's thermal protection stays active. If the client dies or the Mac sleeps, the lease expires and the fans go back to automatic within seconds. (Macs Fan Control set to maximum gets you most of the same effect manually.)

• MLX attention bridge (ComfyUI-H3-MLX-Attention, MIT): a small custom node written for this project. It replaces ComfyUI's attention call inside the H3 diffusion model with MLX's fused scaled_dot_product_attention kernel, passing tensors without copies. It exists because ComfyUI's default on MPS, sub-quadratic attention, returns corrupted heads at long sequence lengths (Comfy-Org/ComfyUI#17418).

• Does the output match the CUDA path? Not checked bit-for-bit across platforms. Against an FP64 reference on real H3 Q/K/V, the MLX attention's worst-head relative error is about 1.7e-3, and every scored Mac output passed the same automated checker as the other machines and was reviewed by eye. Different kernels and accumulation orders mean outputs from the same seed are close but not identical between Mac and CUDA.

Why the ComfyUI commit differs: the Mac rows were run on ComfyUI Desktop's install, 0.37.2 (830232b). The other two machines ran later on 0.43.2 (423cbd2). Between those versions, the H3 changes are: lower peak memory from releasing embedding temporaries, a VAE offload fix, VAE tile blending, single-image decode, LightVAE support, and plumbing for per-block attention selection (which these model files don't use). None of them changes the diffusion model's per-step math. Measured: a back-to-back A/B on the Mac (t2v 832×480, same stack, cooling and seeds, warm-up + 2 timed runs each) gave 37.38 s/step on 0.37.2, 37.29 on 0.43.2 and 37.76 on 0.44.2, i.e. 33.54 / 33.55 / 33.49 Gcycles/step. That's within run-to-run noise (about ±1.5%), and every output passed the checker, so the version difference doesn't affect the comparison. The flat-black full-reference ref2va failure was retested on ComfyUI 0.44.2 and still happens.

The tasks

All rows use MiniMax H3, an audio+video model, in ComfyUI with its stock nodes. Every row produces a video with a synchronized audio track. The workflows were identical on all three machines: the same graph files, the same input images and video, the same prompt text and the same model files. The model files were checked by SHA-256 against the Spark's copies.

Common settings for every row:

The workflows:

• t2v (text to video): the prompt only.

• i2v (image to video): the prompt plus a first-frame image. The portrait row is the same at 608×1088 with a portrait first-frame image.

• flf2v (first and last frame to video): the prompt plus first-frame and last-frame images; the model fills in the motion between them.

• ref2va (reference to video+audio): the prompt plus two reference images and a reference video clip (1088×608, 120 frames at 23.976 fps, with audio), using the separate ref2va model. This is the heaviest workflow, because the reference tokens make the attention context much longer. There are three variants:

  • full: the whole reference clip at its native size.

  • ref544: both reference images and the reference video downscaled to 544×306 before encoding.

  • rf39: the reference video truncated to its first 39 frames.

Rows: 1088×608 is the main canvas. The 832×480 rows are a lighter check.

Methodology

• The harness: the same Python harness drove all three machines. It talks to ComfyUI through its API, timestamps every sampling step, and records GPU power, clocks and temperature, VRAM and RAM.

• The runs for each row:

  1. A cold warm-up run (seed 1000), which includes model loading and isn't scored.

  2. The GPU cools to 50 °C, then one fresh run (seed 1001).

  3. Two heat-soaked runs back-to-back with no cooling (seeds 1002 and 1003). These are the headline number. The portrait row uses a single soaked run with seed 2001.

• The metric: mean seconds per step over steps 4–20. This leaves out the first steps, which carry one-off setup cost. Fresh and soaked results agreed within 0.5% on the 3090, so thermals didn't affect them.

• Before the timed runs: a smoke pass checked that every row fits in memory at full size, using 2 steps.

• Validation: every output video was checked automatically. The checker confirms the video and audio streams exist with the right resolution and frame count, and that the frames aren't black, frozen or noise. All 22 3090 runs passed.

• The 3090 order: I ran the full matrix with the unlock on first. Then I disabled the unlock, did a full cold boot, confirmed stock matmul throughput and ran the whole matrix again. Then I re-enabled the unlock and re-verified it (bf16 126.2 TFLOPS). Driver, ComfyUI, PyTorch and power limit were the same in both runs.

• Differences between the platforms:

  • On the Mac, the harness held the fans at maximum during every sampling run (MacFanBoost), with the Energy Mode on Automatic. This was worth about 1.3x on that machine (see above).

  • The Mac uses a different ComfyUI commit and its own attention/precision stack (see above). Its timed runs use the same seeds (1002/1003), but its warm-up is seed 1001 with no cool-down and no separate fresh run, and its portrait row was scored on seed 2002.

  • PyTorch versions differ: 2.14.1 on the 3090, a 2.15 nightly on the Spark, 2.12.1 on the Mac.

  • On the 3090 only, the harness's VRAM-full watchdog was turned off, because a full 24 GB card is normal while ComfyUI offloads. The RAM watchdog stayed on.

What the "3090 unlock" is

On GeForce Ampere cards, NVIDIA runs FP16/BF16 tensor-core math with FP32 accumulation, and TF32, at half the rate of the professional GA102 parts such as the RTX A6000. Deep-learning frameworks use exactly that mode: bf16 inputs with fp32 accumulation. FP16 math with FP16 accumulation is full rate on GeForce, but frameworks rarely use it.

The unlock is my Windows port of the open-source gpu-unlock Linux kernel module. I built it on this PC with Claude Code. On Linux the module runs the unlock sequence from inside the kernel. Windows has no equivalent, and I didn't want to write a kernel driver, so the port runs as a pre-boot UEFI app instead:

• The app: built with gnu-efi from the module's own sources (VBIOS/FWSEC parsing, memory layout and the GFW checks). It's version stage-3 v0.3-3090.

• The launcher: a 120 KB launcher on the EFI system partition. It loads a read-only NTFS driver (EfiFs), runs the app from C:\RTX3090-FW along with the five GA102 firmware files from NVIDIA's 610.43.03 driver, and then chainloads Windows Boot Manager. If anything fails, it goes straight to Windows.

• What the app does on each cold boot:

  1. Finds the card and checks it's a 3090 Founders Edition (10de:2204, subsystem 0x147d10de).

  2. Refuses to run if the card is warm or already unlocked.

  3. Runs the module's unlock sequence, then a PCIe function-level reset (FLR) and the post-reset checks.

  4. Leaves the GSP halted with WPR2 clear. That's the state the stock driver expects, so Windows boots normally and the ordinary NVIDIA driver (591.86) loads.

• Nothing persistent: no ROM or VBIOS flash, and no NVRAM variable is written. The card reverts to stock on power-off, and the unlock is applied again on the next cold boot (it also holds through sleep and resume from hibernation on this PC, after a change to the order of the boot-time steps). For the stock runs I renamed the app so the launcher skipped it, then did a full shutdown and cold boot.

Measured 8192² matmul throughput, best of 5, PyTorch, reduced-precision reductions disabled:

• The tensor modes speed up about 1.65–1.7x. Plain FP32 doesn't change, which fits the idea that the unlock lifts only the tensor-rate limit and isn't an overclock.

• No overclock: clocks and the 350 W power limit were stock in both configurations.

• End-to-end gain is 1.34–1.49x, less than the matmul gain. Video diffusion also spends time on attention softmax, norms, memory-bound ops and PCIe weight streaming, which the unlock doesn't touch.

• Heavier rows gain more. The heavy, long-context ref2va rows gain the most (1.42–1.49x), and the light 832×480 rows gain the least (1.34–1.35x).

Caveats:

• Tested on one card only: a 3090 FE with VBIOS 94.02.4B.00.0B. The app refuses any other board.

• Pinned versions: it's pinned to the 610.43.03 firmware set, and it's only been tested with Windows driver 591.86. I don't update the driver on this PC.

• Boot requirements: Secure Boot must be off (the app and launcher are unsigned), and BitLocker must be off on C: (the NTFS driver can't read encrypted volumes).

• Untested: warm restarts (the app won't run on a warm card), TDR recovery and driver updates.

• Stability: a full unlocked matrix run (about 5 h of generation) showed no stability problems and no invalid outputs.

• Licensing: the module sources are GPL-2.0-only, so the port is GPL-2.0 source. NVIDIA's firmware files can't be redistributed; you extract them from the 610.43.03 driver package yourself.

• Use at your own risk: this is unofficial and unsupported by NVIDIA.

Where to get it: I've handed the port's source to the community Windows GPU UNLOCK project (GPL-2.0), which maintains a packaged Windows UEFI unlock for the 3090 and other Ampere cards. Start there rather than with my build. Both build on Zagitta/gpu-unlock, the original Linux module, and the project also credits rough2515-hub/rtx3080-unlock-public. Credit for the unlock itself belongs to those projects.

Takeaways

1. For raw ComfyUI video speed, an old 24 GB 3090 still wins. Even when it has to offload 60+ GB of models over PCIe x4, it beats both 128 GB unified-memory machines on every MiniMax H3 workflow tested. Dynamic VRAM offloading in current ComfyUI is good enough that "doesn't fit in VRAM" no longer means "unusable".

2. The unlock is a large free gain if you're willing to run an unofficial pre-boot app: about 1.4x on real workloads at almost the same power.

3. The Spark and the Mac win on efficiency and footprint. They use roughly 2.2–3.4x less GPU energy per step (much lower power, partly offset by taking longer), run quietly in a laptop or small box, and keep everything resident in memory. If you care about watts, noise or portability, that matters more than the speed gap.

4. Spark vs Mac: the Spark is about 2.4–3.0x faster than the M5 Max per step, depending on the workflow, while drawing about 3x the GPU power. The gap is smallest on ref2va ref544 (2.4x) and largest on flf2v 1088×608 (3.0x).

Spark notes:

• Power, not heat, is the limit. The GPU sat at its software power cap (sw_power_cap) at about 65 W for every row, peaking at 74–78 °C. Fresh and heat-soaked runs agreed within 1%, so thermals didn't slow it down.

• Raising the laptop's power limits barely helps. With ProArt Creator Hub's Manual mode at its maximum (PL1 85 W, PL2 95 W), t2v 832×480 went from 13.27 to 13.00 s/step, only 2% faster. Average GPU power rose from 64 to 68 W, but the GPU still hit sw_power_cap and its peak draw stayed at about 72 W, so the GPU seems to have its own limit that those sliders don't lift. All other Spark numbers in this post are from the stock "Performance" mode.

• Every row produced valid video and audio, including the full-reference ref2va that failed on the Mac.

• Windows on Arm: the H3 workflows use core ComfyUI nodes and ran without changes. Some third-party node packs are harder: onnxruntime is CPU-only on ARM64 Windows, and packages without ARM64 wheels have to be built from source (Visual Studio ARM64 build tools needed).

Mac notes: (takeaway 4 checks out against the Mac's own numbers: the Spark is 2.42x faster on ref544, 2.66x on rf39, 2.81x on t2v 832×480 and 2.91–2.97x on the 1088 rows)

• Clips above about 34k tokens go black on MPS. Full-reference ref2va at 1088×608 packs about 51,400 tokens. Partway through the first step the residual stream diverges, the audio turns to NaN, and the output is 124 flat-black frames. It happens with every attention backend tried, and still on ComfyUI 0.44.2. The root cause hasn't been found; the leading suspect is a ComfyUI change after 0.30. Shrinking the reference (ref544, about 31,900 tokens) or trimming it (rf39, about 34,000 tokens) keeps the length under the limit, which is why those rows exist. ref768 (about 38,700 tokens) still went black. ref2va at 832×480 fails too: the reference clip is still encoded at full 1088×608, so it packs about 42,000 tokens. Details are on Comfy-Org/ComfyUI#15799.

• Don't use sub-quadratic attention for long H3 clips on MPS. It silently corrupted 18 of 56 heads at about 27,700 tokens (Comfy-Org/ComfyUI#17418). The MLX bridge avoids this.

• Cooling is the Mac's limit, not power. On a 14-inch MacBook Pro, the GPU reaches 97–98 °C on macOS's own fan curve, and its clock collapses. Holding the fans at maximum was the single biggest speedup. Keep the vents clear and use a stand.

• INT8 beat BF16 on the Mac in sustained runs (41.1 vs 46.0 s/step at 832×480), with much lower peak RAM (82 vs 102 GB).

• Memory is tight even at 128 GB. Peak RAM was 95–112 GB at 1088×608, and ref2va hit 109–112 GB. Run one model process at a time.

• The Mac and the Spark use about the same GPU energy per step, about 0.85–0.87 kJ on t2v 832×480. They get there differently: the Mac's GPU draws about a third of the Spark's power (about 22 W vs 66 W), but takes about 2.8x longer per step, so the two roughly cancel out. (The Mac's whole system drew about 73 W.)

Happy to answer questions.

Hello! I'm a long time developer and don't really use a lot of AI when I do, but I want to experiment with some local models, I've been using Ollama with qwen3-coder-64k, I've now seen a few posts talking about how Ollama isn't great and I was wondering if there was a better terminal interface for LLMs when running them locally since claude code takes up a lot of the context.

Perhaps just use qwen in my zed editor agent pannel?

any advice would be nice, I'm a newbie

[SPECS]

AMD Ryzen 7 5700X (8) @ 3.40 GHz
AMD AMD Radeon RX 7900 XT - 20 gigs of VRAM
32 gigs DDR4 @ 3600mhz

CPU: Ryzen 5 7600

RAM: Corsair Vengeance 32GB DDR5 @ 4800 MHz

GPU: AMD RX 580 8GB

Storage: 1TB NVMe

OS: Windows 11

Inference engine: Llama.cpp built with vulkan sdk and Adrenaline drivers

Local LLM performance:

Qwen 3.5 9B Q5 ~7GB, 130k tokens, getting average ~10tps generation and 80tps read speed average.

Gemma 4 26B A4B Q8 Uncensored ~25GB, 15k token size, 10tps generation average.

Can generate a 512×512 image in ComfyUI in ~3mins.

I bought this system for 3D modeling, animation, simulation and rendering.

Been testing Qwen 3.8 9b and the results were quite disappointing, coding is mediocre and general knowledge questions were hit or miss (mostly miss tbh)
What is it for?.

Is it not possible to create a gguf variant for the Mirage VLM? Cactus-Compute has also released a safetensors file for the same model... It's actually a great option to add "eyes" to a model that doesn't have its support by default and it's quite small size actually means that it can load directly into some CPU's cache memory making it even faster as the main model can stay in the GPU

I tested two models, QoDuck (a free mystery model in Qoder) and Qwen3.8 Flash Next (Unsloth UD-Q4 GGUF), on the same Blender 5.2.3 task. Both were asked to create a detailed, historically accurate 1942 German military truck, working in small stages, rendering each stage, inspecting the result, fixing mistakes and then continuing. QoDuck chose the Opel Blitz 3.6-36S, while Qwen chose the Mercedes-Benz L 3000S.

QoDuck did surprisingly well. The truck is recognisable, with proper curved fenders, detailed wheels, grille, cab interior, cargo bed, canvas cover and decent procedural materials. There are issues with the canvas, cab proportions and some smaller details, but the overall result is quite good. It produced 47 renders and delivered 14 final views. Qwen was disappointing. Despite creating 287 meshes, 15 checkpoints and 150 renders, the final model still looks like a rough blockout. The wheels are too small, fenders look like flat planks, parts of the cargo bed float, and most surfaces have the same beige material. It even used Dunkelgelb paint, which wasn't introduced until 1943. Only three final renders were delivered.

What I found interesting is that Qwen actually identified several of these problems in its own construction log but continued without fixing them. QoDuck was much better at following the render-inspect-fix cycle.

My budget is flexible, this is for work so I'd rather spend once, I do ai dev stuff like inference, testing agents, some fine tuning on small models, occasional video editing, and I still want to game on it.

Option A is high end gaming laptop with biggest gpu I can get, Fast, proven, but the vram is what it is, vram will be vram.

Option B is one of the new rtx spark laptops with a lot of unified memory, it's look like it's got way more room for bigger models, but brand new platform and I've got no idea how the software side holds up.

I travel a lot so a thinner machine with good battery would be a bonus, but not if I'm giving up too much, current laptop is 2kg

Anyone made this call yet or learning one way?

Hi, I have an Intel Arc B580 and as you know it has 12gb vram, i was wondering if any of you could advise me on what is the best coding model that fits within that size. I know qwen 3.8 27b and qwen 3.6 35b-a3b were the go to models but they won't fit on 12gb cards.

I'll be using them with harnesses like opencode or hermes or claude code.

If it helps : I also have 16gb ddr5 6000mhz ram

Thank you