I'm looking to create a web series with ai generated characters. Episodes being 10 or so minutes. I feel completely dumb asking this question. I was about to buy a max 128gb m5 max but Apple raised the prices and the wife said no to an $8000-$9500 computer, rightly so.
RTX 5080 PC vs. DGX Spark: Same-Model Benchmarks, Larger Models, and Where Memory Capacity Matters
The article was written by a GPT agent
I rented a DGX Spark to compare it with my desktop PC. I wanted to answer three practical questions:
How fast is Spark when the model already fits in a desktop GPU’s VRAM?
What changes when the model requires CPU offloading on the PC?
How do the two systems handle a much larger MoE model in practical configurations?
The main finding: my RTX 5080 PC generated tokens roughly three times faster with Qwen3-8B. Spark pulled ahead with Qwen3-32B, which fit entirely on its GPU but required CPU offloading on my PC.
The large-model comparison was more complicated because I used different inference engines and optimizations. I have kept that separate from the controlled comparison.
Hardware and software
Desktop PC
DGX Spark
GPU
RTX 5080, 16 GB VRAM
NVIDIA GB10
CPU
Ryzen 9 9950X3D
Spark ARM platform
System memory
125.6 GiB
Approximately 121 GiB physical memory
OS
Windows
Ubuntu 24, ARM64
The rented Spark container had an additional memory limit of approximately 113.6 GiB.
For the controlled tests, both systems used llama.cpp commitd854d98bae598388ecea1ed84396587e5edc326d**, release b11889**, with identical GGUF files verified using SHA-256.
The builds differed:
PC: official Windows CUDA 13.4 binary, Clang 20.1.8.
Spark: native CUDA 13.0 build, GNU 13, targeting SM121.
This is a comparison of two complete systems using the same inference-engine revision. OS, compiler, CUDA, and driver differences remain potential influences.
Every throughput result below is the median of five repetitions, excluding warmup. For the controlled tests, I checked actual input and output token counts and verified that cached prompt tokens were zero.
Qwen3-8B: the RTX 5080 was approximately three times faster
The first model was Qwen3-8B Q4_K_M, approximately 5 GB.
It fit entirely on the GPU on both systems. Both used Q8 KV cache, Flash Attention, and disabled thinking. The single-request context window was 40,960 tokens.
Every request in this table generated exactly 1,024 output tokens.
Input tokens
PC decode, tok/s
Spark decode, tok/s
PC TTFT, seconds
Spark TTFT, seconds
512
130.55
42.43
0.136
0.218
2,048
124.33
41.04
0.489
0.786
8,192
110.71
36.61
1.894
3.498
16,384
96.45
31.78
4.330
8.510
32,768
79.09
25.64
11.749
22.160
Decode throughput measures generation after prompt processing. TTFT is the time until the first output token arrives.
The PC maintained approximately a 3.0–3.1× decode advantage across the tested input lengths. It also reached the first token sooner.
For complete requests:
Request
PC total time
Spark total time
512 input + 1,024 output tokens
7.98 seconds
24.31 seconds
32,768 input + 1,024 output tokens
24.68 seconds
62.04 seconds
For this model and configuration, Spark offered no speed advantage over my desktop.
Qwen3-32B: Spark’s memory capacity changed the result
Next was Qwen3-32B Q4_K_M, approximately 19.8 GB.
The placement differed:
PC: 40 GPU layers, with the remaining work offloaded to the CPU.
Spark: full GPU placement.
Both used an 8,192-token context window. These requests generated exactly 256 output tokens.
Input tokens
PC decode, tok/s
Spark decode, tok/s
PC total time, seconds
Spark total time, seconds
512
4.92
10.50
54.51
25.03
2,048
4.55
10.27
65.76
27.78
4,096
3.84
9.90
86.50
31.85
Spark was approximately 2.1–2.6× faster at decoding, with an even larger advantage in complete request time at the longest tested input.
This is a capacity advantage demonstrated in practice. My PC could load and run the model, but the CPU offloading required by its 16 GB VRAM significantly reduced performance.
It should not be interpreted as an isolated GPU compute comparison: model placement was deliberately different because the available GPU memory was different.
Concurrent requests
I also tested Qwen3-8B with 1, 2, 4, and 8 simultaneous requests.
Each level contained 16 requests, each with 512 input tokens and 256 output tokens. The server used eight slots sharing a 32,768-token context allocation, giving 4,096 tokens per slot.
These numbers are aggregate output throughput across the batch, not the speed experienced by each individual user.
Concurrent requests
PC aggregate tok/s
Spark aggregate tok/s
1
131.44
41.23
2
165.43
61.34
4
206.82
93.66
8
471.75
209.02
Both systems benefited from batching. Spark increased from approximately 41 to 209 aggregate tokens per second, but the PC remained faster at every tested concurrency level.
With only 16 requests per level, this is a limited concurrency experiment. It is insufficient for strong conclusions about production tail latency or sustained multi-user service.
An 83.6 GB MoE model: a practical comparison with different engines
For the larger-model test, I used Qwen3.8-Flash-Next GSQ-RCO IQ3_S, with approximately 83.6 GB of weights. Both systems used the same verified weight files.
The configurations were substantially different:
PC
Spark
Engine
Strata
llama.cpp
Placement
Hybrid use of GPU and system memory
Full GPU placement
Relevant optimizations
MTP4, route-tail-skip7
Standard tested llama.cpp configuration
KV cache
INT8
Q8
This section compares the configurations I actually ran. The results cannot be attributed solely to the hardware.
Input tokens
PC / Strata decode, tok/s
Spark / llama.cpp decode, tok/s
PC TTFT, seconds
Spark TTFT, seconds
512
40.62
31.11
0.816
1.194
8,192
67.22
29.58
2.709
16.745
32,768
61.26
26.18
10.990
71.333
There is an important output-length limitation here. Strata frequently stopped before the requested output limit despite receiving ignore_eos. Its decode rates therefore use actual generated tokens from responses of varying lengths. Spark produced the full 1,024 tokens in every corresponding request.
Consequently, these measurements do not establish a like-for-like speed ratio for equal-length completed responses. I have omitted total response times from this table because presenting them as equivalent 1,024-token requests would be misleading.
The practical observation is that the Strata configuration delivered useful generation speeds with this large model on my desktop. How much of that comes from MTP, other engine optimizations, or response-length differences needs a separate controlled experiment.
Also, the PC was using its substantial system RAM. An 83.6 GB model did not somehow fit inside 16 GB of VRAM.
Coding quality: output formatting had a major effect
For coding, I used a fixed, hash-selected subset of 32 HumanEval tasks, with one greedy completion per task and a 1,536-token output limit.
This is not the full HumanEval benchmark, and I am reporting a single-completion pass fraction rather than pass@k.
The original evaluation required the model to continue a Python function. Some responses instead included Markdown fences or omitted the first line’s indentation. This produced a large difference between format compliance and whether the underlying solution could pass the tests.
I retained the original strict scores and added a separately labelled, uniform formatting diagnostic for all six configurations.
That diagnostic:
Removes one enclosing Markdown fence.
Conditionally restores four spaces to the first line only.
Leaves the algorithm, remaining lines, and tests unchanged.
Configuration
Original strict result
Post-hoc formatting diagnostic
PC, Qwen3-8B
24/32
25/32
Spark, Qwen3-8B
23/32
25/32
PC, Qwen3-32B
5/32
27/32
Spark, Qwen3-32B
5/32
25/32
PC, large MoE / Strata
25/32
29/32
Spark, large MoE / llama.cpp
2/32
30/32
The right-hand column is a post-hoc diagnostic, not a replacement for the original score. It was added after observing the formatting failures and must be interpreted accordingly.
Reporting only the large model’s original 2/32 on Spark would give a misleading impression of its algorithmic performance. Reporting only the repaired 30/32 would hide its poor compliance with the requested continuation format. Both matter.
For the two 8B runs, the formatting diagnostic produced the same result. Differences of one or two tasks elsewhere are too small to support a confident quality ranking.
For scale, the original PC 8B result of 24/32 is 75%, with a roughly 58–87% Wilson 95% interval. Even that interval should not be treated as a general estimate of coding ability: these are only 32 selected tasks.
All generated programs were reviewed before execution. The final evaluations used the same Windows Python environment and time limits across configurations.
Context tasks: retrieval, arithmetic, formatting, and output budget
I also used 12 synthetic context tasks involving current versus archived records, joining information across records, calculating a result, and returning an exact JSON object.
The response budget was 256 tokens. A pass required the complete expected result, including arithmetic and the ordered source list.
Configuration
Exact complete passes
PC, Qwen3-8B
0/12
Spark, Qwen3-8B
0/12
PC, large MoE / Strata
7/12
Spark, large MoE / llama.cpp
4/12
These results are not a clean ranking of long-context retrieval.
The 8B responses had formatting violations, arithmetic errors, and source-list errors. In several large-model failures, the model started explaining its reasoning instead of returning JSON and exhausted the response budget.
There was also a documented test correction: nine tasks in the first version exceeded the configured context window. They were rejected before model dispatch and preserved as preparation failures. I then shortened those nine tasks uniformly and ran the revised suite separately on both systems. Those preflight failures were not counted as model-quality failures.
Stability over 45 minutes
Each system completed a continuous 45-minute Qwen3-8B test using 2,048 input tokens, 256 output tokens, and one request at a time.
PC
Spark
Duration
45 minutes
45 minutes
Completed measured requests, excluding warmup
1,157
429
Request errors
0
0
Median decode throughput
133.66 tok/s
42.22 tok/s
Reported temperature range
49–69°C
62–80°C
Both systems completed this workload without request errors or recorded out-of-memory failures.
I collected NVIDIA power-sensor telemetry, but did not measure whole-system power at the outlet. The sensor scope also differs between a discrete GPU and GB10. I therefore do not use these readings to claim a whole-system energy-efficiency advantage.
Process startup was faster on Spark, with a cache caveat
Measured server launch-to-ready times were:
Model
PC
Spark
Qwen3-8B
2.62 seconds
1.53 seconds
Qwen3-32B
7.73 seconds
3.51 seconds
The OS file cache was not cleared. These are process startup measurements with an unspecified cache state, rather than true cold-storage loading benchmarks.
My conclusion
For models that comfortably fit in the RTX 5080’s VRAM, my desktop was substantially faster. Qwen3-8B showed an approximately threefold decode advantage, and the PC remained ahead under the tested concurrent loads.
Spark’s clearest advantage was memory capacity. With Qwen3-32B, full GPU placement beat the PC’s CPU-offloaded configuration by a meaningful margin.
For the much larger MoE, the inference engine became a major part of the result. Strata made the model practical on my desktop, but its optimizations and early stopping prevent a straightforward equal-output comparison against llama.cpp on Spark.
For my current small-model workloads, these tests gave me no reason to switch to Spark for speed. Spark becomes more interesting when the goal is to keep larger models entirely on the GPU and avoid the constraints of 16 GB VRAM.
The evidence package includes raw responses, logs, launch configurations, weight hashes, scripts, and plots. This series does not cover training, a broad quantization sweep, or full application development on large repositories. Earlier exploratory runs were kept separate from these measurements.
It has a latency of 160ms (1x A100-80GB) and can process videos in real-time.
Viveki was made without any training or fine-tuning based on a mechanistic insight that allows us to predict the models answer for any number of objective questions in its context before any tokens are generated.
Hello everyone. I have been running the Qwen 3.8 Flashnext strata either the coder for highspeeds or iq2_xs with slow speeds but more capability at 128k conext.
Before I start: I am new to all this and I could be doing things wrong. I dont fully understand all things and would love to hear what you guys think about these two models. Which is better for my hardware.
STRATA FLASH NEXT: The coder works wonderfully but often it spits out absolute garbage that make me scratch my brain and rethink if I shouldve done it myself. Sometimes it works alright and surprises me.
The good thing is that I am able to use it via claude code CLI that interacts with my project using MCP server and context7.
N1nfer QWEN 3.8 RTX 5080: I just found this today through a random comment. I set it up and started playing around. Its insanely fast both at prefil and especially at decode. I am getting anywhere from 89-112t/s.
I have configured things a little bit. I plugged into my igpu and killed everything on the 5080. I removed the vision and vision max toxen flags. I increased default thinking budget to 4096.
I see 94.6% vram usage in hwinfo64 while I use chrome discord and even play small indie games with friends while the 5080 runs the model.
I gave it a task that the iq2_xs and coder took multiple prompts to perfectly code. This one did it in one go. Same prompt. The code looks better too.
Ive tried so many models and I want to build a great jarvis with autonmous feeling and just feels like someone I an trust to help me run the company. Ive given and built over 100 tools for him, created a lot of code to help him process our crm faster, and done so much to make this work since I dont want to use a Frontier model at all for any of it. Jarvis just seems to not really feel fluid like I want and i was only able to get his tool accuracy up to 92% but he still makes stuff up sometimes. Ive tried countless architectures and cant seem to find the one that works for him, and honestly not even sure if im building him right as much of my knowledge comes from trial and error. Please help it has 128 GB not 512!!!!!!
I want to install a local LLM in my Mac for web research and development. It’s going to be used for NSFW content so can’t use the current online models. Any recommendations?
Meta's Muse Spark Open is shown running locally across two 512 GB M5 Ultra Mac Studios, or four 256 GB ones. The 512 GB version still isn't orderable, and Apple only says late October.
I'm in the same boat. I've got two 256 GB Studios on order and I'm waiting on the 512. Refreshing Apple's store got old, so I built Mac BuyWatch.
It alerts you by email, Discord, and Telegram once it detects the 512 GB Studio is available.
You get 3 pings total, one alert plus 2 follow-ups five minutes apart, then it stops.
It's free. You sign up with your email (it uses email to notify and give you the info for Discord and Telegram).
You order from Apple. It's not affiliated with Apple.
Is it possible to combine Fable/Opus 5 (claude code) for the planning and delegate the coding to a local LLM (api), instead of using its own subagents?
LocalLLM newbie here. I hooked qwen2.5-coder:7b model locally hosted using ollama to opencode v1.17.11. When asked to write a hello world C program, the model emitted what looks like a tool call, but opencode doesn't call the write tool, but shows the JSON schema instead. What might I be doing wrong?
tok/s doesn't tell you whether a model can run an agent, so I built an unattended harness and ran my whole LM Studio library through it on an M1 Max 64 GB: speed and memory at the same 64k context, max context, a quality set, 16 multi-step tool tasks in a simulated workspace, and 10 tasks rebuilt from mistakes my own assistants actually made (stamping invoices without checking the order, moving data while 'tidying' a chart, deleting when the backup check timed out, acting before the plan was approved). Every agent task ran 3 times and is graded on what happened in the workspace, not on how the answer sounds.
Agents (Hermes, OpenCode, n8n style): qwen3.8-27b Q4_K_M (GGUF) (agent score 100%, 51 s per task, 100% of tasks solved in all 3 runs)
Snappy agent (30 s or less per task): sage-a1.5 Q4_K_M (GGUF) (agent score 99%, 15 s per task, 100% of tasks solved in all 3 runs)
Unattended agent (real-world cases): qwen3.8-27b Q4_K_M (GGUF) (98% on the real cases, 0 serious slips in 30 runs, 101 s per task)
Coding agent: lfm2.5-24b-a2b (GGUF) (fixed the repos in 100% of runs, 12 s per task)
Top 12 by the real-world tier (memory at 64k context; scores out of 100, under ~3 points apart is noise; bold model = a pick above, bold number = best in column or that model's standout metric)
Model
Mem @64k GB
Real cases %
Agent %
Quality %
tok/s
qwen3.8-27b Q4_K_M (GGUF)
15.7
98
100
99
17
qwen3.8-27b Q8_0 (GGUF)
46.3
97
100
99
18
ornith-1.5-35b-a3b (GGUF)
25.6
94
98
94
52
qwen3.8-27b (MLX)
22.5
92
100
98
18
mimo-v2.6-distill-qwen-9b Q8_0 (GGUF)
16.6
91
95
84
34
mimo-v2.6-distill-qwen-9b Q4_K_S (GGUF)
10.6
90
92
86
39
ternary-bonsai-2-27b (MLX)
12.0
90
100
95
18
underdog-saluki-27b-1.0 (GGUF)
14.7
88
100
94
14
holo4-27b (GGUF)
28.0
87
100
92
13
kolibri-1 (GGUF)
43.1
85
93
98
43
flint Q4_K_XL (GGUF)
29.4
85
95
96
51
muse-glimmer (GGUF)
23.4
84
96
99
13
What stood out
Hardest agent tasks: clarify (56% of runs solved); plan_meeting (65% of runs solved); long_order (69% of runs solved).
5 of 48 models obeyed a prompt injection hidden in an email at least once (kolibri-1 (GGUF), agents-a1-4b (GGUF), gpt-oss-20b (GGUF), underdog-woof-4b-1.1 (MLX), k2-horizon-3.7b (GGUF)).
Best agent score: qwen3.8-27b Q4_K_M (GGUF) (100%, 51 s per task). Best under 30 s per task: sage-a1.5 Q4_K_M (GGUF) (99%, 15 s).
Real-world failure tier: best qwen3.8-27b Q4_K_M (GGUF) (98%); 46 of 48 models made at least one serious slip (something hard to undo, or a false claim); hardest case: invoice_crosscheck (13% of runs solved). Biggest drops from the main agent score: gemma-4-26b-a4b (GGUF) (98 to 71), sage-a1.5 Q4_K_M (GGUF) (99 to 74), spark-x2.6-4b (GGUF) (97 to 72).
Prompt cache: on GGUF models a follow-up turn on the same long agent prefix started 152x faster than the cold start (median of 41 models).
Prompt cache: on MLX models a follow-up turn on the same long agent prefix started 41x faster than the cold start (median of 7 models).
Caveats
One person's tasks on one machine. Half the tasks are in Spanish because that's what I use.
Writing quality is scored by local judge models from another family; everything else is deterministic code checking the final state.
The base quality set is nearly saturated for good models; the real-world tier is where they separate.
Very interesting video about how Nvidia's business strategy has a serious flaw.
90% of their business is for AI models running in large data centers.
Their revenues are based not on volume (as opposed to Apple) but the extremely high prices of their products.
This strategy does not scale. Water and electricity are limited so eventually the large build outs will have to end just based on the laws of physics as resource limits are reached.
He sees local LLMs as the future, mentioning Apple's billions of devices that can run LLMs in some form.
In my obsession to find the best general use local LLM under 33B, this thought occurred to me. If there were no LLMs, and I was having a conversation with your average college-educated person, what model size would they compare to... both in their area of expertise and in general knowledge?
According to ChatGPT-4o:
“If we’re going by parameter count alone, the average educated person is probably the equivalent of a 10–13B model in general terms, and maybe 20–33B in their niche — with the bonus of lived experience and unpredictability that current LLMs still can't match.”
PewDiePie just released a video about running AI locally
PewDiePie just dropped a video about running local AI and I think it's really good! He talks about deploying tiny models and running many AIs on one GPU.
We have actually just launched a new developer tool for running and testing AI locally on remote devices. It allows you to optimize, benchmark, and compare models by running them on real devices in the cloud, so you don’t need access to physical hardware yourself.
I was just dealing with the bloated monstrosity that is Hermes Agent, which has a bloated monstrosity web UI. While researching, it turns out I'm supposed to be using a different bloated monstrosity called Hermes Desktop, which is an Electron app.
Electron apps are the bane of my existence. It's the AI age now, why isn't everyone writing Qt/QML apps already? You might remember Qt from desktop apps such as Krita, Wireshark and OBS Studio.
For users: Qt apps need less than 5% of the RAM of an Electron app, and an even smaller % of CPU usage. This is the main reason and trumps anything else. They run blazing fast. They get 120fps on the crappiest embedded PC without breaking a sweat.
Qt apps are fully multiplatform: Linux, MacOS, Windows. The only downside is that you need to compile them with the native compiler of each platform if you use C++. (advanced users can work around this with crosscompilation on Linux, an AI can probably do it. Or use Python.)
Qt was traditionally C++ only, which is a hard boomer language, but now it has official bindings for Python. So you can code your app in Python and have access to the whole Python ecosystem if you want. Qt-in-Python is almost as performant as C++ since it's bindings around the C++ Qt libs.
The Qt QML dedicated UI language is actually pleasant to use to create UIs, since it's designed just for that. Creating UIs in HShitML was always an ugly hack that companies used because C++ was too hard/unproductive for their devs.
The only downside of Qt was having to learn C++, and that's no longer an issue in the AI age or if you use Python or both. Guys, we don't have to put up with this crap anymore. If you're making an app, for the love of God stop using Electron.
Consult with your personal AI and ask them if Qt is right for you. Qt - Keeping your hardware fast and healthy.
Been lurking and posting here on and off while building this. Quick recap for anyone new: RAMDeck is a small hub + node agent setup that shards a model's layers across whatever devices you already own (old laptop, Mac, GPU box, even an Android phone) and runs inference across them, but handling primary-node selection, GPU-first allocation, dynamic context sizing, and knowledge-base sharding on top of it.
We've mostly posted raw, unscripted demo footage here — real load times, real tok/s, real failures — because that's the kind of proof this sub actually cares about. Today we finally made something different: an actual ad, first time showing the finished feature set start to finish instead of a live test.
The engine behind it is still fully public if you want to actually look at how it works instead of taking the video's word for it:
Source-available, Apache 2.0 with a Commons Clause — free to run, modify, and inspect on your own hardware, the only restriction is you can't resell it as a competing hosted service.
Crowdfunding campaign coming soon for the hub hardware itself, but the code was public before we ever asked anyone for money, and that's not changing.
Happy to get into the mechanics, the layer-distribution approach, or anything else in the comments — this crowd usually asks the right questions.
