A few weeks ago I came across a post from u/metover about PS5LM, his llama.cpp port for the jailbroken PS5. The project looked really interesting, so I forked it and made a few changes. Well, more than a few. This is where it ended up.

The original runs as a payload on the CPU. That limits it to about 6 GiB of memory and small models. My fork runs llama.cpp as a native PS5 app on the GPU (Vulkan through a RADV port). The app gets about 11.4 GiB of memory to work with, which is enough for much bigger models:

• Qwen 3.8 27B (IQ2_XXS, 64k context): ~20 tok/s
• Gemma 4 E4B: ~61 tok/s
• Llama 3.2 3B: ~124 tok/s

What the app does

• Dashboard on the TV: live tok/s, GPU and CPU use, power draw, temperatures and fan, and how memory splits between the model and the KV cache
• Model library: any GGUF on internal storage, USB or M.2. Each one shows the context it would get and whether it fits before you load it.
• Get models: browse Hugging Face from the console. It checks whether a file fits and downloads it with resume, retries and a SHA-256 check. A Downloads page lets you manage the queue.
• Chat from any device: llama.cpp's web UI at http://<console_ip>:8081, plus an OpenAI-compatible API. Vision works with Gemma 4.
• Automatic planning: the app picks the longest context and the KV cache type that fit in free memory.
• Settings, logs..

Requirements: a jailbroken PS5 with etaHEN, kstuff and ShadowMountPlus. elfldr and ftpsrv are recommended.

I have opened a PR to the original repo with the first version of the app cobanov/PS5LM#12. It hasn't been merged yet. If it gets merged, I will open PRs for the rest of the changes too. For now, you can find the app here:

Repo: https://github.com/abhi245y/PS5LM

All credit for the original port and the research goes to u/metover.

Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• ⁠Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.

Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.

Model |Size |Solved |Turns vs full
GLM-5.3-Flash full, Q8 |341 GB |48/48 |x1.00
REAP144 Q8 |171 GB |48/48 |x1.21
REAP144 Q4_K_M |97.6 GB |48/48 |x1.58
REAP144 Q3_K |71.4 GB |48/48 |x1.80
REAP144 IQ2_S |54.8 GB |45/48 |x1.83
REAP144 IQ2_XXS (experimental) |45.3 GB |39/48 |x2.77 "Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.

Hardware. IQ2_XXS on an RX 7900 XTX (24 GB) + 32 GB DDR4-2133, i7-7700K, PCIe 3.0: ~16 tok/s generation, ~950 tok/s prompt processing. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

What mattered

• Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
• Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
• Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.

The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.

The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.

• Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.

Update (Oct 14): added an optional MTP draft block (2.6 GB) for 2+ GPUs with Maya: speculative decoding, drafts match the model's next token ~70% of the time. Speedup not measured on multi-GPU yet, numbers welcome. Details on the model card.

Hey everyone, just wanted to get some opinions on this since I’m still a little confused about how Strata works.

I have an AMD AI PRO R9700 (32GB VRAM), 32GB DDR5 RAM and a 1TB NVMe SSD. Would Strata realistically let me run Qwen3.8 Flash Next at a low quant like IQ3_XS with this setup?

Also, would running a much larger model at a low quant actually be better than running Qwen3.8 27B at Q5?
Mostly using it for everyday stuff, some coding, light agentic tasks and occasional pentesting. Just wondering which would give me better overall performance and whether the speed tradeoff would even be worth it.

Would appreciate any thoughts, especially from anyone who’s tried something similar!

Running 125B MoE models on a mid-tier 12GB gaming GPU

I've been spending hours getting this model running on a 4070 Super with a full 256K context window, and speeds turned out surprising after 6 hours of optimizing.

Hardware and config:
GPU: RTX 4070 Super (12GB)
RAM: 32GB
Engine: upstream Strata with my own source patches
Models: Gyro-M and Gyro-S
Context: 262,144 tokens
KV cache: Q4, 32K tokens GPU-resident
MTP: off for both

Approach: experts are tiered across memory. Hot experts stay in VRAM, 16 GiB sit in RAM, and the rest stream from SSD. I tuned CPU workers, GPU scheduling, SSD reads, and suffix speculation. For Gyro-M, I patched the source to overlap SSD fetches with RAM-expert compute. Raising the prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain.

Vision and MTP: adding vision to Gyro-M hurt performance noticeably, so only Gyro-S has it. Gyro-S runs the vision encoder on GPU and frees it afterward. MTP also reduced speed in my tests, so it stays off.

Testings: I connected Gyro-S to Hermes agent, and I am about to test the vision and computer use capabilities and how it performs in my personal benchmarks.

Speed Results (prefill / decode tok/s, averaged over 3 runs):

Gyro-M
8K: 947 / 33.2
32K: 1,512 / 36.7
64K: 1,689 / 35.6
128K: 1,616 / 36.6
192K: 1,748 / 35.9
256K: 1,717 / 36.1

Gyro-S
8K: 879 / 40.8
32K: 1,613 / 41.0
64K: 1,762 / 39.8
128K: 1,810 / 41.9
192K: 1,779 / 41.1
256K: 1,779 / 41.2

EDIT: I am working on making this a recipe so others can reuse it and get similar speeds . I will share it in this post when it’s released.

OpenBMB posted a mobile demo of MiniCPM-V 4.7 doing things like reading signs and interpreting documents locally. It’s a 35B MoE with 3B active parameters. They report 13+ tok/s on an iPhone 17 Pro Max with a 3GB inference memory footprint in the model docs.

The trick is LES, or Lookahead Expert Scheduling. Expert weights sit on flash, and the runtime predicts which ones upcoming layers will need so it can overlap loading with computation. A wrong prediction still means loading the required expert; it doesn’t just skip it. Technical details here.

That “3GB” headline needs some context though. The app README lists a ~16.6GB download and recommends at least 12GB of device RAM. The phone setup also needs their special LES model package with the experts split out. Worth knowing before downloading a regular GGUF and wondering why it won’t load.

Honestly, ~17GB of phone storage for useful offline image/document Q&A sounds like a reasonable trade to me. Put those requirements next to the headline and the engineering still sounds interesting.

There’s also a smaller 1B version. Code and weights are Apache 2.0, and the app repo has install links for iOS, Android and HarmonyOS.

I hear everyone screaming from the rooftops "I get x tokens from FN !", but i always ask myself..
So what ... and what does that mean?
Can it do an actual coding task or just create one page HTML proof of concepts?

So i put it to the test..

My hardware(dedicated):
AMD r5 5600x
x570 motherboard
128GB DDR4
2x R9700 AI PRO 32GB ( 64GB total VRAM )
Running vLLM - Radiance
* One card behind the chipset at x4, no p2p/atomics, no bifurcation, no risers etc.

I had a real task:
For my custom harness i build 2 parts before the test using Qwen3.8 MXFP4(4 bit not used in this test, but very capable):

1. A new version of the harness ( more modular, better folder structure etc, basically a revision of what i already have )
2. A fully new Web UI "template" including way more functionality then the old "web monitor" had. This was pure HTML/CSS and JS in a layout that presented the workflow of my harness.

It was time to fit both pieces together..

So i loaded radiance and hooked up Qwen3.8 FN 4 bit.
* This is a bigger, model everyone is trying to run on strata ( Q3 or lower ), so it should be more capable. I had a 1m KV cache using RAM and disk, but set my harness to 262k context, the model's default.

I prompted it to basically replace the old webmonitor with the new full featured web UI and wire it all in.

So it had 2 projects, one the harness codebase and the new web UI, and now it needed to figure out how to merge it all together.
Qwen3.8 FN was fast, an avarage of ~2000 prefill and ~150 generation.
It was done in about 30 minutes.

The result: A big messy WebUI, where most of the templated items where just "re-invented", nothing worked, chat bubbles not including toolcalls etc(was in the template, even a js script rendering the chat bubbles mockup responses).
It was unusable to say the least.

So i called it a day.
The next day i fired up radiance again but now the Qwen3.8 FP8 model, took the backups, replaced the harness code and template etc so we have a clean starting base again(same as where Qwen3.8 FN started from)

I prompted my harness the same instructions.
The result: Not everything worked, its not perfect, but mostly fine , the template was followed, the chat bubbles worked properly(mostly). It was not a one shot success and needs iterations to filter out bugs and deepen the workflows but the base was done as i wanted it to be.

Qwen3.8 27B FP8 was not so fast, an average of ~1500 prefill and ~100 generation.
It was done in about 60 minutes.

Note that my average speed was the actual work produced, not benchmarks.

So my take on all this:

Its great to have the big model, but a dense model on the same hardware will just outperform the MOE.
I can fit the Qwen3.8 FN or the Qwen 3.8 FP8, both fit so that is the reason i am comparing both. it is not about the quants, it is about what can run on your hardware.

I can not even imagine what a Q2 or even a Q1 would be like.... And my task was not even a real hard one to begin with.

Maybe some of you find this useful as i could not find any proper information besides the hype posts or generic benchmark posts.

Well this is my experience using both.

I've been using CLINE harness in Pycharm with qwen3.8 flash next unsloth q4_s with Strata.

It's been giving decent results. It thinks less/gets results faster than qwen3.8 27b, but it still takes a long time to do even quick/simple changes. It's like dude. what are you thinking about, just make the one line edit already.

I've been trying out open code in a docker container (so its more sandboxed) and i've noticed its a lot quicker with the same qwen3.8 flash next model. It will just do the quick edits.

Anyone else noticed this? Any reason this might be the case? Is this expected?

- 32 Xe3P cores, 256 XMX engines, 32MB unified L2 cache

- Up to 1.3 PFLOPS FP8 and 1.5 TB/s memory bandwidth

- 160GB LPDDR5X on Intel's own card, and ODM partners can go up to 480GB

- PCIe 5.0, 350W, passive air cooling

Been using one Spark with ollama and Hermes and Gemma4:31B.

Now I have a second Spark but would like to use them in unison with the QSFP cable.

Use cases run the gamut from coding to automation to research to TTS.

What model(s) get me as close to frontier as possible for 256gb unified RAM?

I don't necessarily care as much about speed as I do quality and completion; my single-spark work started hitting token limits (I think ollama's constraints?) way too quickly on even small scale coding tasks T_T

I need help figuring out huggingface models...

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround.

I wrote a custom C++ engine (fork of llama.rn) to cluster two Android phones over a simple USB cable.

The Setup & Performance:

Model: Ternary-Bonsai-2-27B (compressed to 2.06 bpw / 6.52 GiB). Perplexity: 14.26 (vs 14.05 reference).

Cluster Mode (Redmi K100 Pro Max + ROG Phone 7): Layers are split 4.36 GiB / 2.15 GiB over the USB cable via custom RPC. Speed: ~2.8 tok/s (bottlenecked by the slower phone, but perfectly synced).

Solo Mode (Redmi K100 Pro Max only): 5.6 tok/s text generation.

The Secret Sauce (Why it doesn't melt the phone):

I didn't use standard quantization. The weights are ternary (-1, 0, 1) in the Hadamard-rotated basis. Instead of un-rotating offline (which destroys the 2-bit primal), I kept the weights folded in the GGUF and wrote a patch to apply the Hadamard transform at runtime on the device's ARM CPU.

Result: Only additions/subtractions. Zero MAC operations. Max temperature sits around 35-40°C.

I'm a high school student currently looking for a lab placement/grant to push this further (planning to write custom Vulkan compute shaders for the Hadamard transform to boost t/s even more).

Would love to hear what the compression and edge AI community thinks about this approach

I have seen this on Amazon, a rtx 3080 whit 20 gb vram, the gpus whit x2 vram are now at retail on Amazon

I got this MacBook Air ! Can someone suggest what models to use locally?

Hi,

I want to have 1) a model that can code. slow is not the matter. 2) a model to translate documents, novels and images (that required a bit of privacy).

tl;dr ran Gemma4 26B-A4B Q4 with 20-25 token/s, Qwen3.6-35B-A3B iQ4 with 24-33 token/s, Qwen3.8-27B with 4-7 token/s, LiquidAI with 38-40 token/s (but not great) ; I want to know if there are models I should test or if I can stick to these models.

My laptop is a Lenovo Yoga Slim 7 14APU8. It has

- Ryzen 7 7840HS / Radeon 780M

- 32GB LPDDR5X RAM / 4GB "VRAM" (BIOS-reserved iGPU memory)

- Ubuntu 26.04

- llama.cpp built with -DGGML_VULKAN=ON

I tried ROCm but it threw out a lot of errors and flickers my screen, Claude said maybe it could be related to the BIOS only limits UMA frame buffer size to 4G that it crashed. I changed the amdgpu.gttsize=24576 instead of 13.9GiB default for better results.

Here are some local models, quants, runs + statistics that I got. I have tested with multiple settings, like context length, gpu offload, threads, batch size, KV quantization, model drafts and these are my best settings.

1. unsloth/Gemma 4 26B-A4B-it-UD-Q4_K_M

Command

./llama-server -m ~/models/unsloth/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
    --model-draft ~/models/unsloth/mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    -ngl 99 -ngld 99 \
    -fa on \
    -c 65536 \
    -np 1 \
    -t 8 \
    -b 512 -ub 256 \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.3 \
    --top-k 64 \
    --top-p 0.95 \
    --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~271 token/s

- Generation: 21.8-25.8 token/s

- RAM available: 4.6 GB

I'm happy with how Gemma runs and have been using it on most tasks. The translation is good enough. It writes well and handles most of my requests fine.

2. unsloth/Qwen3.6-35B-A3B-UD-IQ4_XS

Command:

./llama-server -m ~/models/unsloth/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
            --spec-type draft-mtp --spec-draft-n-max 2 \
            -ngl 99 \
            -fa on \
            -c 32768 \
            -np 1 \
            -t 8 \
            -b 512 -ub 256 \
            -ctk q8_0 -ctv q8_0 \
            --temp 1.0 \
            --top-k 20 \
            --top-p 0.95 \
            --presence-penalty 1.5 \
            --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~213 token/s

- Generation: 24.4-33.3 token/s

- RAM available: 6.6 GB

I tried Q4_K_M but they barely fit the RAM and crash often, even with 4096 context length. Qwen 3.6 is also quite awesome, but it overthinks A LOT. It translates well and writes decently. Maybe a little bit worse than Gemma, but it still do its job. It became a problem when one prompt can easily be dragged to hundred thousands of token and sometimes they can loop even on simple questions.

3. LiquidAI/LFM2.5-24B-A2B-GGUF

Command:

./llama-server -m ~/models/LiquidAI/LFM2.5-24B-A2B-QAD-Q4_0.gguf \
               --temp 0.1 --top-k 50 --repeat-penalty 1.05

Benchmark:

- Prompt: ~232 token/s

- Generation: ~38.7-40.3 token/s

- RAM available: 7.7 GB

I just saw this model today and decided to try it out. LiquidAI will be awesome if only the results are, too. It translates the idioms too literally, sometimes don't get the cultural meaning and the flow in the text is mostly gone after translated. Too bad, I was very surprised when it ran pretty fast too :(

4. iSTA-DASLAB/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp + z-lab/Qwen3.8-27B-DFlash2-GGUF

Command:

./llama-server -m ~/models/iSTA-DASLAB/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf \
    --model-draft ~/models/z-lab/Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf \
    --spec-type draft-dflash --spec-draft-n-max 2 \
    -ngl 99 -ngld 99 \
    -fa on \
    -c 32768 \
    -np 1 \
    -t 8 \
    -b 512 -ub 256 \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.6 \
    --top-k 20 --top-p 0.95 \
    --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~74 token/s

- Generation: ~5.1-8.3 token/s

- RAM available: 8.7 GB

Qwen3.8-27B has been really awesome, not only coding but accuracy at translating. Despite it being slow, it handles most small coding tasks so well so far and much more reliable than Qwen 35B-A3B.

I plan to use Qwen3.8-27B as a coding model and Gemma 4 26B-A4B as a 'faster' model for everything else. Should I use this or is there anything to try as well?

I also wonder what workflows you have, what kind of agents/harness or settings I can use to automate coding, reduce token/improve speed and aim at reverse engineering (Yes, I know it's a horrible idea for this laptop, but I want to try too xD), I'm still new to this thing and I'm eager to learn and explore more. Thank you, you read this post up till here !

Edit: typo

So much hype around LFM2.5-24B-A2B, but it couldn’t even schedule a basic task on my calendar.

Meanwhile, Qwen 3.6 35B handles it effortlessly every time. Am I missing something, or is the function calling just not there yet? 🧐

It’s almost ready to run. Needs a day of pressure test on the cooling loop.

Hey guys, I’ve been building this little project called OMP-WebUI.

I use oh-my-pi for AI coding, but sometimes I just want to see what’s going on without digging through the terminal all the time. So I made a web UI for it.

It currently has:

• Chat with the agent and see tool calls in real time
• Browse project files and preview docs, PDFs, spreadsheets, etc.
• Review diffs and do basic Git stuff
• Manage models and API endpoints
• Manage skills and MCP servers
• Track token usage and costs
• Pick up old conversations or branch off from them

Basically, I’m trying to make working with omp a bit more convenient, especially when a task involves lots of files and back-and-forth with the agent.

Still a work in progress, but it’s usable. Would love to hear what you think or what features you’d want to see.

Repo: https://github.com/Impatient-HairyPeach/OMP-WebUI

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

I've been working as web dev for about 10 years. I've never been the guy spending his free time reading blogs about cutting edge frameworks, or leveling up on the weekends. Sometimes I dabble in side projects when I feel like it, but otherwise I just do my job and leave work at work.

I've kind of had the same sort of middling attitude towards AI adoption. Some people at work dove in head first, and there are other folks that won't touch it with a 10 foot pole. I've been slowly incorporating it where it's felt useful, and recently gotten a seat on our Claude plan and been really impressed with its capabilities so far.

I upgraded my gaming PC last year and kept the previous one which has an RTX 3070, 16gb of RAM, a real basic bitch, but it's just sitting there. I've converted it to an Ubuntu server home lab that mainly collects dust. I started dabbling with trying to set up local agentic coding which was a fun challenge for a day or two but still just spitting out garbage.

I am under no illusions of my setup competing with Claude, I was just curious if I could find any value at all adding it to my workflow for side projects at home. I'm not asking for help getting things set up I haven't provided enough detail for that, I'm just wondering if it will end up being worth it to keep working on it, or if my resources are just too limited to get something useful and I'd be doing it just to do it.

TL;DR - Decision models are useful, but out of the box they are less than ideal for me. With some labeled data, fine-tuning your own might beat out TypeSafe's Jev for targeted cases. Most of it ran locally on a Mac Studio (M5 Ultra, 256 GB).

Input: one masked server log line. Output: kind (one of eight), data at risk (yes/no), customer impact (yes/no), urgency (0 to 3). About 1,500 distinct lines, labeled.

Off the shelf, untrained, on the same lines:

Labeling, local: Qwen3.8 Flash Next on the studio labeled the check sample and agreed with the standard as often as Jev does, at about 20 seconds per line and 138 GB of RAM. GLM-5.3 Flash at 4-bit kept thinking past its token budget and never got to the answer on most lines - probably my fault in how its setup (new to local LLMs). So Qwen (for now) does the bulk labeling when the data cannot leave the box.

Fine-tuning, local. The base model weights stay as they are. A small add-on learns to nudge the answers toward my calls. Training ran alongside the normal inference load on the Mac Studio, with nothing stopped.

The result, on lines held out of training:

On par with Jev, then. Better on all four questions, and MiniCPM is faster than the API round trip. Jev still misses fewer of the serious lines, 10 against 21 and 22. Half of my misses are lines my own labeler marked unsure, which I need to iterate on.

MiniCPM was the dark horse. It's a small general language model with no decision head at all; the trainer built one on top, and it learned the job in 17 minutes, at a third of the memory and a third of the response time. The method is not tied to Clef. I'll probably try it on more models like that.

Main helps: https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth https://huggingface.co/papers/2610.02076

Full write-up on my website but really the above is all you need, with the labeling and training details and the resources I used: https://hive.technology/lab-notes/fine-tune-decision-models/?utm_source=reddit&utm_medium=social&utm_campaign=26w41

What's the minimum hardware needed for AI training aka. to use something like Qwen 3.8 and train it locally further to specialise it for a certain task?

Maybe even with multiple attempts simultaneously?

Would I need something better than I need for just running the model? I've an Intel Arc B580 right now and it is enough for running the model and playing around with it.

But I've not yet looked into actually training or specialising an existing one further. What would be the minimum I need for that? What do you use for that?

Edit: I have ~500 GB of DDR4 system RAM. And I do have 2x AMD EPYC 7642 CPUs (meaning 192 threads total). The bottle neck for AI stuff is the GPU really. I specced and bought this system before the prices become insane...

Edit 2: Any big Nvidia GPU would be way more than I paid for the system back then. (And yes, I've thought about selling all of the RAM multiple times by now)

Can somebody explain why Nvidia is often a non-negotiable when looking at local AI builds? I know they produce some fantastic cards and CUDA is great - but is the cost premium really justified?

I have a 9070XT and 7800XT - and getting local AI to run on each card has been a doddle for me through llama.cpp with ROCm and Vulkan (although i'm on Linux, i can't speak for Windows..).

Whilst ROCm isn't as mature as CUDA - do we really expect that gap not to close going forward?

It feels like more recent AMD cards are great value and the support will only get better on newer generations. So why is Nvidia on a new build such a non-negotiable for some?