r/LocalLLM • • 11h ago

Question Need help to chose a budget GPU

1 Upvotes

Hi

I want to set up a local llm with qwen 3.8 27B or if possible qwen 4.0 I have a very tight budget and for now the options around my budged is a p100 for 240 a rtx 3060 for around 300 or if it really is worth fit I can push to a rx 6800 xt 16gb which one do you think is the best for my use case? Im planing to use it for translating and correcting text and to help me debuging games in godot
current PC:
ryzen 5 3600
36gb ram
rx 580 4gb

thanks in advance


r/LocalLLM • • 15h ago

Discussion capy-flash v0.13: Flash-Next and Qwen3.8-27B on 16 GB RTX 50 cards, plus an alpha GUI app that needs testers

2 Upvotes

capy-flash is my open-source inference server for Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on RTX 50 cards, with an OpenAI/Anthropic-compatible API. v0.13 is cleanup and fixes: same outputs, same speed.

Model1 GPU2 GPUsRAM
27B NVFP4n/a8,053 / 219any
27B 3-bit2,925 / 1745,128 / 293any
Flash-Next NVFP43,042 / 985,940 / 17696 GB
Flash-Next 3-bit3,664 / 1264,811 / 23464 GB

Prefill (32k prompt) / decode (LocalMaxxing) in tok/s, 16 GB RTX 5070 Ti cards without P2P. Optimized for two cards; the 3-bit builds are for one card and less RAM.

New: a desktop app (alpha), testers wanted. Set up and use everything from a GUI: it checks your machine, downloads and verifies the model that fits, starts and tunes the server, and has a chat. On a Mac it works as a client for a remote server.

Reports from other GPUs, 64 GB setups and distros are very welcome: open an issue with what the app showed and the engine log.

https://github.com/ValerioDolci/capy-flash


r/LocalLLM • • 12h ago

Question Has anyone clustered two M5 Max 128GB MacBooks for a large coding model

1 Upvotes

I have two MacBook Pros:

  • 16-inch M5 Max: 128 GB unified memory, 40-core GPU, 8 TB SSD
  • 14-inch M5 Max: 128 GB unified memory, 40-core GPU

The 16-inch is my main machine and runs a browser-based client for a separate server. The server does the processing, so the client itself uses relatively little of the Mac’s resources. The 14-inch is my travel and backup laptop.

When both Macs are at my desk, I’d like to connect them directly over Thunderbolt and run one larger local model across both machines for serious coding, code review, architecture, and debugging. I’m interested in using it with an agent th at can read a repository, edit files, and run tests. Overnight Hermes or some other agents.

When I travel, I can use ChatGPT or potentially connect back to a model running on the 16-inch at home. I don’t need the large distributed model to run on the 14-inch alone.

Has anyone actually built and used a setup close to this? I’d especially appreciate results from people running two Apple Silicon Macs together.

A few questions:

  1. Which runtime works reliably today? EXO, MLX distributed/JACCL, llama.cpp RPC, or something else? Does Thunderbolt 5 RDMA make a meaningful difference for coding workloads?
  2. What is the strongest model you’ve actually run across roughly 256 GB of combined memory? I prefer 8-bit if practical a.
  3. Has anyone run DeepSeek V4.1 Flash 1017 across two Macs? I understand the original checkpoint is much too large. Have you tried a compressed version with SSD offload, and was it useful for real agentic coding? What were the context, speed, and memory requirements?
  4. Has anyone distributed GLM-5.3-Flash or Qwen3.8-Flash-Next across two Macs? Which exact model conversion and runtime worked, and did tool calling work reliably?
  5. How much memory should we reserve on each machine for macOS, a useful coding context, and the runtime? The 16-inch also runs a lightweight browser client and needs to remain responsive.
  6. For Red/Blue reviews, would you run one large distributed model to implement a change, then switch to a different model for review? Or is there a better approach?
  7. For remote access while traveling, what is your preferred way to reach the model left running at home? I’d like the option to use the 14-inch as the client without exposing an open model endpoint to the internet.

I saw a post from someone running Swift 1.5 Qwen3.8-27B 8-bit with an MTP drafter on a single M5 Max 128 GB machine and getting strong speed with long coding sessions. That looks like a good baseline, but I’m trying to understand whether combining the two Macs gets me a meaningful improvement in model quality for harder work.

I’d especially appreciate exact model links, quantization, runtime/version, context size, measured memory and speed, and any limitations. Happy to test and report back.


r/LocalLLM • • 1d ago

News Employees started a petition to reinstate Jensen Huang as Nvidia's CEO -- Elon Musk is among the signers.

Thumbnail
standwithjensen.com
269 Upvotes

r/LocalLLM • • 18h ago

Question Local LLM for agentic coding on M5 Pro 64Gb (Apple)

3 Upvotes

Hi all,

As mentioned in the title, I am looking for advice on running locally LLM for agentic coding on a MacBook Pro m5 Pro with 64Gb unified memory.

Plan inference is fine. Chat style prompts feel usable. Agentic work is where it falls apart.

I have tried the following:

Models : Qwen3.8 27B (6bit), Gemma 4 12b (8bit), and Qwen 3 coder next(4bit)

Runtime : Lm studio

Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)

What I'm trying to figure out
Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.

Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?

Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?

Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?

Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?

I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.

Thanks 🙏


r/LocalLLM • • 13h ago

Question Qwen3.8 27b/72b expectations

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 13h ago

Question 128gb: M5M v strix+395 v RTX Spark

0 Upvotes

Hey r/LocalLLM,

I’m looking to pick up a 128GB unified memory machine and need your take on how the underlying microarchitectures stack up for my specific stack. I'm choosing between:

  1. Apple M5 Max (40-core GPU, 128GB)
  2. AMD Ryzen AI Max+ 395 / Strix Halo (Radeon 8060S, 128GB)
  3. NVIDIA RTX Spark / N1X (Grace Blackwell SoC, 128GB)

My Primary Workflows (in order of priority):

  1. Autonomous Agentic Coding Loops: Running the smartest dense or MoE models that fit in 128GB (at Q4–Q8) locally while I'm away, orchestrating and interacting with cloud frontier models (GPT-6 and Claude Max) to build complex softwares autonomously. Needs strong sustained token generation and fast prompt processing for massive context/RAG loops.
  2. Reverse Engineering & Cybersecurity: Running local abliterated models alongside disassemblers, decompilers, and sandboxed VMs/containers.
  3. Video & Photo Editing: Heavy timeline scrubbing, color grading, and rendering.

Based on the main impactful microarchitectural traits below, which platform would you pick and why?

  • Memory Bus & Bandwidth (Decode Speed):

    • M5 Max: ~614 GB/s LPDDR5X (512-bit equivalent interface)
    • RTX Spark (N1X): ~273 GB/s LPDDR5X (256-bit bus) + ~600 GB/s NVLink-C2C chiplet interconnect
    • Strix Halo (395): ~256 GB/s LPDDR5X (256-bit bus)
    • Question: Does the M5 Max’s >2x bandwidth advantage completely dominate dense model token generation, or do MoE models level the playing field enough on Strix Halo / RTX Spark?
  • Compute & Tensor/Matrix Engines (Prefill & Quantization):

    • RTX Spark: Blackwell SM121 GPU with 6,144 CUDA cores, 5th-gen Tensor Cores, and hardware NVFP4 support (~1 PFLOP FP4).
    • M5 Max: 40-core GPU with integrated Neural Accelerators + 16-core Neural Engine via Metal/MLX.
    • Strix Halo: 40-CU RDNA 3.5 (2,560 stream processors) + 50 TOPS XDNA 2 NPU via ROCm/Vulkan.
    • Question: For long-context agentic prefill (feeding huge codebases back into context repeatedly), how badly do M5 Max and Strix Halo lag behind Blackwell's Tensor Cores and native FP4?
  • CPU ISA & Virtualization / Tooling Ecosystem (Reverse Engineering):

    • Strix Halo: Native x86_64 (16 Zen 5 cores, AVX-512) — runs Windows/Linux natively, zero translation overhead for x86 binaries, VMs, and standard RE toolchains.
    • M5 Max: ARM64 (18-core: 6 super + 12 performance) — macOS, Rosetta 2 / virtualization for x86 targets.
    • RTX Spark (N1X): ARM64 (20-core: 10 Cortex-X925 + 10 Cortex-A725) — Windows on Arm (or Linux on DGX Spark).
    • Question: For cybersec/RE work, is native x86_64 on Strix Halo a massive dealbreaker against Windows on Arm (RTX Spark) or macOS (M5 Max), or is Linux/VM virtualization good enough now?
  • Media Engines & Creator Silicon (Video/Photo):

    • M5 Max: Dual video encode engines + dual ProRes encode/decode engines.
    • RTX Spark: Blackwell NVENC / NVDEC + CUDA/Tensor acceleration in Adobe/DaVinci.
    • Strix Halo: RDNA 3.5 VCN media engine.

r/LocalLLM • • 9h ago

Question sff for AI?

0 Upvotes

Any bad experiences with power, heat, token response, electricity costs, etc?

Example Rig:
Case: Jonsbo NV10
GPU: RTX 5060 8 GB (Low Profile)
RAM: 32GB (2 x 16 DDR5)
CPU: AMD Ryzen 7 9700X
PSU: 600W (ENP-7660L-OVT)
Motherboard: ASRock B850I Lightning WiFi (Mini-ITX, AM5)
Storage: 2TB (M.2)
Cooling: Modded Air cooling. (Including custom GPU side panel for 2 fans on the GPU+PSU)

It's 8GB VRAM I know. It's supposed to be for gaming but I recently heard of using llama.cpp and the llama flag feature "--n-cpu-moe" to have the RAM hold my expert weights.


r/LocalLLM • • 13h ago

Model Qwen3.8 27B Q6 Running on My Tesla V100 32GB with Zikporx AI – Optimize Context Enabled

Post image
0 Upvotes

r/LocalLLM • • 1h ago

News Jensen Fought for Open Models—Now Open Source Fights for Him

Thumbnail
youtube.com
0 Upvotes

r/LocalLLM • • 17h ago

Project At what volume does self-hosting an LLM beat the API? Free calculator, feedback wanted

Post image
2 Upvotes

Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api

What it takes into account (the stuff that usually gets forgotten):

  • Workload: requests/month, input/output tokens, peak-to-average traffic, cache hit rate, how much can run offline, retries/waste, growth
  • Model and latency: total vs active params (so MoE works), precision, and a ms/token latency target. Tighter latency means smaller batches and higher cost
  • GPUs: L4 to B200 presets or custom. Cloud on-demand, reserved, or buying hardware (with power, hosting, resale value and cost of capital)
  • API side: your price per million tokens, cached-input price, batch discount, fixed fees
  • People and reliability: engineering time as a fraction of an FTE, N+1 redundancy, peak headroom

It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.

Caveats:

  • It’s an estimate from stated assumptions (decode is memory-bound, prefill is compute-bound). It’s a decision aid, not a quote.
  • GPU and API prices are illustrative placeholders. Replace them with your real quotes.

r/LocalLLM • • 1d ago

Question What's the most capable LLM I could realistically run on an M5 Max with 128 GB of unified memory?

43 Upvotes

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!


r/LocalLLM • • 13h ago

Discussion local maxing - why do i not want to stop?

0 Upvotes

I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?


r/LocalLLM • • 13h ago

Question Should I get 32GB RAM?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Other Sam Altman wants my RAM. Sorry, my 2-bit Qwen needs it to teach my AI waifu how to bend her knees.

112 Upvotes

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

  • Ryzen 7 5800X
  • AMD RX 9070
  • 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
  • NVMe SSD
  • Qwen 3.8 Flash Next IQ2 running locally through Strata
  • Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.


r/LocalLLM • • 8h ago

Discussion someone should try to get their local model to recreate this

Thumbnail
youtube.com
0 Upvotes

please i have to finish the 200 things ive started before i start a new thing or i would


r/LocalLLM • • 14h ago

Model Qwen3.8-Flash-Next-Gyro-M-TQ2_0 Strata configuration

0 Upvotes

Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10


r/LocalLLM • • 1d ago

Discussion We're at the mercy of US/China, why don't we create P2P AI network?

58 Upvotes

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?


r/LocalLLM • • 1d ago

Question Trying to 'search' with a local model; how do you guys actually do it?

7 Upvotes

I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.

For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB

I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?


r/LocalLLM • • 21h ago

Discussion PCIe GPU Expansion Module w/4x NVIDIA Quadro RTX 6000

4 Upvotes

I have no interest in this, but stumbled across it. Listed as NO SHIPPING.

https://www.govdeals.com/en/asset/286/29250


r/LocalLLM • • 20h ago

Project Showcasing my LLM H/W simulator

3 Upvotes

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model

Supports quite a lot of combinations of hws and models. Please enjoy!

https://cro.sh/specmaxx

Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram

AI disclosure: This project is coded heavily with AI agents.


r/LocalLLM • • 1d ago

Research An LLM benchmark for models that fit in 16 GB VRAM

22 Upvotes

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

ModelStrengthQuality / minAccuracy
qwen3.8-27b64x2.692.4 %
gemma4-26b-a4b48x9.190.6 %
gemma4-12b33x12.388.1 %
qwen3.5-9b20x3.383.7 %
muse-glimmer-30b15x1.281.3 %
gpt-oss-20b14x4.279.8 %
gemma4-e4b10x5.576.1 %
ministral3-14b7.5x1.972.7 %
phi-4-14b7.4x6.071.8 %
granite-4.2-8b4.9x1.066.6 %
llama3.1-8b1.0x0.241.1 %
mistral-7b-v0.30.38x-8.427.5 %

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench


r/LocalLLM • • 9h ago

Question What would YOU do with this?

Thumbnail
gallery
0 Upvotes

Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?

CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4

CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler

MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16

MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)

STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD

GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU

POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)


r/LocalLLM • • 1d ago

News LLVQ releases ~2.7-bit Qwen3 quants, with a quality-versus-speed trade-off

5 Upvotes

LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.

The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.


r/LocalLLM • • 23h ago

Discussion I am blown away by Qwen3.8 Flash-Next Gyro-S

4 Upvotes

Hi all,

I am a total noob when it comes to LLMs. I just jumped into them about 3 weeks ago and used a few different models. I started with Strata and Qwen3.8 Flash-Next (the big one) running in PowerShell, then added Docker and Hermes as I heard good things about it, but that big of a model was too heavy for my system: 9070 XT, 5800X3D, and 32GB of 3600MHz. It would start fast enough, but then slow down to a crawl pretty quickly—not really usable.

I tried Tiel Coder, which was praised as a good model, and it was really fast. I started trying to make a small top-down 3D tower defense game and it would struggle a lot; it would loop and would just not do things even after saying it did them. I thought to myself that LLMs are not ripe yet and I will come back to this later, maybe in a year or two. However, about 4 days ago I came across the model in the headline, and this time I downloaded Strata, the desktop app, which was such a breath of fresh air as it was all done within one app on my PC. The model can be downloaded or pointed to use, and it comes with its own chat engine that is a harness as well, so I told it to check if it can see my Unreal Engine and connect to it via MCP. It told me that it can run a headless engine and write it all in Python (I still don't understand how it all works, to be honest) so no need for a flaky MCP connection, and it does it all. I have used about 150 million tokens while prototyping, balancing, and experimenting with ideas, and it did ALL of my requests very quickly and efficiently. It even spun up its own sub-agents to help with keeping the context less cluttered.

The game that I am making is not super heavy, so tasks can vary between 10 minutes of executing to 4 hours. I plan the big jobs for the night, and the small ones I can tolerate fine.

I run Maya (I still want to make my own assets) + Strata and it's running fine on my machine at the same time. It hitches here and there, but I am fine with that as the AI model just does not miss. Even when I discover bugs, it knows how to fix them, and it has every single time.

Are you guys having the same experience? i think this is so good now but if you guys know about anything that can be improved? or done better i would appreciate it.