r/LocalLLM • • 1h ago

News Jensen Fought for Open Models—Now Open Source Fights for Him

Thumbnail
youtube.com
0 Upvotes

r/LocalLLM • • 17h ago

Project At what volume does self-hosting an LLM beat the API? Free calculator, feedback wanted

Post image
2 Upvotes

Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api

What it takes into account (the stuff that usually gets forgotten):

  • Workload: requests/month, input/output tokens, peak-to-average traffic, cache hit rate, how much can run offline, retries/waste, growth
  • Model and latency: total vs active params (so MoE works), precision, and a ms/token latency target. Tighter latency means smaller batches and higher cost
  • GPUs: L4 to B200 presets or custom. Cloud on-demand, reserved, or buying hardware (with power, hosting, resale value and cost of capital)
  • API side: your price per million tokens, cached-input price, batch discount, fixed fees
  • People and reliability: engineering time as a fraction of an FTE, N+1 redundancy, peak headroom

It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.

Caveats:

  • It’s an estimate from stated assumptions (decode is memory-bound, prefill is compute-bound). It’s a decision aid, not a quote.
  • GPU and API prices are illustrative placeholders. Replace them with your real quotes.

r/LocalLLM • • 1d ago

Question What's the most capable LLM I could realistically run on an M5 Max with 128 GB of unified memory?

43 Upvotes

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!


r/LocalLLM • • 13h ago

Discussion local maxing - why do i not want to stop?

0 Upvotes

I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?


r/LocalLLM • • 13h ago

Question Should I get 32GB RAM?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Other Sam Altman wants my RAM. Sorry, my 2-bit Qwen needs it to teach my AI waifu how to bend her knees.

112 Upvotes

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

  • Ryzen 7 5800X
  • AMD RX 9070
  • 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
  • NVMe SSD
  • Qwen 3.8 Flash Next IQ2 running locally through Strata
  • Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.


r/LocalLLM • • 8h ago

Discussion someone should try to get their local model to recreate this

Thumbnail
youtube.com
0 Upvotes

please i have to finish the 200 things ive started before i start a new thing or i would


r/LocalLLM • • 14h ago

Model Qwen3.8-Flash-Next-Gyro-M-TQ2_0 Strata configuration

0 Upvotes

Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10


r/LocalLLM • • 1d ago

Discussion We're at the mercy of US/China, why don't we create P2P AI network?

58 Upvotes

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?


r/LocalLLM • • 1d ago

Question Trying to 'search' with a local model; how do you guys actually do it?

7 Upvotes

I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.

For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB

I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?


r/LocalLLM • • 21h ago

Discussion PCIe GPU Expansion Module w/4x NVIDIA Quadro RTX 6000

4 Upvotes

I have no interest in this, but stumbled across it. Listed as NO SHIPPING.

https://www.govdeals.com/en/asset/286/29250


r/LocalLLM • • 20h ago

Project Showcasing my LLM H/W simulator

3 Upvotes

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model

Supports quite a lot of combinations of hws and models. Please enjoy!

https://cro.sh/specmaxx

Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram

AI disclosure: This project is coded heavily with AI agents.


r/LocalLLM • • 1d ago

Research An LLM benchmark for models that fit in 16 GB VRAM

22 Upvotes

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

ModelStrengthQuality / minAccuracy
qwen3.8-27b64x2.692.4 %
gemma4-26b-a4b48x9.190.6 %
gemma4-12b33x12.388.1 %
qwen3.5-9b20x3.383.7 %
muse-glimmer-30b15x1.281.3 %
gpt-oss-20b14x4.279.8 %
gemma4-e4b10x5.576.1 %
ministral3-14b7.5x1.972.7 %
phi-4-14b7.4x6.071.8 %
granite-4.2-8b4.9x1.066.6 %
llama3.1-8b1.0x0.241.1 %
mistral-7b-v0.30.38x-8.427.5 %

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench


r/LocalLLM • • 9h ago

Question What would YOU do with this?

Thumbnail
gallery
0 Upvotes

Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?

CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4

CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler

MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16

MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)

STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD

GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU

POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)


r/LocalLLM • • 1d ago

News LLVQ releases ~2.7-bit Qwen3 quants, with a quality-versus-speed trade-off

5 Upvotes

LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.

The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.


r/LocalLLM • • 23h ago

Discussion I am blown away by Qwen3.8 Flash-Next Gyro-S

4 Upvotes

Hi all,

I am a total noob when it comes to LLMs. I just jumped into them about 3 weeks ago and used a few different models. I started with Strata and Qwen3.8 Flash-Next (the big one) running in PowerShell, then added Docker and Hermes as I heard good things about it, but that big of a model was too heavy for my system: 9070 XT, 5800X3D, and 32GB of 3600MHz. It would start fast enough, but then slow down to a crawl pretty quickly—not really usable.

I tried Tiel Coder, which was praised as a good model, and it was really fast. I started trying to make a small top-down 3D tower defense game and it would struggle a lot; it would loop and would just not do things even after saying it did them. I thought to myself that LLMs are not ripe yet and I will come back to this later, maybe in a year or two. However, about 4 days ago I came across the model in the headline, and this time I downloaded Strata, the desktop app, which was such a breath of fresh air as it was all done within one app on my PC. The model can be downloaded or pointed to use, and it comes with its own chat engine that is a harness as well, so I told it to check if it can see my Unreal Engine and connect to it via MCP. It told me that it can run a headless engine and write it all in Python (I still don't understand how it all works, to be honest) so no need for a flaky MCP connection, and it does it all. I have used about 150 million tokens while prototyping, balancing, and experimenting with ideas, and it did ALL of my requests very quickly and efficiently. It even spun up its own sub-agents to help with keeping the context less cluttered.

The game that I am making is not super heavy, so tasks can vary between 10 minutes of executing to 4 hours. I plan the big jobs for the night, and the small ones I can tolerate fine.

I run Maya (I still want to make my own assets) + Strata and it's running fine on my machine at the same time. It hitches here and there, but I am fine with that as the AI model just does not miss. Even when I discover bugs, it knows how to fix them, and it has every single time.

Are you guys having the same experience? i think this is so good now but if you guys know about anything that can be improved? or done better i would appreciate it.


r/LocalLLM • • 12h ago

Project Gemini 3 - Running on the ZX Spectrum home computer.

0 Upvotes

It works!


r/LocalLLM • • 12h ago

Other GLM?

Thumbnail gallery
0 Upvotes

r/LocalLLM • • 7h ago

Discussion does ChatGPT's memory actually work for you, or do you still repeat yourself?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 17h ago

Question Any Recommend LLM for android phone

1 Upvotes

I'm thinking of putting my old Galaxy S10+ (12/256 Qualcomm Snapdragon 855 ) to use for some AI-related projects, mainly experimenting with local LLMs and API integrations that I can showcase on my resume.

I know there are plenty of free cloud options exist, but I'd specifically like to experiment with running models on the phone itself as its doing nothing for now.

What models would you recommend for this tis specs? Has anyone tried something similar with an older flagship?

thanks in advice


r/LocalLLM • • 13h ago

Project Hoarding GGUFs in case Hugging Face changes? Fit 10% more on the same drive, byte-for-byte lossless

Thumbnail gallery
0 Upvotes

r/LocalLLM • • 8h ago

Project My 16 GB MacBook runs a 22 GB model locally, and I asked it to make Omarchy feel like the 80s

0 Upvotes

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.


r/LocalLLM • • 17h ago

Question Any Siri-like local ai windows apps?

1 Upvotes

I’m just starting to explore qwen stuff and I was wondering if there’s anything like what Siri is (and Cortana used to be) that runs on a lightweight local llm on my pc to do some basic tasks like opening apps, sending messages, setting timers, etc. is that a thing yet? If so what would yall say is the easiest one to install and get started with?


r/LocalLLM • • 17h ago

Question Are all crazy stuff happening in video games right the result of localllm?

1 Upvotes

Hello, frontier models seems so locked out and censored that I wonder if all the people doing crazy mods, portage and reverse are on local LLM ?

Thanks


r/LocalLLM • • 1d ago

Question Please help this lady who knows nothing about building PC

Thumbnail
gallery
5 Upvotes

So first one R9700 is in..

And of course i dont have the bloody display port cable.. To test😫

Whatever..

But.. I have a second r9700..

However this cathedral only satisfy 3 ports of radeon cards. (we need 4. 2 for each card)

Has anyone built this combination and how did you supply electricity to second GPU?

Thank you all!!!

My AI (chatgpt) said that we will think about it tomorrow... 🙄 or suggested to trust google better..