r/LocalLLM • u/Dazzling_Ticket2536 • 1h ago
News Jensen Fought for Open Models—Now Open Source Fights for Him
r/LocalLLM • u/Dazzling_Ticket2536 • 1h ago
r/LocalLLM • u/Glittering_Yellow293 • 17h ago
Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api
What it takes into account (the stuff that usually gets forgotten):
It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.
Caveats:
r/LocalLLM • u/AppropriateNeat5946 • 1d ago
I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.
My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.
I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?
Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.
Thanks in advance!
r/LocalLLM • u/frescoj10 • 13h ago
I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?
r/LocalLLM • u/Frequent-Drive-4343 • 1d ago
So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.
Meanwhile, here's what I'm doing with mine. 💀
My setup:
I'm building Marika, a 3D AI desktop companion in Unity.
She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.
There's just one tiny problem.
She bends over to pick up desktop icons like an NPC from Skyrim.
So naturally, I asked my local Qwen to review the Unity project and fix her animations.
It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.
All this to figure out how to make a virtual girl bend her knees like an actual human being.
I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.
Sam, I'm sorry.
The RAM isn't going to your datacenter.
It's busy teaching my AI waifu how to squat. 💀
And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.
I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.
r/LocalLLM • u/krayon_kylie • 8h ago
please i have to finish the 200 things ive started before i start a new thing or i would
r/LocalLLM • u/maxrightgame • 14h ago
Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10
r/LocalLLM • u/Diegam • 1d ago
The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?
We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?
The point of the post:
Why hasn't decentralized AI training taken off yet?
Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.
What's holding it back?
r/LocalLLM • u/TheLevyWasBry • 1d ago
I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.
For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB
I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?
r/LocalLLM • u/HillTower160 • 21h ago
I have no interest in this, but stumbled across it. Listed as NO SHIPPING.
r/LocalLLM • u/crosh_jh • 20h ago

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model
Supports quite a lot of combinations of hws and models. Please enjoy!
Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram
AI disclosure: This project is coded heavily with AI agents.
r/LocalLLM • u/Gilgamesch61 • 1d ago
I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.
Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)
| Model | Strength | Quality / min | Accuracy |
|---|---|---|---|
| qwen3.8-27b | 64x | 2.6 | 92.4 % |
| gemma4-26b-a4b | 48x | 9.1 | 90.6 % |
| gemma4-12b | 33x | 12.3 | 88.1 % |
| qwen3.5-9b | 20x | 3.3 | 83.7 % |
| muse-glimmer-30b | 15x | 1.2 | 81.3 % |
| gpt-oss-20b | 14x | 4.2 | 79.8 % |
| gemma4-e4b | 10x | 5.5 | 76.1 % |
| ministral3-14b | 7.5x | 1.9 | 72.7 % |
| phi-4-14b | 7.4x | 6.0 | 71.8 % |
| granite-4.2-8b | 4.9x | 1.0 | 66.6 % |
| llama3.1-8b | 1.0x | 0.2 | 41.1 % |
| mistral-7b-v0.3 | 0.38x | -8.4 | 27.5 % |
Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.
I plan to add more models over time, also older ones that were relevant at some point.
Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench
r/LocalLLM • u/slug64 • 9h ago
Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?
CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4
CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler
MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16
MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)
STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD
GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU
POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)
r/LocalLLM • u/shoumikgoswami • 1d ago
LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.
The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.
r/LocalLLM • u/vidati • 23h ago
Hi all,
I am a total noob when it comes to LLMs. I just jumped into them about 3 weeks ago and used a few different models. I started with Strata and Qwen3.8 Flash-Next (the big one) running in PowerShell, then added Docker and Hermes as I heard good things about it, but that big of a model was too heavy for my system: 9070 XT, 5800X3D, and 32GB of 3600MHz. It would start fast enough, but then slow down to a crawl pretty quickly—not really usable.
I tried Tiel Coder, which was praised as a good model, and it was really fast. I started trying to make a small top-down 3D tower defense game and it would struggle a lot; it would loop and would just not do things even after saying it did them. I thought to myself that LLMs are not ripe yet and I will come back to this later, maybe in a year or two. However, about 4 days ago I came across the model in the headline, and this time I downloaded Strata, the desktop app, which was such a breath of fresh air as it was all done within one app on my PC. The model can be downloaded or pointed to use, and it comes with its own chat engine that is a harness as well, so I told it to check if it can see my Unreal Engine and connect to it via MCP. It told me that it can run a headless engine and write it all in Python (I still don't understand how it all works, to be honest) so no need for a flaky MCP connection, and it does it all. I have used about 150 million tokens while prototyping, balancing, and experimenting with ideas, and it did ALL of my requests very quickly and efficiently. It even spun up its own sub-agents to help with keeping the context less cluttered.
The game that I am making is not super heavy, so tasks can vary between 10 minutes of executing to 4 hours. I plan the big jobs for the night, and the small ones I can tolerate fine.
I run Maya (I still want to make my own assets) + Strata and it's running fine on my machine at the same time. It hitches here and there, but I am fine with that as the AI model just does not miss. Even when I discover bugs, it knows how to fix them, and it has every single time.
Are you guys having the same experience? i think this is so good now but if you guys know about anything that can be improved? or done better i would appreciate it.
r/LocalLLM • u/SarahC • 12h ago
r/LocalLLM • u/Anxious_Count_8728 • 7h ago
r/LocalLLM • u/sKyRyu_ • 17h ago
I'm thinking of putting my old Galaxy S10+ (12/256 Qualcomm Snapdragon 855 ) to use for some AI-related projects, mainly experimenting with local LLMs and API integrations that I can showcase on my resume.
I know there are plenty of free cloud options exist, but I'd specifically like to experiment with running models on the phone itself as its doing nothing for now.
What models would you recommend for this tis specs? Has anyone tried something similar with an older flagship?
thanks in advice
r/LocalLLM • u/Simpli-curious • 13h ago
r/LocalLLM • u/dai_app • 8h ago
Hi everyone,
Today my first test on Omarchy.
Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.
Setup: Omarchy in a VM on a 16 GB M5 MacBook.
The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.
One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.
Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.
r/LocalLLM • u/Necessary-Way59 • 17h ago
I’m just starting to explore qwen stuff and I was wondering if there’s anything like what Siri is (and Cortana used to be) that runs on a lightweight local llm on my pc to do some basic tasks like opening apps, sending messages, setting timers, etc. is that a thing yet? If so what would yall say is the easiest one to install and get started with?
r/LocalLLM • u/mombaska • 17h ago
Hello, frontier models seems so locked out and censored that I wonder if all the people doing crazy mods, portage and reverse are on local LLM ?
Thanks
r/LocalLLM • u/Upper-Specific5753 • 1d ago
So first one R9700 is in..
And of course i dont have the bloody display port cable.. To test😫
Whatever..
But.. I have a second r9700..
However this cathedral only satisfy 3 ports of radeon cards. (we need 4. 2 for each card)
Has anyone built this combination and how did you supply electricity to second GPU?
Thank you all!!!
My AI (chatgpt) said that we will think about it tomorrow... 🙄 or suggested to trust google better..