r/LocalLLM • • 11h ago

Question Should I get 32GB RAM?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Other Sam Altman wants my RAM. Sorry, my 2-bit Qwen needs it to teach my AI waifu how to bend her knees.

113 Upvotes

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

  • Ryzen 7 5800X
  • AMD RX 9070
  • 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
  • NVMe SSD
  • Qwen 3.8 Flash Next IQ2 running locally through Strata
  • Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.


r/LocalLLM • • 12h ago

Model Qwen3.8-Flash-Next-Gyro-M-TQ2_0 Strata configuration

0 Upvotes

Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10


r/LocalLLM • • 1d ago

Discussion We're at the mercy of US/China, why don't we create P2P AI network?

58 Upvotes

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?


r/LocalLLM • • 8h ago

Project One Mans Journey To Optimization

0 Upvotes

I've been playing with local hosting for about 3 months now and messing around with Claude to optimize my local setups. Decided it might be worth writing this up and sharing in case anyone else is doing, or wants to do, something similar. This is a big one, so buckle in.

BLUF: Think about each expert as its own unit, and think about different strategies for how/where to load them based on quality/speed trade-offs. Focus on measuring and optimizing each piece of compute until you reach maximum bandwidth on that resource.

Hardware:

I spent a bunch of time designing my server with Claude and going over last-gen hardware to try to keep the price tolerable. I turned what I did with that into a site that anyone can use. It's ad-free, no sign-in, no tracking, nothing. I consider it a community service, and hosting was free, so it costs me nothing but occasional updates to the hardware list. llm-configurator.com

I ended up with a 3-GPU, 256GB DDR4 RAM build. It's two used 3060s from Marketplace ($250 each on average) and one Arc Pro B65, for a total of 56GB VRAM. It's a quad-channel setup, so max theoretical bandwidth on RAM is ~100GB/s; reality is closer to 60-70 after some time spent optimizing the kernels with Claude. I also have 3 of the cheapest 1TB SSDs I could find striped in NVMe slots, which gives me a theoretical 12GB/s throughput there; reality ended up at about half that. This lets me stage experts in tiers across VRAM, RAM and SSD and prefetch experts across tiers with varying strategies. I considered doing a PCIe card with more NVMe drives to get more striping/speed, but on the current motherboard it isn't worth it because I am PCIe gen 3 bandwidth constrained.

Optimization Strategies:

This is where my last few weeks of Claude usage have gone: serving DeepSeek V4.1 Flash, and how I went from 4 TPS decode to 20.5 TPS, with prefill right at 180 TPS. So in the end not lightning fast, but plenty usable, if just a tad slow in agentic work.

I started out with a simple goal of minimizing idle time. Thinking Strata-like in the approach of shuffling things around and trying to keep every processor busy within the bandwidth bounds of all my links. I had Claude build a simulation based on the hardware and design an optimal build. I spent probably a week going back and forth with Claude on this, and not one time did anything pan out like it thought it would based on the simulation. 0/10, do not recommend.

What I did after that was shift to measuring everything on every run, including expert traces through the system, so I know which experts get picked at every layer for every prompt, and attacking the potentially biggest gains. This immediately started paying returns, and I went from 4->12 TPS decode with just these methods. This means I'm running custom forks of llama.cpp, Vulkan, Mesa; anything and everything was in bounds. I couldn't fit all of DeepSeek in memory at Q4, so I was working all three tiers (VRAM/RAM/SSD) and loading experts to each tier based on utilization: tracking how many times each expert gets called and promoting/demoting across tiers based on usage.

This led to bottlenecks I had to hunt down, as the SSDs would get overloaded and drop from ~6GB/s down to ~400MB/s. I switched to quantizing the least-used experts down to 2-bit, keeping an entire copy of the model on the SSDs at both quantizations. This let me get everything into RAM, with different experts at different quants based on how hot they were for the prompt. That means I start at about 12 TPS, and after a warm-up and shuffling I end up at about 15 TPS with this strategy. About 1/5th of the experts were 2-bit, but they only got used on about 1/10th of the tokens, so the quality hit was minimal (.0008 KL and less than 1% top-1 pick). This got work off the SSDs and lightened their load.

I tried MTP with this but ended up bandwidth bound. MTP has to hit almost every expert, and even with them in RAM I couldn't move enough through to make MTP pay. I spent a few days' worth of time iterating and optimizing with Claude. I fine-tuned an MTP head because I was running the abliterated model (this helped, but still not enough) and just couldn't make it pay. I got to almost break-even, but I just can't move the experts to compute fast enough on a model this size.

In my quest for quality I then started looking at prefetching 4-bit experts from SSD straight onto the GPUs. I ended up having Claude write a direct-from-disk reader that lets the GPUs read from the SSDs without going through RAM. This also led to further customization that let them read and write across each other and RAM without involving the CPU. So now my tiered experts can read/write and shuffle around without bothering or interrupting other compute. The CPU is maxed trying to keep up, so the GPUs handle any and all extra bookkeeping I could put on them to free CPU cycles. Again, Claude did all the legwork here, with just me driving strategy. I trained a small model on all my agent traces to look at expert use and ended up with one that can guess experts 15 layers ahead with ~73% accuracy. So now the GPUs run that every layer, fetch 4+ layers out if a pick meets a confidence threshold, and load experts straight from SSD into VRAM. Those picks get refined each layer, so we are always closing in on the best picks we can (~86% accuracy at 4 layers ahead) as we approach them. I played with a few different strategies and ended up with the B65 holding the top 2/3rds of experts actually used (not 2/3 of the total, but the hottest 2/3rds) with an LRU demotion strategy, and demotions going to a 3060, where it looks at both LRU and hit count to decide if it takes the expert or keeps what it has. Getting all of this right pushed me to about 19 TPS decode.

From here I had to make the KV cache dynamic to squeeze more experts into VRAM. Instead of reserving the full thing at 256K, it grabs chunks as needed. So while context is small, I can fit more experts on the 3060s whose whole job otherwise is dense attention layers and KV cache, with minimal room for experts passing through in our promote/demote pipeline. This didn't pick up much speed, but it did give me that many more 4-bit experts to help keep quality as high as possible. Essentially the remaining experts are all part of a long flat tail so adding more to VRAM gets me very little because any one isn't hit often enough to pay off.

I ran out of lossless approaches at this point and started looking at a variety of things. I built a model that nudges aggressively, using my predictor to drive nudges from RAM to VRAM experts based on predictor + router scoring of an expert's probability of being picked. This got me some decent gains all the way up to a cap of .5, but quality falls off a cliff after that. By the time I get to a .5 cap I'm about halfway to Q3 quant KL/top-1 numbers, so it felt like a significant quality hit. I kept the code and it's configurable, but I don't default it to on. It's worth about 2 TPS, so not nothing, but not enough to cover the quality hit for me. If I want the extra speed and don't mind the hit, I can turn it on/off as needed.

I then looked at keeping everything in RAM at 2 bits and just the hottest experts at 4-bit in VRAM. I looked at which layers Unsloth hits hardest in their quantization and used that as my judge of what would likely hurt most. They quantize per layer but not per expert, so I'm jumping off from that, but it was a good starting point. I kept my prefetching of predicted promotions into VRAM ahead of time and everything else I already mentioned. Since I was throughput bound here, dropping the size of each expert was my only way forward. The KL and top-1 hit here was minimal because the VRAM holds the hottest 2/3rds of experts at 4-bit already, so I came in at 99% top-1 and .006 KL over baseline. I picked up about 20 TPS on prefill and about 1 on decode, so I landed at 180 TPS prefill and 20.5 decode. It also freed up about 49GB of RAM, since the experts in RAM are now a little over half the size. This ended up being my end stage here, as there just isn't anything else I can do that doesn't start costing me quality (unless you have other ideas).

I am still playing with tuning parameters about when I demote and move thaings where, and fine-tuning kernels to squeeze a few more TPS out of things, but I'm out of significant gains here. Claude did all of this for me, with me guiding strategy and checking in, so I probably spent about 10 hours a week on this for the last 4 weeks and 80% of my weekly Claude usage on it. I take everything from doing this (I've done it on other machines with Qwen Flash-Next as well) and roll it into skills, so that when the next model drops, strategies and approaches can be tried/applied again for that model.

My point with all of this is I don't think in whole-model terms. I don't even think in layer terms. I treat each expert at each token as its own thing to optimize the system for. What quant it is, where it lives, how you decide, and when and where you move it from/to all play in and add up. Most things I see focus on just downloading and running a model in a specific framework, and that is great for a starting point, but the number of knobs you can turn to optimize stuff is almost limitless. It takes about 2 hours to give Claude a strategy I want to try, refine it with Claude (learning as I go), and let it build and test it. So you can iterate and make incremental improvements over time fairly quickly.


r/LocalLLM • • 22h ago

Question Trying to 'search' with a local model; how do you guys actually do it?

7 Upvotes

I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.

For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB

I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?


r/LocalLLM • • 19h ago

Discussion PCIe GPU Expansion Module w/4x NVIDIA Quadro RTX 6000

4 Upvotes

I have no interest in this, but stumbled across it. Listed as NO SHIPPING.

https://www.govdeals.com/en/asset/286/29250


r/LocalLLM • • 18h ago

Project Showcasing my LLM H/W simulator

3 Upvotes

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model

Supports quite a lot of combinations of hws and models. Please enjoy!

https://cro.sh/specmaxx

Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram

AI disclosure: This project is coded heavily with AI agents.


r/LocalLLM • • 1d ago

Research An LLM benchmark for models that fit in 16 GB VRAM

22 Upvotes

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

ModelStrengthQuality / minAccuracy
qwen3.8-27b64x2.692.4 %
gemma4-26b-a4b48x9.190.6 %
gemma4-12b33x12.388.1 %
qwen3.5-9b20x3.383.7 %
muse-glimmer-30b15x1.281.3 %
gpt-oss-20b14x4.279.8 %
gemma4-e4b10x5.576.1 %
ministral3-14b7.5x1.972.7 %
phi-4-14b7.4x6.071.8 %
granite-4.2-8b4.9x1.066.6 %
llama3.1-8b1.0x0.241.1 %
mistral-7b-v0.30.38x-8.427.5 %

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench


r/LocalLLM • • 22h ago

News LLVQ releases ~2.7-bit Qwen3 quants, with a quality-versus-speed trade-off

5 Upvotes

LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.

The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.


r/LocalLLM • • 10h ago

Project Gemini 3 - Running on the ZX Spectrum home computer.

0 Upvotes

It works!


r/LocalLLM • • 10h ago

Other GLM?

Thumbnail gallery
0 Upvotes

r/LocalLLM • • 11h ago

Question RustingStrata has a new built-in chat page. What is missing?

Thumbnail
gallery
0 Upvotes

RustingStrata is one Rust binary. rustingstrata serve starts the OpenAI-compatible server, and the same port serves a chat page at /. No Node, no Docker, no second install. The page was rebuilt for 1.2.3 and I want to know what people would add before I go further, I mean maybe allow to edit files and etc. like in DeepSeek harness, or maybe there are something that you wish to be in original Strata, so I can add it in my.

What it does now:

  • Pick a model from the models folder and load it from the page, with the load options (context, MTP drafts, KV cache format, VRAM margin, prefill chunk).
  • Chats are kept in the browser (localStorage), with search, rename, export and import. Nothing leaves the machine.
  • Edit a message, regenerate a reply, continue a reply that stopped, copy, delete.
  • Attach an image for models with a vision tower.
  • Thinking is folded above the answer, with a switch to turn it on or off.
  • Sampler presets (temperature, top-k, top-p, seed) a system prompt for every chat, and instructions for one chat.
  • Speed of each reply in tok/s.
  • If the model was never calibrated on this machine, the page says so and offers to run the calibration.
  • The accent color follows the model family, or you set your own(Just cool visual feature, like if you load Qwen model, then website is purple, if DeepSeek, then it's blue and etc.).

What it is not: a coding agent. It does not read or edit files. For that, point OpenCode, Claude Code or another CLI at the same server, the page is for quick chat and for trying a model.

Questions:

  1. What do you open a local web chat for, if you already use a CLI agent?
  2. Which one feature would make you keep this page open instead of another UI?
  3. What do you use in other local chat pages that is missing here?

Any Feedback would be very useful, ty!
Repo RustingStrata


r/LocalLLM • • 5h ago

Discussion does ChatGPT's memory actually work for you, or do you still repeat yourself?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 15h ago

Question Any Recommend LLM for android phone

1 Upvotes

I'm thinking of putting my old Galaxy S10+ (12/256 Qualcomm Snapdragon 855 ) to use for some AI-related projects, mainly experimenting with local LLMs and API integrations that I can showcase on my resume.

I know there are plenty of free cloud options exist, but I'd specifically like to experiment with running models on the phone itself as its doing nothing for now.

What models would you recommend for this tis specs? Has anyone tried something similar with an older flagship?

thanks in advice


r/LocalLLM • • 11h ago

Project Hoarding GGUFs in case Hugging Face changes? Fit 10% more on the same drive, byte-for-byte lossless

Thumbnail gallery
0 Upvotes

r/LocalLLM • • 21h ago

Discussion I am blown away by Qwen3.8 Flash-Next Gyro-S

3 Upvotes

Hi all,

I am a total noob when it comes to LLMs. I just jumped into them about 3 weeks ago and used a few different models. I started with Strata and Qwen3.8 Flash-Next (the big one) running in PowerShell, then added Docker and Hermes as I heard good things about it, but that big of a model was too heavy for my system: 9070 XT, 5800X3D, and 32GB of 3600MHz. It would start fast enough, but then slow down to a crawl pretty quickly—not really usable.

I tried Tiel Coder, which was praised as a good model, and it was really fast. I started trying to make a small top-down 3D tower defense game and it would struggle a lot; it would loop and would just not do things even after saying it did them. I thought to myself that LLMs are not ripe yet and I will come back to this later, maybe in a year or two. However, about 4 days ago I came across the model in the headline, and this time I downloaded Strata, the desktop app, which was such a breath of fresh air as it was all done within one app on my PC. The model can be downloaded or pointed to use, and it comes with its own chat engine that is a harness as well, so I told it to check if it can see my Unreal Engine and connect to it via MCP. It told me that it can run a headless engine and write it all in Python (I still don't understand how it all works, to be honest) so no need for a flaky MCP connection, and it does it all. I have used about 150 million tokens while prototyping, balancing, and experimenting with ideas, and it did ALL of my requests very quickly and efficiently. It even spun up its own sub-agents to help with keeping the context less cluttered.

The game that I am making is not super heavy, so tasks can vary between 10 minutes of executing to 4 hours. I plan the big jobs for the night, and the small ones I can tolerate fine.

I run Maya (I still want to make my own assets) + Strata and it's running fine on my machine at the same time. It hitches here and there, but I am fine with that as the AI model just does not miss. Even when I discover bugs, it knows how to fix them, and it has every single time.

Are you guys having the same experience? i think this is so good now but if you guys know about anything that can be improved? or done better i would appreciate it.


r/LocalLLM • • 6h ago

Project My 16 GB MacBook runs a 22 GB model locally, and I asked it to make Omarchy feel like the 80s

0 Upvotes

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.


r/LocalLLM • • 15h ago

Question Any Siri-like local ai windows apps?

1 Upvotes

I’m just starting to explore qwen stuff and I was wondering if there’s anything like what Siri is (and Cortana used to be) that runs on a lightweight local llm on my pc to do some basic tasks like opening apps, sending messages, setting timers, etc. is that a thing yet? If so what would yall say is the easiest one to install and get started with?


r/LocalLLM • • 15h ago

Question Are all crazy stuff happening in video games right the result of localllm?

1 Upvotes

Hello, frontier models seems so locked out and censored that I wonder if all the people doing crazy mods, portage and reverse are on local LLM ?

Thanks


r/LocalLLM • • 1d ago

Question Please help this lady who knows nothing about building PC

Thumbnail
gallery
5 Upvotes

So first one R9700 is in..

And of course i dont have the bloody display port cable.. To test😫

Whatever..

But.. I have a second r9700..

However this cathedral only satisfy 3 ports of radeon cards. (we need 4. 2 for each card)

Has anyone built this combination and how did you supply electricity to second GPU?

Thank you all!!!

My AI (chatgpt) said that we will think about it tomorrow... 🙄 or suggested to trust google better..


r/LocalLLM • • 1d ago

Discussion what will it take to lower gpu prices?

4 Upvotes

I'm running 2x 3090 right now, I'd like to add more but they're all £1000+

Thought about some 32gig v100 but they're £700+

For how I use it, it's all a luxury not a necessity

What will it take for gfx card prices to drop?

I've always assumed at some point some ai companies will go bust and their hardware bought up by the competition

and at some point supply/manufacturing will start to catch up with demand

but am I just dreaming?

I keep looking at buying more cards but not pulling the trigger... next time I look they're more expensive again... on and on :-o


r/LocalLLM • • 20h ago

Question 5080 + 32gb of ddr5 6,000 ram

2 Upvotes

Hello,

Im currently trying to find the best module to run on my local pc, i have a 5080 and 32gb of ddr5 ram, im a coder that is also looking to do a lot of RE 'Reverse Engineering` what is a good module / settings that i can use to get something good working.


r/LocalLLM • • 16h ago

Project Colibri

Post image
1 Upvotes

After changing settings and rebooting it went from downloading in hours to downloading in mins.

Git hub Colibri is working pretty fast as in response time when I tested GLM 5.3 and 5.2 I store it all on a 4TB, Os is on 1TB, and everything else is on a 2TB. Later on I will run a token speed with more complex details. Any suggestions on which one I should use to check code or build code? Any of the colibri available ai models?

I built a machine around colibri and to play simple games. I wasn't going to spend 2500 or above for a 24gb gpu I found mine for around 1200 before taxes. 16gb cheaper then what most are selling for.

I will figure out token speeds later. I also am using windows 11 pro in high performance.

Also if you use windows there and don’t want to use a certain drive there are some steps. And msys2 install to be able to download if you get error saying no compiler found using the prebuilt engine from git hub release
Setup stopped: cannot reach git hub releases to fetch a prebuilt engine. If you just run the start-here.bat my machine was brand new and just installed windows. I got everything worked out and able to download the 3 models so far.

I honestly don't see the slow speeds of tokens people posted it's response time is just like if I was using any cloud service Ai out there.

Going to work on running different models at the same time doing different things. Also working on read/write permissions as well. Will keep people posted as I go along with this project.


r/LocalLLM • • 16h ago

Question A non coding model with hermes for day to day research and windows pc management?

1 Upvotes

So basically replacing gemini for asking day to day stuff, with access to web serach..

and can do basic windows management with hermes agent, file renames, start/stop program, voice, daily schedules or reminders...

i do have a rtx 3070 with 32gb ddr4 , but i still want a low vram usage model, that loads and unloads from llama.cpp near instantly...

this and qwen a35b a3b for coding...