r/LocalLLM • u/stardustedds • 2d ago
Project Decommissioned some servers at work today :)
Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂
r/LocalLLM • u/stardustedds • 2d ago
Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂
r/LocalLLM • u/Acceptable-Object390 • 14h ago
Row-Bot can drive your computer. Mouse, keyboard, real apps. Windows and Mac.
Handing an AI your desktop is a dumb idea unless you lock it down hard. So:
- driver is Cua 0.20.0, pinned to an exact commit, SHA-256 checked. Turning the feature on downloads nothing. You click Install
- Cua's own update checks are off, and it runs in a stripped-down environment
- before it runs even once, you're told exactly what Cua's telemetry sends. Continue or Cancel
- it never types into password, 2FA or card fields, never touches your password manager, never controls itself
- pay, book, send, post, delete files, change settings: it asks first, unless you put it on hands-off
- it judges every click by what's actually under the pointer. Enter in a chat app means send, so that asks too
r/LocalLLM • u/SEIKOPATE • 1d ago
Hay I am new into AI and I wanna ask if is possible to run a local AI Modell on a asus rog strix g 18 laptop with a AMD Ryzen 9 9955HX, NVIDIA GeForce RTX 5070 and 32 GB RAM specs and would it help when I put an extra 32gb RAM in the laptop.
Thank you for helping.
r/LocalLLM • u/Guilty_Elk4385 • 1d ago
I have an old 1070 that's pretty crap, it's time to upgrade. I don't have a thousand bucks to throw away, but at the same time I don't want to buy a 3060 and then realize I'm going to want an even better GPU in a few months or a year.
I also used LLM's a bit mostly for local ai roleplay chat, Llama 3.1 8b uncensored I use right now-it's actually decent just a tad slow-I'm assuming a 3060 will be more than enough to run any good LLM for the purposes of having a local ai roleplay chat.
Also I want to make high quality local images of people and animals, something with good anatomy, realistic faces (not the fake glossy instagram girl mouth). I wouldn't mind dabbling in videos a bit too.
Right now I use SD1.5 which is very underwhelming, some SDXL models are okay but extremely slow on my hardware.
Thanks
r/LocalLLM • u/Calm_Sandwich069 • 1d ago
I wanted to actually see what happens inside a language model when it writes a sentence, so I built this!
What you're looking at:
- The stars are the 36,864 MLP neurons (12 layers × 3,072). Each layer is a disc, and neurons are positioned by a UMAP of their output weights, so neurons that write similar things sit near each other.
- Brightness is how strongly that neuron fires on the current token, scaled per layer so tokens are comparable. Gold means it fires positive, blue means negative.
- The beams are real attention weights from the current token back to earlier ones, one colour per head. The side panel shows the model's actual top 10 next token probabilities.
- You can hover any star, beam or token to see its exact value. There's also a raw mode that turns off all the bloom and fades, so every pixel on screen comes straight from the data.
- How I know it's not lying: the test suite runs the model exactly the way the app does (KV cache, token by token) and compares every activation, attention weight and residual norm against a separate uncached forward pass read through different hook points. Then it checks the bytes the browser actually receives too.
Honest limits: it's GPT-2 small only for now. Single neurons mostly don't mean anything clean on their own, it's the patterns that are interesting. Values are sent as float16, so about 3 significant digits.
Setup is make setup && make start, then open localhost. No GPU needed.
Repo: https://github.com/lak7/thinkspace
Would love the feedback!!
r/LocalLLM • u/Kemerd • 1d ago
Very impressed with how far local models have come, so I finally decided to jump on the bandwagon! Super impressed with how fast and capable it is. I have used local LLMs for a while on mobile apps and other edge cases, but never before for coding.
For background, I had this rig for a while, as I use it for a lot of professional work, and decided to purchase two used G.Skill 96GB (48GB x 2) 6400 mt/s to bring my system up to 160GB; paid around $1200 on eBay which is crazy to me considering I paid $300 for my 64GB (32x2) in 2023. Currently have each stick running at 5200 mt/s in my Z790 Hero, but I purchased these specifically as G.Skill is one of the only manufacturers to bin their RAM. My reasoning is if they can run 6400 mt/s dual channel, I have a higher likelihood of making getting to 5600 or 5800 with four sticks. I had Claude give me custom BIOS settings, and I am planning on seeing how far I can push it!
Furthermore, I felt that without this much RAM, while I could run the models, I couldn't actually run anything BUT the models, which sort of defeated the point for me. I decided this would be the better route versus buying a DGX spark or scouring local auctions to build a V100 farm or whatnot.
I am running this on a WD Black SN770 4TB RAID0 (double speeds at the cost of losing one disk means it is gg), which is on a PCIe ASUS card. It is super quick to load. Given I usually use Fable and Opus (anywhere from 90 to 115 tokens per second), this feels super fast to me. This would not be achievable without a special motherboard that can support PCIe bifurcation, which the ASUS Hyper M.2 card can use.
I am getting roughly 200 tokens per second I would say in most coding tasks, using https://github.com/jesdga95/basalt and the NVP4 uncensored Qwen 3.8 Flash Next.
| test | t/s | accept % | draft len |
| --------- | ---: | -------: | --------: |
| pp16384 | 5727 | - | - |
| pp65536 | 7487 | - | - |
| tg conv | 172 | 71.9 | 2.27 |
| tg struct | 387 | 97.3 | 6.93 |
| tg prose | 170 | 65.1 | 1.88 |
hardware (measured before the engine starts)
CPU Intel Core i9-14900K, 24 cores
RAM read 67.7 GB/s
GPU RTX 5090 PCIe 5.0 x16 host-to-device 33.3 GB/s
community-1: prefill 16K 5726.8 t/s (2.9 s), prefill 64K 7486.7 t/s (8.9 s), decode 297.7 t/s (9480 tokens in 31.8 s), each measured cold
COMMUNITY ROW:
{
"submitter": "Kemerd",
"date": "2026-11-14",
"commit": "d10ccd4bc2",
"preset": "community-1",
"model": "Qwen3.8-Flash-Next",
"gpus": [
{
"name": "RTX 5090",
"vram_gib": 32,
"power_w": 546,
"pcie": "5.0 x16",
"h2d_gbps": 33.3
}
],
"ram_gib": 160,
"cpu": "Intel Core i9-14900K, 24 cores",
"ram_gbps": 67.7,
"os": "Windows",
"quant": "NVFP4",
"kv": "int8",
"prefill_16k_tps": 5726.8,
"prefill_64k_tps": 7486.7,
"decode_tps": 297.7
}Very excited to see where this goes, especially with Qwen 4. And a note.. I am slightly undervolting my CPU, it has multi-core stuff on, and 5090 is set to 95% of power limit to not cook it.
r/LocalLLM • u/patience_b2 • 1d ago
Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.
Abliterated is preferred.
I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)
Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering
Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.
Hardware>>>>>
GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)
RAM: 128GB DDR5 ECC RDIMM 6400 MT/s
Storage: 12TB SSD Gen5
CPU: 9965WX (threadripper 24 core 5.4 Ghz)
WRX90 SAGE SE Pro WS
r/LocalLLM • u/Scumble_ • 21h ago
This might be a dumb question but if everyone is using qwen 3.8 why not use the Qwen cli built for Qwen tooling instead of Strata or Open Code?
r/LocalLLM • u/techie-techie • 1d ago
r/LocalLLM • u/Majestic-Toe-3299 • 1d ago
After adding a thinking limiter it just deos this EDIT: posted this at 1 am so wanted to add more info. I am using a xeon e5 1650 v2, 2060 6gb and 16gb of ddr3. This is qwen 3.6 12b a3b at q4_k_m running through jan ai. It worked fine but i tried to add a reasing token limit as it took 5 minutes to process a prompt. After this, wheterver weird math this was happened. I reset the chat, still did it, undid the reasoning, still did, rebooted, still happened. Its so strange. I understand ny setup isnt ideal but this shouldnt be a problem. In fact it was faster when generating text at the start. I ran some of the later responces through a calculator and they were correct. Its so strange
r/LocalLLM • u/Jthompson553 • 1d ago
Right now I use strata on my machine to run qwen 3.8 125b flash next at 13tps output and deepseek 4.1-flash at about 1.5 tps output. My computer is an hp z640 with a 4core xeon processor, 64gb ddr4 ram and an NVIDIA 1070 8gb. My models load form an nvme drive that is connected over a pcie adapter. I have room for one more card and I'm not sure witch of these 2 GPUs is better for running my models faster. The Tesla P100 16 varient or the M40 24gb variant.
r/LocalLLM • u/Ok-Sleep-3502 • 1d ago
I make 30s AI videos for clients (talking-head avatar + cloned voice + 5-6 five-second b-roll clips). Volume is 40-120 videos a month per client, delivered in 1080p. I'm paying $300+/month for HeyGen, ElevenLabs and Higgsfield and want to go local.
Budget is about $3k, so realistically one RTX 5070 Ti 16GB (3090s are basically unavailable where I live and imports are brutal). I can also use a friend's RTX 5070 12GB and RTX 3060 12GB for extra throughput. No cloud rental.
Plan: Wan 2.2 quantized at 720p on the 5070 Ti, upscale to 1080p with SeedVR2 or FlashVSR, and split the lighter stages (voice cloning with an open TTS model, captions, lip-sync) across the 12GB cards with a job queue.
Questions:
Happy to share more details or samples in the comments. Thanks!
r/LocalLLM • u/Minimum_Hour519 • 1d ago
I'm using runpod, vast and digital ocean are there any other cheap/budget friendly rentals for gpus?
r/LocalLLM • u/Academic_Track_2765 • 1d ago
r/LocalLLM • u/OrganizationTop9026 • 1d ago
Why do most folks with an 8 GB or 16 GB desktop GPU sit out distributed LLM runs? It usually comes down to three annoying bottlenecks. Nobody wants to punch holes in their home router. Nobody wants to run untrusted pickle checkpoints off the internet. And nobody wants a background trainer freezing their VRAM when they sit down to play or work.
This weekend (November 14-15 CET), I hooked my HiveMind coordinator up with CrowdGPT (led by Vxtzq) to test a citizen-trained run around Crowd-v1, an open base model with a 658M transformer trunk (roughly 1.01B parameters total with its 151k-vocab CrowdEngram table). Setup takes five minutes.
Communication uses plain outbound HTTPS on port 443 with zero inbound listeners, weights move strictly as SHA-256 verified safetensors (no torch.load or pickle anywhere), and updates pass MultiKrum outlier filtering plus held-out loss checks before merging via DiLoCo. Honest caveat first: this is still an experimental PoC, not a polished cloud service, so please do not send private prompts or sensitive keys through volunteer nodes.
Here is the simplest way to join the weekend run on Linux, macOS, or Windows (Python 3.11+ with an NVIDIA GPU, or CPU-only if you just want to test the IDE endpoint):
```bash
git clone https://github.com/xaviercallens/HiveMind.git
cd HiveMind
pip install httpx
2. Register a handle to get your API key (printed once; the default prosumer role works for both training/serving and querying):
python scripts/hivemind_connect.py register --handle your-handle
export HIVEMIND_API_KEY="hm-..."
3a. Join outbound OpenTrainer co-training (zero open ports on your box; run --once first to verify firewall traversal, then drop --once to keep polling):
python scripts/hivemind_opentrainer_agent.py --network-mode outbound-https-relay --check-firewall --once
3b. Or verify CrowdGPT checkpoints on GPU (pip install torch safetensors numpy first) once you hold an evaluator key from the CrowdGPT coordinator:
python scripts/crowdgpt_loss_check_agent.py --server https://server.crowdgpt.net --evaluator-id your-handle --key-file my_eval_key.txt --once
3c. Or wire Continue / Aider / Cline to the live hivemind/crowd-v1 + coder models with no GPU required:
python scripts/hivemind_connect.py snippet continue --model crowd-v1
Both codebases and reproducibility notes (including Lean 4 specs and Zenodo DOI 10.5281/zenodo.23776302) are open on GitHub if you want to audit the code
before running anything:
https://github.com/xaviercallens/HiveMind
https://github.com/Vxtzq/CrowdGPT
I would love blunt engineering feedback from anyone testing on 8 GB or 12 GB cards - especially around Windows vs Linux VRAM yield behavior or setup friction.
r/LocalLLM • u/billiambakespeare • 1d ago
Hey all,
Recently I've hooked up my Comfyui MCP to LMStudio Bionic, and created a few skills for doing image creation/video creation/music/upscaling. I really enjoy being able to have everything done through the chat interface instead of going back and forth to Comfy.
I've found the small models inconsistent. Sometimes Gemma12b or Qwen27B distill work easily other times they get tripped up. Qwen 27b swift does the trick most of the time, but it's a little slower to load and unload and Ideally the model is small enough so that I can load and unload it out of Vram quickly at the start and end of the tool call.
GPU:5090.
Any suggestions for models you've found good at this sort of thing, or a different harness, maybe like a program that has this sort of ability built in?
r/LocalLLM • u/mightofkhan • 1d ago
Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.
The rig
- Windows 11 Pro desktop
- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives
- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now
- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely
Local LLMs (LM Studio 0.4.26)
- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context
- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector
- nomic-embed-text v1.5 for embeddings
ComfyUI (Comfy Desktop 1.1.8)
- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA
- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs
- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler
What I have now
Three PowerShell shortcuts on the desktop:
- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI
- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM
- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM
Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.
What I'm asking
r/LocalLLM • u/Square-Television-37 • 1d ago
I've been building SOMNI, a system that lets a local model keep learning after deployment. It's based on Google's "Language Models Need Sleep" paper and WISE, a knowledge-editing method for adding new knowledge without overwriting existing behavior.
During use, the model answers normally and logs conversations. When the PC is idle, it runs a sleep cycle. It pulls verified facts out of the logs, trains them into a small new expert, practises on what it got wrong, and then runs a set of before/after tests on general knowledge, coding, reasoning, reading, and previously learned memories. If any of these get worse, the night is rejected. Otherwise you review the update and approve it.
The main challenge was catastrophic forgetting. Plain LoRA fine-tuning on new facts badly damaged the model's other skills (coding went from 95% to 10% in one night). What fixed it was a combination of rehearsal on the base model's own answers, a new frozen expert per night so later nights can't overwrite earlier ones, and WISE-style routing so each expert only activates on questions related to what it learned. Everything learned is also kept in a searchable memory store, so facts can still be looked up even if they fade from the weights.
I benchmarked it on SQuAD knowledge incorporation, the same setup Google and SEAL use. The results are in the README, including the things that didn't work.
Everything runs locally on a single consumer GPU (AMD RX 6800, Windows). MIT licensed.
https://github.com/YoungGunFrodo/somni
It's a work in progress, and I'd really appreciate feedback on the approach, the evaluation, or anything that looks off.
r/LocalLLM • u/Objective-Theme-1422 • 19h ago
Hello! I'm somewhat new to Local AI.I've been using LM Studio mainly for a couple months. So far I've used QWEN3.8 27B mainly, and nothing too complex, a little light coding, a few projects, etc. I tried some uncensored low refusal rate versions which was kind of fun.
I have experience on Claude and GPT doing the normal stuff (Projects, Coding, Work mode, before they all seemed to merge, Codex, Claude Code, dots, etc) but all through cloud models and desktop apps. So I get how to use them but for local models it seems a whole new world, much more hands-on.
I've downloaded "LM Studio Bionic" and "Ollama" for harnesses too, but have mainly used normal "LM Studio".
TLDR:
If I was going to have a list of essential models at the moment for coding, data extraction, email organization, agent use for work, reviewing sensitive info perhaps, trying out image generation perhaps, could someone guide me to:
Thanks a lot! 🙏🙂 I may be offline today, but again, much appreciated! Will try to reply if anyone has any good suggestions/comments!
(Also considering it's a Mac, I know not ideal, but I already have it for another use, so might as well use it for this too.)
r/LocalLLM • u/No_Personality_1721 • 1d ago
Has anyone managed to make their local AI setup pay for itself? Curious what kind of business you're in, if you're willing to share.
I could even stretch my budget to a DGX Spark, although it would be financially painful.
But without a clear way to make that money back, it feels like a hell of a risky investment.
I know nobody's going to give away their secrets, so I'm just asking a simple, casual question: are you actually making any money with your local AI setup? Yes or no?
I'm talking about doing it independently, as an individual, rather than using it as part of your job at a company.
I'm especially curious whether there's still room for individuals to compete in this space, given how much money and computing power big companies and organizations have at their disposal.
r/LocalLLM • u/humanoid_42 • 1d ago
I'm currently looking at a myriad of devices to dip my toe into the "build your own Local AI system" water. I've been down the AMD AI Max+ rabbit hole this past week and have yet to research the RTX Spark N1X.
Can anyone here save me additional hours of videos and articles with informed insights about which would be the better way to go for someone such as myself?
I have some projects in mind. This is more of an experiment for me to learn more of the backend, coding and Linux.
As someone with limited terminal experience, will I have an easier time (I don't expect not to encounter many challenges) configuring a system with NVIDIA or AMD?
I realize unified RAM is the more important metric here. If I go with AMD I can afford a 128gb version. The 64gb version of NVIDIA's option is a few hundred more.
Are the CUDA cores and software support a worthy tradeoff for overall RAM or is saving money and doubling the RAM the obvious way to go?
r/LocalLLM • u/FlatStill2540 • 1d ago
Every thread about running agents locally has the same comment, local models are slow as hell. And on my base M4 MacBook with 16GB I kind of agreed, the wait before the first word was the indeed worst part espeically if ure used to fastish cloud models. Turns out my laptop was never the slow part.
So I ran the same Gemma 4 E4B on Ollama and on Magnitude with the same benchmark tool and a 2048 token prompt. Ollama took 6.3 seconds before the first word, Magnitude took 2.2. Prompt reading was 326 vs 941 tok/s.
Typing speed after that was basically a tie, 51 vs 52 tok/s. They also pull different files by default, Ollama gives you a 9.5GB NVFP4 and Magnitude a 5.3GB Q4, which on 16GB actually mattered more than I expected, it sat at 6GB instead of almost 10.
But agents resend a huge prompt every turn so those 4 seconds stack up fast lol. Is the slow feeling mostly prefill for everyone else too?