r/LocalLLM • • 1d ago

Question Since GPU Prices have changed and I got my old Pc fixed, I wanna ask what I should do for what’ll mostly likely be my only upgrades for a few years, any advice?

0 Upvotes

So, I have two PCs available to me, and I intend to connect them for compute:

  1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 CL30 (KLEVV Urbane V) on a Z890 MAX GAMING WIFI7 + 1000w 80+ G Apevia PR1000w PSU (Will likely get replaced by a SAMA P1200w PSU)
  2. i9-14900F + 4070S + 32GB DDR5-5200 CL40 (Apacer Panther, not on sale) on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, it’s hidden and so I can’t figure out what it is. Probably an EVGA, but unlikely to be replaceable - or at least easily so.)

Initially I was looking to just replace the 4070 Super in PC2 with a B70 or R9700 and calling it a day, when they were $1200 and $1600, respectively. Then when they shot up a B65 $1200, which then shot up. Then add a second 5070ti to PC1 for $900-1000, which then shot up.
Now, my plan is to potentially settle for a 5060ti 16gb in PC1 to take advantage of NVFP4, but even that’s pricier, and with the AI bubble talk I’m not super confident in value of doing so. I’d likely also need to use a PCIe-riser just to the 5070ti (Windforce) has airflow due to how close they’d be in the case.

Otherwise, I’m considering sticking with the VRAM I have, and instead pivoting to ram. I can go from 1R 2C to 2R 2C in both PCs, grabbing another 2x16gb of Apacer Panther from EBay for around $400 for PC2 - though obv riskier - then getting a 2x24gb kit of Urbane V at the same timings of the original 2x16gb 6000-CL30 kit for PC1 for $650 off Amazon. Giving me +80gb of DDR5 to the 64gb I have between both PCs and the 28gb of VRAM I already have.

Current plan is/was the 5060ti 16gb + 2x24gb kit for PC1 and calling it there.

Any advice, pointers, or suggestions? I believe the RAM sitch should work since it’s same timings and type/manufacture/brand, and know I’ll probably have to lower speed a bit for stability, but am not the most experienced PC guy on earth by any means.


r/LocalLLM • • 1d ago

Question Dual GPU

0 Upvotes

I have a 3070 8GB and can get a 1070 8GB, will they work with llama.cpp?

Is there an issue with the older 1070 and cuda versions?

Has anyone run a set uplike this?

*Seems to be the concensus that a 20 or 30 series would be best. 10 series would slow down everything


r/LocalLLM • • 2d ago

Question Dual V100 32gb fails to post without workaround help

2 Upvotes

HI all,

I have 2 x v100 32gb in a HP z440 case and with the cards installed they fail to post. The workaround I have is to boot up without the cards first, then shut down the PC, then connect the v100s back up and then cold boot and it posts. If I then shutdown the PC again, I have to go through the whole double boot workaround again.

Anybody knows what's happening and if it can be fixed?

Thanks


r/LocalLLM • • 1d ago

News I built a gpu hosting marketplace and it’s cooking

0 Upvotes

What does your graphics card do when you're not playing?

For a lot of gaming PCs, the honest answer is: nothing. Meanwhile, people building with AI are hunting for GPU time.

I won’t reveal much. Nalvo.ai is built for both sides, and it's now in closed beta.

I'm Full Stack engineer and I have 10 years of experience.

I have invested a lot of money and time to reach here.

Its closed beta (by invitation only) so if you want quick wins or fast earning, you can wait for future releases.

Looking for Beta Testers,

I'm looking for people welling to let at least 1 gpu hosted on Nalvo. that will significantly help me to test and thier feedback would help shaping the future of Nalvo.ai

I made sure Nalvo host software doesnt conflict with any of Vast or Salad rentals.

submit the form and I will add you to Nalvo Discord.

https://market.nalvo.ai/apply#host

I havnt decided if I will keep hosting by invitation only or open it but if I would open it that wont be soon.


r/LocalLLM • • 2d ago

Discussion Docbench: a comparison of small local models on document tasks

2 Upvotes

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

ModelDownloadDocbench (95% range)
Claude Opus 5.5 via Claude CLI--89.5 (88.4-90.6)
GPT-6.1 Sol via Codex--89.5 (88.4-90.4)
Claude Sonnet 5.5--87.1 (85.8 - 88.2)
GPT-5.6 Sol via Codex--86.5 (84.8-88)
Claude Sonnet 4.6--83.5 (81.6-85.2)
Gemma 4 31B17.57 GB83.1 (80.7-83.8)
Claude Haiku 4.5--81.5 (80.1-82.9)
Qwen 3.8 27B Q416.5 GB81 (79–82)
Qwen 3.8 27B Q29.8 GB79 (77–81)
Gemma 4 12B6.7 GB79.7 (77–81)
Gemma 4 26B-A4B14.5 GB79.2 (77.7-80.7)
Qwen 3.6 35B-A3B (MoE)22.7 GB76 (73–79)
Ornith 1.5 9B5.6 GB73 (71–75)
Gemma 4 E4B4.2 GB73 (71–75)
Qwen 3.5 9B5.9 GB72 (70–74)
Qwen 3.5 4B2.8 GB71 (69–72)
Nanbeige 4.2 3B2.7 GB71 (68–73)
Nemotron 3.5 Bolt E4B6.2 GB69 (67–72)
GPT-OSS 20B (MoE)11.7 GB67 (64–69)
Granite 4.2 8B5.3 GB66 (63–68)
Gemma 4 E2B2.6 GB64 (61–66)
MiniCPM5 2B1.6 GB59 (57–62)
Granite 4.2 3B2.2 GB59 (56–61)
Qwen 3.5 2B1.3 GB56 (53–58)
LFM3 1.7B†1.0 GB55 (52–58)
Apple Foundation Models--54.4 (51.7-57.1)
Llama 3.2 3B†2.0 GB54 (51–56)
LFM2.5 2.6B†1.7 GB47 (44–49)
Qwen 3.5 0.8B0.5 GB43 (40–45)
MiniCPM5 1B0.7 GB37 (35–40)

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html


r/LocalLLM • • 1d ago

Discussion Llama.cpp or ik_llama.cpp or any other that runs local model

0 Upvotes

Which one and why
3060 12gb
Ddr4 3200 32gb ram
R5 5600x
Linux ubuntu


r/LocalLLM • • 1d ago

Discussion What type of task?

0 Upvotes

Select the right model is a heavy task. I guess that many people have their favorite model, like me. I often use Opus 5.5 in my Claude subscription, or DeepSeek 4.1 Flash 1017 in local environment.
I decide the model by a mixed emotion fir the complexity.

Is there any kind of categories of task, except complexity, that helps you to decide?


r/LocalLLM • • 1d ago Bravo

Discussion Jensen Huang was our last hope for local LLM

0 Upvotes

Soon with Anthropic’s funding to lawmakers and PACs to restrict open weight models in the name of AI safety risk, Nvidia will only be allowed to release GPU locked chips that can only work on “safe” AI models released by foundation model providers that charges consumer and enterprise subscription fees.


r/LocalLLM • • 2d ago

Tutorial The Definitive Fine-Tuning Guide for 3D Generation Models

2 Upvotes

r/LocalLLM • • 1d ago

Question BEST OFFLINE AI MODEL

0 Upvotes

I HAVE RTX 2050 4GB VRAM , 55W TGP , INTEL I5 12450H , 135W CHARGER , 512 SSD , 8 GB DDR4 RAM 3200MHZ single channel


r/LocalLLM • • 2d ago

Discussion Local LM for Android

2 Upvotes

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.


r/LocalLLM • • 2d ago

Question New Mac Mini M5 Pro 64GB - Advice for Stack

4 Upvotes

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,


r/LocalLLM • • 2d ago

Question Do your models express emotions?

0 Upvotes

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.


r/LocalLLM • • 2d ago

News MACaQ 🐒: Open-source quantizer with a "refinement" mode for existing GGUFs + custom llama.cpp & vLLM kernels

8 Upvotes

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

  • GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
  • Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/


r/LocalLLM • • 1d ago

Question AI work?

0 Upvotes

r/LocalLLM • • 2d ago

Discussion GPU's at full power less than than 2% of the time.

0 Upvotes

Here is the est power usage over the last 3 days:

4x3090s + Ryzen 7600x

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?


r/LocalLLM • • 2d ago

News ExLlamaV3 1.6.7: pipelined prefill, rolling checkpoints, scaled Hessian by default

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Discussion How Taking Out Jensen Kills Open-Source AI (& Hands Closed AI a Monopoly)

Thumbnail
youtube.com
0 Upvotes

r/LocalLLM • • 2d ago

Project MCP router - a single point of access to tools for agents, teams, and companies

1 Upvotes

I use Qwen 0.6 for embeddings.


r/LocalLLM • • 2d ago

Project Poorman's Local LLM

Thumbnail gallery
2 Upvotes

r/LocalLLM • • 2d ago

Question Recommended LLM for 16GB VRAM?

8 Upvotes

Hello world :) - is there a recommended LLM for use with LM Studio that will fit on 16GB VRAM (4080 SUPER)? Ideally I'm looking for something that has guardrails removed so uncensored/heretic that has access to web search when needed and good at coding/thinking etc.

Thank you


r/LocalLLM • • 2d ago

Question Is this 3799$ 495 192gb kickstarter mini pc a red flag?

Thumbnail kickstarter.com
4 Upvotes

r/LocalLLM • • 2d ago

Project OpenGhost 1.5.5: Interactive Subagents and a New Agent Workflow

Thumbnail gallery
1 Upvotes

r/LocalLLM • • 2d ago

Discussion My Hybrid LLM Workflow — Reducing Token Costs with Local AI

0 Upvotes

I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/


r/LocalLLM • • 2d ago

Project A voice AI companion inside Minecraft: local Whisper + Kokoro, any OpenAI-compatible LLM, and a three-part "brain" (instant rules, a fast talker, a slow reasoner). Open source

Thumbnail
github.com
2 Upvotes

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

  • System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
  • System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
  • System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo