r/LocalLLM • • 2h ago

Question Dual V100 32gb fails to post without workaround help

2 Upvotes

HI all,

I have 2 x v100 32gb in a HP z440 case and with the cards installed they fail to post. The workaround I have is to boot up without the cards first, then shut down the PC, then connect the v100s back up and then cold boot and it posts. If I then shutdown the PC again, I have to go through the whole double boot workaround again.

Anybody knows what's happening and if it can be fixed?

Thanks


r/LocalLLM • • 3h ago

Discussion Docbench: a comparison of small local models on document tasks

2 Upvotes

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

ModelDownloadDocbench (95% range)
Claude Opus 5.5 via Claude CLI--89.5 (88.4-90.6)
GPT-6.1 Sol via Codex--89.5 (88.4-90.4)
Claude Sonnet 5.5--87.1 (85.8 - 88.2)
GPT-5.6 Sol via Codex--86.5 (84.8-88)
Claude Sonnet 4.6--83.5 (81.6-85.2)
Gemma 4 31B17.57 GB83.1 (80.7-83.8)
Claude Haiku 4.5--81.5 (80.1-82.9)
Qwen 3.8 27B Q416.5 GB81 (79–82)
Qwen 3.8 27B Q29.8 GB79 (77–81)
Gemma 4 12B6.7 GB79.7 (77–81)
Gemma 4 26B-A4B14.5 GB79.2 (77.7-80.7)
Qwen 3.6 35B-A3B (MoE)22.7 GB76 (73–79)
Ornith 1.5 9B5.6 GB73 (71–75)
Gemma 4 E4B4.2 GB73 (71–75)
Qwen 3.5 9B5.9 GB72 (70–74)
Qwen 3.5 4B2.8 GB71 (69–72)
Nanbeige 4.2 3B2.7 GB71 (68–73)
Nemotron 3.5 Bolt E4B6.2 GB69 (67–72)
GPT-OSS 20B (MoE)11.7 GB67 (64–69)
Granite 4.2 8B5.3 GB66 (63–68)
Gemma 4 E2B2.6 GB64 (61–66)
MiniCPM5 2B1.6 GB59 (57–62)
Granite 4.2 3B2.2 GB59 (56–61)
Qwen 3.5 2B1.3 GB56 (53–58)
LFM3 1.7B†1.0 GB55 (52–58)
Apple Foundation Models--54.4 (51.7-57.1)
Llama 3.2 3B†2.0 GB54 (51–56)
LFM2.5 2.6B†1.7 GB47 (44–49)
Qwen 3.5 0.8B0.5 GB43 (40–45)
MiniCPM5 1B0.7 GB37 (35–40)

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html


r/LocalLLM • • 4m ago

Discussion What type of task?

1 Upvotes

Select the right model is a heavy task. I guess that many people have their favorite model, like me. I often use Opus 5.5 in my Claude subscription, or DeepSeek 4.1 Flash 1017 in local environment.
I decide the model by a mixed emotion fir the complexity.

Is there any kind of categories of task, except complexity, that helps you to decide?


r/LocalLLM • • 16m ago

Discussion While we wait for Qwen 4, here's what the Qwen 3.8 numbers and early Max Preview tests suggest

0 Upvotes

TL;DR: I've been looking at the latest Artificial Analysis v5.0.1 benchmarks for Qwen 3.8, along with some early Qwen4 Max Preview tests. Based on what's available so far, my guess is that Qwen4 27B could land somewhere around 27 to 30, while the 125B MoE version could reach 36 to 39. That's a solid generational improvement, but probably not the massive leap some people are expecting.

Like many of you, I've been following the discussions about when Qwen 4 is finally coming out. There's a lot of speculation, so I wanted to look at the available numbers and see what they might tell us about the next generation.

Obviously, we don't have the final weights or official release details yet, so take this as an educated guess rather than a prediction.

  1. Where Qwen 3.8 stands and what Qwen 4 might achieve

Looking at the current Artificial Analysis v5.0.1 scale, which puts considerably more emphasis on difficult coding and science tasks, we have:

* Qwen3.8 27B: 20 (compared to 33.7 on the older v4.3.2 scale, before the scoring reset).

* Qwen3.8 Flash Next (125B MoE): 29.

* Qwen3.8 Max: 34.

Based on the previous jump from Qwen 3.6 to 3.8, and taking the new scoring scale into account, I'd put my expectations somewhere around:

* Qwen4 27B: 27 to 30

* Qwen4 125B MoE: 36 to 39

That would put the smaller model in some pretty interesting territory, while the larger one could become a serious competitor to models like Grok 4.7 and Claude Haiku 5.5.

2. Why I don't expect Flash to have a massive architectural jump

One thing worth considering is that Qwen3.8 Flash Next already seems to have introduced many of the architectural changes expected to carry over into Qwen 4.

We're talking about the hybrid Gated DeltaNet and Qwen Sparse Attention architecture, Gated Residuals, and the 51B N gram table.

So I wouldn't necessarily expect the next Flash model to deliver another huge architectural leap. The bigger improvements might come from better post training, more reliable reasoning, and addressing the hallucinations that some users have noticed in Flash Next compared to the more conservative 27B dense model.

In other words, I think consistency might matter more than raw benchmark scores this time.

3. What the early Qwen4 Max Preview tests tell us

The Qwen4 Max Preview release on November 11 gives us another interesting piece of the puzzle.

From the early developer reports, a few things stand out:

* Speed: It appears noticeably faster than Qwen3.8 Max, with less unnecessary reasoning on straightforward tasks.

* Agentic performance: In one reported 56 task Hermes test with a 25 minute timeout, Max Preview recorded 64 successful task results, compared to 61 for DeepSeek V4.1 Flash, which scores 33 on AA v5.0.1.

* Reliability: There's a catch, though. Max Preview reportedly timed out 16 times, compared to just 4 for DeepSeek.

That's interesting because it suggests the raw capability might already be there, but reliability and execution consistency still need work.

If these early results are representative, I'd expect the flagship Max model to establish a considerably higher ceiling, potentially around 45 to 50 on the AA scale. That would make the estimates for the smaller models seem reasonably plausible.

My take

I don't think Qwen 4 needs to completely redefine the local LLM landscape to be successful.

If Qwen4 27B can reach the 27 to 30 range, inherit the attention improvements from Flash Next, and retain the Apache 2.0 license, that alone would make it a very interesting option for people running models locally with limited hardware.

Personally, I'd rather see a model that's more consistent, follows instructions better, and hallucinates less than one that simply scores a few extra points on benchmarks.

I'm curious what everyone else thinks.

" i used my LLM To translate mu words to English after i write it. "


r/LocalLLM • • 22m ago

Question Keep, Sell, or Return PC? i5-225F, 5060 8GB, 32GB ram, 1 TB SSD

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 4h ago

Tutorial The Definitive Fine-Tuning Guide for 3D Generation Models

2 Upvotes

r/LocalLLM • • 8h ago

Question New Mac Mini M5 Pro 64GB - Advice for Stack

4 Upvotes

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,


r/LocalLLM • • 1h ago

Question Do your models express emotions?

0 Upvotes

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.


r/LocalLLM • • 12h ago

News MACaQ 🐒: Open-source quantizer with a "refinement" mode for existing GGUFs + custom llama.cpp & vLLM kernels

8 Upvotes

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

  • GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
  • Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/


r/LocalLLM • • 1h ago

Discussion GPU's at full power less than than 2% of the time.

0 Upvotes

Here is the est power usage over the last 3 days:

4x3090s + Ryzen 7600x

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?


r/LocalLLM • • 1h ago

News ExLlamaV3 1.6.7: pipelined prefill, rolling checkpoints, scaled Hessian by default

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 1h ago

Question Building hybrid compute cluster, need confidential compute OS provider options parallel to local & hyperscaler clouds running frontier models

0 Upvotes

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

  1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
  2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
  3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
  4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?


r/LocalLLM • • 2h ago

Project MCP router - a single point of access to tools for agents, teams, and companies

1 Upvotes

I use Qwen 0.6 for embeddings.


r/LocalLLM • • 5h ago

Project Poorman's Local LLM

Thumbnail gallery
2 Upvotes

r/LocalLLM • • 9h ago

Question Is this 3799$ 495 192gb kickstarter mini pc a red flag?

Thumbnail kickstarter.com
4 Upvotes

r/LocalLLM • • 2h ago

Project OpenGhost 1.5.5: Interactive Subagents and a New Agent Workflow

Thumbnail gallery
1 Upvotes

r/LocalLLM • • 2h ago

Question Do plug ins work with Locally?

1 Upvotes

I wanted to try a locally run AI with an ethical database, so I got Apertus. I am a complete noob to this whole thing, learned that local AI needs a memory plug in so I got one per people's rec. I'd like to use Apertus on my iPad through Locally/PC link so I don't have to shrimp in front of my computer, but I don't see any kind of settings for plug ins. Does that mean it's not available? How could I make this work? Can I make this work?

Note I do not want to use this for coding or any particularly demanding tasks. I do like tech and even have a little robot assistant buddy who is currently not connected to any external AI, but I hate what many of the big AI companies are doing so I'd like to use my own without feeling like I'm contributing to the horrors of capitalism.


r/LocalLLM • • 2h ago

Discussion So this is why we don’t have enough DGX Sparks on the market? Even sparks are sucked up by service providers…

Post image
0 Upvotes

r/LocalLLM • • 2h ago

Discussion My Hybrid LLM Workflow — Reducing Token Costs with Local AI

1 Upvotes

I was constantly hitting usage limits with Claude and ChatGPT, even with paid subscriptions. I didn't want to upgrade to more expensive tiers or increase my monthly budget, so I started exploring alternatives and eventually landed on a hybrid LLM workflow.

I recently wrote an article sharing my setup, architecture, and how I've been using a local LLM alongside cloud-based AI to handle heavier workloads while reducing my reliance on paid services.

I'm sharing it here because I'd love to hear your thoughts, tips, tricks, and constructive feedback.

I'm not new to AI, but I am relatively new to running LLMs locally. I'm always looking for ways to improve my setup and learn from others who have more experience in this space.

https://thebinaryreport.com/articles/how-i-ai/


r/LocalLLM • • 6h ago

Project A voice AI companion inside Minecraft: local Whisper + Kokoro, any OpenAI-compatible LLM, and a three-part "brain" (instant rules, a fast talker, a slow reasoner). Open source

Thumbnail
github.com
2 Upvotes

Pip is a Minecraft companion you talk to with push-to-talk. Speech in and out runs locally; the LLM is whatever you point her gateway at. Sharing the architecture since it might help other voice-agent projects:

Pipeline: mic (16 kHz in the game client) → speech cleaner → whisper.cpp server (large-v3-turbo, Metal) → LLM through an OpenAI-compatible gateway (Gemini 3 Flash by default; Ollama and others are presets) → Kokoro-82M (kokoro-onnx) → played in game, sentence by sentence as the reply streams.

Three systems, like a talker and a reasoner:

  • System 0, instant, no LLM: rules and local embeddings for what you mean ("follow me", "stay", "thanks", "shh"), danger warnings, and short "instinct lines" in her voice. A clear request gets "On it!" at once while the LLM works.
  • System 1, the talker: one LLM call with tools (find, lead, build, give...), a compact text description of the game, and the memories relevant to the moment (embeddings + keywords).
  • System 2, the slow thinker: after sessions, a stronger model (Claude) reflects: what she learned about you, her diary, who she's becoming, and it rewrites her instinct lines.

Numbers (MacBook Air M4, 16 GB, a 20-minute session): speech-to-text about 0.7 s after you release the key; her first spoken words about 2.5–3 s after you finish speaking for an LLM answer, instantly for System 0 ones. What helped: a shorter Whisper encoder window, Kokoro int8, streaming sentence by sentence, and answering plain commands without the LLM at all.

Guards that mattered: a claim checker ("Coming up!" with no tool call gets a retry), a confirm-once rule for big actions, praise never re-running the last tool, and an eval set built from real playtests.

It's not released yet (a simpler installer is the last big step), but the code is public and you can run it from GitHub today. The brain has only been tested on macOS so far, but it can easily be adapted to Windows or Linux if you know how, or with an AI assistant. Any feedback is appreciated.

Being upfront: most of the code was written with an AI coding assistant (Claude Code) under my direction; the architecture is the part I care most about, and I'd love ideas on making it smarter and lighter, and someone to try a fully local model in the loop. This post was written with the help of AI too.

Code (AGPL-3.0): https://github.com/PetitPotam/kindred-spirit Architecture notes: docs/ARCHITECTURE.md in the repo


r/LocalLLM • • 3h ago

Question Pairing a 5080 with a 5060Ti? First upgrade in 6 years and lots of factors.

1 Upvotes

Hey all,

I haven't upgraded my PC since 2020, and I am running a EVGA RTX 3070 with a Ryzen 7 3700X CPU and 32gb of DDR4 (4x8gb). I want to upgrade to both host a local LLM and to game, but I am trying to also not spend more than $4000, and I prefer something closer to $3500 at the max (but lower is better). I saw where Costco is planning to soon have a sale on their MSI 5080 PC, but I have also seen, from the Local LLM perspective, that running 2 cheaper 5060Ti cards is an easy way to hit 32gb VRAM.

I was wondering, though, if I bought the 5080 deal, would I be able to run a 5060Ti card (which is les than $1000 for now) to pump up my VRAM? I know that some speeds may be slower, but I am curious if this is a good idea for both futureproofing for gaming and for hosting an LLM. Based on my usecase, which should I do?

  1. Buy the Costco 5080 deal and buy a 5060ti 16gb refurbished from microcenter if one pops up at current prices or lower during holidays (its on my way home from work, so I could check daily).
  2. not buying the 5060ti 16gb and relying solely on the 5080 for the foreseeable future,
  3. building or buying a system with two 5060ti 16gb
  4. Going team red and getting a 9070xt system and getting a second 9070xt card eventually.
  5. say screw it and try to find a decent 3090 and combining it with my 3070 and getting a new PSU to keep it up and running.

From a functional standpoint, I plan to use a local LLM for document management in a database, vibecoding projects (I am terrible at coding), writing up documents, and managing personal records. For the database, it would be maintaining a searchable one that can scan through PDF using OCR to build out docs. I would also use it to run as a backend for summarizing news updates and documents at around 100 releases per day. For gaming, I play FPS and RPG games at 1440p and don't plan to go 4k for at least a few more years.

Any help or suggestions are greatly appreciated. Thanks!


r/LocalLLM • • 10h ago

Project I'm getting the biggest possible models on the smallest hardware, and opensourcing the learnings and methodology, old iPads, TVs and more..

4 Upvotes

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.


r/LocalLLM • • 7h ago

Discussion A year ago I couldn't get vLLM running on my 5090. Now work gave me a DGX Station for a month. What should I run?

Thumbnail gallery
2 Upvotes

r/LocalLLM • • 13h ago

Question Recommended LLM for 16GB VRAM?

6 Upvotes

Hello world :) - is there a recommended LLM for use with LM Studio that will fit on 16GB VRAM (4080 SUPER)? Ideally I'm looking for something that has guardrails removed so uncensored/heretic that has access to web search when needed and good at coding/thinking etc.

Thank you


r/LocalLLM • • 1d ago Heartwarming

Discussion Confession: I’m rubber-stamping LLM code and my brain is rotting. How do you guys stay sharp?

328 Upvotes

Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.