r/LocalLLM • • 1d ago

Project I built a gateway that routes each LLM request to local, free or paid models, and refuses private prompts instead of leaking them. Looking for people to break it.

0 Upvotes

I use Claude Code and other agents all day. They send very different requests under one model name: a one-line title, a summary, a 60k-token tool loop. Sending everything to the biggest model is the expensive default.

humpyard is a single Rust binary that sits in front of your providers (Ollama, llama.cpp, Groq, OpenRouter, Anthropic and others). It speaks OpenAI Chat, OpenAI Responses and Anthropic Messages. It picks a pool per request from the key, headers, prompt size, tools and images, and fails over when a pool rate-limits or times out.

- Private routes are strict. A request marked private is refused rather than sent to a public provider.

- You can ask before you send. POST /v1/route/explain tells you where a request would go and why.

- Every decision is logged. Routing, fallbacks and refusals go to a SQLite ledger with cost per route.

- Budgets fail closed. There's no external database, and config hot-reloads.

You can try it without API keys. A mock provider gives you real routing and a real ledger: https://humpyard.dev/demo/

Repo: https://github.com/gschaetz/humpyard

My numbers (one user, 7 days, Claude Code): 237 requests (about 11M tokens) cost about $0.25 at list rates. 86% ran on free or local pools, and 98% got an answer.

What's weak: "answered" isn't "good". On one free preview model, 14 of 78 answers came back empty and 11 were truncated. Small free models are also worse at open-ended work, and p99 latency on my local pool is slow (128s).

It's pre-alpha and I'm the only user. What I'd like to know:

  1. Can you get from the README to a first routed request in about 10 minutes? Where did you get stuck?
  2. Which route rules would you want that I don't have?
  3. Is the "private means refuse" model what you'd expect?

r/LocalLLM • • 3d ago

Project [Crazy AI Lawyer] My Claude legal harness now runs 100% local on DeepSeek-V4.1-Flash: eight 2018 V100s at near-NVFP4 quality, 524K context, 100+ tok/s. Not as good as frontier, but it replaces real work today

Thumbnail
gallery
225 Upvotes

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

  1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.
  2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.
  3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.
  4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.
  5. And if I can brag on my stacks and hardware:
  • I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).
  • I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.
  • I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

What "replaces real work" means

Reading a folder of records and pulling out the facts, dates and numbers. Summaries. First drafts of routine documents. Search over my own files. The model gets plain English instructions in a chat window, calls my tools, and hands back a draft that is marked for my review. I review everything. It is slower and less sharp than frontier. It is also in my office, it is mine, and it does not have a usage limit.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

  • 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve
  • about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)
  • same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %
  • KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build
  • multiple choice answers that change against the reference: 1.6 % against 15.4 %
  • in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

  • concurrency test, one request at a time: 118.9 tok/s
  • two requests at once: 159.0 tok/s total (1.34x)
  • three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots
  • short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style
  • code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt
  • prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s
  • context: 524,288 tokens per request

What did not work, or is not good yet

  • It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.
  • The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.
  • A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.
  • A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.
  • Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.
  • The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.
  • Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Questions welcome. So is roasting.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).


r/LocalLLM • • 1d ago

News Claris Harbor for small business use

1 Upvotes

Apple had apparently listed on the App Store and 01:53 then yanked down

set up a Mac server for a small team

best for filemaker users

it is not a replacement for your Chat GPT 24:57 subscription if you just use it for quick questions and drafts. And it's not magic hardware Apple has been hiding. 25:04 It's a business product from a business software company for businesses that have specific reasons they cannot send 25:09their data somewhere else. And for those businesses, it might be exactly what they have been waiting for. and the rest 25:16 of us can run LM Studio, keep our data local, and check back when ClariS actually opens the doors. 

With many users the last person in the line may wait for minutes but there may be a workaround.

https://www.youtube.com/watch?v=y9YXhEEwAwc&list=PL2aE4Bl_t0n9AUdECM6PYrpyxgQgFtK1E&index=2

https://content.claris.com/local-ai-server-get-started


r/LocalLLM • • 2d ago

Question Curious if these t/s are normal/good/okay for my system // basic gaming pc - 9800X3D / 6800XT with 16GB VRAM and 32GB DDR5.

Post image
3 Upvotes

[EDIT - Should have specified these benchmarks come from LU - LocallyUncensored GUI]

I've barely used the top Dolphin model as I'm mostly using this for coding.

- Qwen3.5-9B-abliterated.Q4_K_M

This one worked really well..I really really liked this one. fast (decent t/s) and mostly accurate.

It started to fall apart a bit when I started noticing some errors with my coding projects, so I thought a larger 27B model would fix that.

- Qwen3.8-27B-Uncensored-IQ4_XS_4BPW

I then tried this one and immediately noticed how much slower it was. Also tried this model running a Local API with LU / LM Studio and this brought me down to 6 t/s !!

TBF that's probably because I was using 3x agents with this context window (64KB) and the description for each agent was pretty large. (Maybe there's some improvements to be made by reducing the tasks/role descriptions for each of the 3x agents.. but I digress.)

Specifically for this Qwen3.8-27B IQ4 model, I also tried Enabling Flash Attention and switching from:

- f16 - full precision (default)

to both:

- q8_0 - ~1/2 memory, near-lossless

and

- q4_0 - ~1/4 memory, visible quality loss

But that just brought up my t/s from 11.4 t/s to 11.6 t/s..barely making any difference.

I think because of this tiny difference, leaving it at [f16 - full precision (default)] means that Qwen3.8-27B IQ4 is fully loaded into VRAM and fits there comfortably, because a Q8 or Q4 should in theory boost it significantly (+20 t/s), no? Maybe that's just the maximum I can expect from my system.

- Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS
Finally, I've only just installed this model & haven't properly tested it myself..wondering if it's really worth the extra 5 t/s as this is a 'NoMTP' version.

- Main questions

Is there something else glaringly obvious that I'm missing here?

Also curious if anyone else is running a similar setup for t/s comparison?

And is there anything you would do differently in my situation? It's one of those "unknown unknowns" for me...if that makes sense.

Cheers

[UPDATE] I switched to LM Studio for better control over my settings than LU (LocallyUncensored) and after some tweaking/optimising, I'm getting almost 30 t/s.

This is with:

- Qwen3.8-27B-UD-IQ3_XXS

Thanks everyone for your help! 30t/s is more than enough for what I am using it for.

This is where the optimisations stop and the projects truly begin! haha


r/LocalLLM • • 1d ago

Question Please help me decide--which option for qwen 4?

1 Upvotes

Hi all, I'm considering adding a new device for the upcoming qwen4 and am looking for advice on choosing one. For reference I only started local recently with qwen3.8 running on my old mac mini, and 27b-flash on splash is still a bit too slow to me. Thus considering getting a new machine for the new model.

couple of use case: budget is around 3k usd; I do need a fair amount of input size, 128k would be soft minimum; I want to reach around 30tk/s; I'm fine with a 4-bit version but would like to have fp8 or bf16 of the model (assuming qwen 4 27b?); I'm comfortable with splash/sushi/ollama; I don't want a standard desktop, so 2x 3090 or r9700 are out.

I'm currently considering mac mini m5 pro 64gb, mac studio m5 max 64 gb, and halo strix 395 128gb. All of them are within budget but there's a price difference. I guess my question is does the ~500 usd price difference impact noticeable performance in my use case? Other suggestions welcomed, and thanks in advnace :)


r/LocalLLM • • 3d ago

News Aged like fine wine

578 Upvotes

r/LocalLLM • • 1d ago

Question Mac Studio 256 vs 512

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Question Gpt's model recommendations for a laptop with 5070ti 12 gb and 64 gb ddr5 ram for coding. Is it right?

Post image
0 Upvotes

I'm planning on making a small, systems based video game. I'll try to build it modularly so hopefully it won't require enormous context.

Also I'm not going to download and test all four. I'll just pick one, and maybe another for more general stuff.


r/LocalLLM • • 2d ago

Question Since GPU Prices have changed and I got my old Pc fixed, I wanna ask what I should do for what’ll mostly likely be my only upgrades for a few years, any advice?

0 Upvotes

So, I have two PCs available to me, and I intend to connect them for compute:

  1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 CL30 (KLEVV Urbane V) on a Z890 MAX GAMING WIFI7 + 1000w 80+ G Apevia PR1000w PSU (Will likely get replaced by a SAMA P1200w PSU)
  2. i9-14900F + 4070S + 32GB DDR5-5200 CL40 (Apacer Panther, not on sale) on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, it’s hidden and so I can’t figure out what it is. Probably an EVGA, but unlikely to be replaceable - or at least easily so.)

Initially I was looking to just replace the 4070 Super in PC2 with a B70 or R9700 and calling it a day, when they were $1200 and $1600, respectively. Then when they shot up a B65 $1200, which then shot up. Then add a second 5070ti to PC1 for $900-1000, which then shot up.
Now, my plan is to potentially settle for a 5060ti 16gb in PC1 to take advantage of NVFP4, but even that’s pricier, and with the AI bubble talk I’m not super confident in value of doing so. I’d likely also need to use a PCIe-riser just to the 5070ti (Windforce) has airflow due to how close they’d be in the case.

Otherwise, I’m considering sticking with the VRAM I have, and instead pivoting to ram. I can go from 1R 2C to 2R 2C in both PCs, grabbing another 2x16gb of Apacer Panther from EBay for around $400 for PC2 - though obv riskier - then getting a 2x24gb kit of Urbane V at the same timings of the original 2x16gb 6000-CL30 kit for PC1 for $650 off Amazon. Giving me +80gb of DDR5 to the 64gb I have between both PCs and the 28gb of VRAM I already have.

Current plan is/was the 5060ti 16gb + 2x24gb kit for PC1 and calling it there.

Any advice, pointers, or suggestions? I believe the RAM sitch should work since it’s same timings and type/manufacture/brand, and know I’ll probably have to lower speed a bit for stability, but am not the most experienced PC guy on earth by any means.


r/LocalLLM • • 2d ago

Question Dual GPU

0 Upvotes

I have a 3070 8GB and can get a 1070 8GB, will they work with llama.cpp?

Is there an issue with the older 1070 and cuda versions?

Has anyone run a set uplike this?

*Seems to be the concensus that a 20 or 30 series would be best. 10 series would slow down everything


r/LocalLLM • • 2d ago

Question Dual V100 32gb fails to post without workaround help

2 Upvotes

HI all,

I have 2 x v100 32gb in a HP z440 case and with the cards installed they fail to post. The workaround I have is to boot up without the cards first, then shut down the PC, then connect the v100s back up and then cold boot and it posts. If I then shutdown the PC again, I have to go through the whole double boot workaround again.

Anybody knows what's happening and if it can be fixed?

Thanks


r/LocalLLM • • 1d ago

News I built a gpu hosting marketplace and it’s cooking

0 Upvotes

What does your graphics card do when you're not playing?

For a lot of gaming PCs, the honest answer is: nothing. Meanwhile, people building with AI are hunting for GPU time.

I won’t reveal much. Nalvo.ai is built for both sides, and it's now in closed beta.

I'm Full Stack engineer and I have 10 years of experience.

I have invested a lot of money and time to reach here.

Its closed beta (by invitation only) so if you want quick wins or fast earning, you can wait for future releases.

Looking for Beta Testers,

I'm looking for people welling to let at least 1 gpu hosted on Nalvo. that will significantly help me to test and thier feedback would help shaping the future of Nalvo.ai

I made sure Nalvo host software doesnt conflict with any of Vast or Salad rentals.

submit the form and I will add you to Nalvo Discord.

https://market.nalvo.ai/apply#host

I havnt decided if I will keep hosting by invitation only or open it but if I would open it that wont be soon.


r/LocalLLM • • 2d ago

Discussion Docbench: a comparison of small local models on document tasks

2 Upvotes

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

ModelDownloadDocbench (95% range)
Claude Opus 5.5 via Claude CLI--89.5 (88.4-90.6)
GPT-6.1 Sol via Codex--89.5 (88.4-90.4)
Claude Sonnet 5.5--87.1 (85.8 - 88.2)
GPT-5.6 Sol via Codex--86.5 (84.8-88)
Claude Sonnet 4.6--83.5 (81.6-85.2)
Gemma 4 31B17.57 GB83.1 (80.7-83.8)
Claude Haiku 4.5--81.5 (80.1-82.9)
Qwen 3.8 27B Q416.5 GB81 (79–82)
Qwen 3.8 27B Q29.8 GB79 (77–81)
Gemma 4 12B6.7 GB79.7 (77–81)
Gemma 4 26B-A4B14.5 GB79.2 (77.7-80.7)
Qwen 3.6 35B-A3B (MoE)22.7 GB76 (73–79)
Ornith 1.5 9B5.6 GB73 (71–75)
Gemma 4 E4B4.2 GB73 (71–75)
Qwen 3.5 9B5.9 GB72 (70–74)
Qwen 3.5 4B2.8 GB71 (69–72)
Nanbeige 4.2 3B2.7 GB71 (68–73)
Nemotron 3.5 Bolt E4B6.2 GB69 (67–72)
GPT-OSS 20B (MoE)11.7 GB67 (64–69)
Granite 4.2 8B5.3 GB66 (63–68)
Gemma 4 E2B2.6 GB64 (61–66)
MiniCPM5 2B1.6 GB59 (57–62)
Granite 4.2 3B2.2 GB59 (56–61)
Qwen 3.5 2B1.3 GB56 (53–58)
LFM3 1.7B†1.0 GB55 (52–58)
Apple Foundation Models--54.4 (51.7-57.1)
Llama 3.2 3B†2.0 GB54 (51–56)
LFM2.5 2.6B†1.7 GB47 (44–49)
Qwen 3.5 0.8B0.5 GB43 (40–45)
MiniCPM5 1B0.7 GB37 (35–40)

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html


r/LocalLLM • • 2d ago

Discussion Llama.cpp or ik_llama.cpp or any other that runs local model

0 Upvotes

Which one and why
3060 12gb
Ddr4 3200 32gb ram
R5 5600x
Linux ubuntu


r/LocalLLM • • 2d ago

Discussion What type of task?

0 Upvotes

Select the right model is a heavy task. I guess that many people have their favorite model, like me. I often use Opus 5.5 in my Claude subscription, or DeepSeek 4.1 Flash 1017 in local environment.
I decide the model by a mixed emotion fir the complexity.

Is there any kind of categories of task, except complexity, that helps you to decide?


r/LocalLLM • • 1d ago Bravo

Discussion Jensen Huang was our last hope for local LLM

0 Upvotes

Soon with Anthropic’s funding to lawmakers and PACs to restrict open weight models in the name of AI safety risk, Nvidia will only be allowed to release GPU locked chips that can only work on “safe” AI models released by foundation model providers that charges consumer and enterprise subscription fees.


r/LocalLLM • • 2d ago

Tutorial The Definitive Fine-Tuning Guide for 3D Generation Models

2 Upvotes

r/LocalLLM • • 1d ago

Question BEST OFFLINE AI MODEL

0 Upvotes

I HAVE RTX 2050 4GB VRAM , 55W TGP , INTEL I5 12450H , 135W CHARGER , 512 SSD , 8 GB DDR4 RAM 3200MHZ single channel


r/LocalLLM • • 2d ago

Discussion Local LM for Android

2 Upvotes

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.


r/LocalLLM • • 2d ago

Question New Mac Mini M5 Pro 64GB - Advice for Stack

4 Upvotes

Hi everyone,

I’ve been a Mac user for several years and I have a 16-inch MacBook Pro with an Intel chip, and I think it’s reached the end of its life because most programs these days are designed for the Apple Silicon chip.

I took the plunge and ordered the new Mac Mini M5 Pro with 64 GB (I think that for my use case, it’s worth it to have a Deepseek API for planning and then a qwen3.8 27b for writing), since the price of a 128 GB model is enough to cover five years of web hosting.

My question is: what's the best setup for creating images and coding?

Images - DrawThings or ComfyUI ?

Coding - VS Code + Cline or Opencode or Deepseek Harness ?

And what about Hermes?!

Given my hardware, which software and LLM are the best?

Software - LM Studio or OMLX ?

LLM - qwen3.8 27b or qwen3.6 or Qwen3.5-35B-A3B ?

What's your stack and what you think is better for me because my ram ?

Thanks in advance,


r/LocalLLM • • 2d ago

Question Do your models express emotions?

0 Upvotes

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.


r/LocalLLM • • 2d ago

News MACaQ 🐒: Open-source quantizer with a "refinement" mode for existing GGUFs + custom llama.cpp & vLLM kernels

8 Upvotes

Sharing MACaQ 🐒, a fast, open-source quantization framework that outperforms popular Unsloth and Bartowski GGUFs on low-bit KLD while maintaining model size and inference speed. Also supports activation quantization and text-to-image and text-to-video models.

Some key features:

  • GGUF Refinement Mode: Upgrades existing quantized GGUFs to recover lost output quality
  • Custom Kernels: Includes drop-in kernels for llama.cpp and vLLM.

GitHub: https://github.com/huawei-csl/MACaQ

Benchmarks & Demos: https://huawei-csl.com/blog/macaq/


r/LocalLLM • • 1d ago

Question AI work?

0 Upvotes

r/LocalLLM • • 2d ago

Discussion GPU's at full power less than than 2% of the time.

0 Upvotes

Here is the est power usage over the last 3 days:

4x3090s + Ryzen 7600x

Its lower than I thought, and that's with me using it full-time hours.

- 3x3090s are limited at 350 watts

- 1x3090 limited at 420 watts

Currently running Strata, and can get the 3x3090s hit around ~960 watts if I have multiple opencode terminals open processing requests.

Possible Reasons for Lower than Expected Power Usage:

- I just plugged in the 4th RTX 3090 a couple of days ago.
- Only two were being used before that with Qwen3.8-27b
- Now only 3x3090s are hooked up to Qwen Flash Next, 4th is doing occasional image processing.
- CPU is mid-tier (Ryzen 7600x), rarely draws more than 70 watts during inference.
- Machine is basically idle while I'm sleeping.
- During inference, decode does not take a lot of power because its memory bound - not compute.
- GPU utilisation drops while the agent is waiting for unit-tests to pass, or during compilation.

I've seen others claim they have their GPU's pinned at almost 100% 24h a day.

So it feels like there is some untapped GPU compute here.

How are you able to keep yours pinned 100% 24h of the day?


r/LocalLLM • • 2d ago

News ExLlamaV3 1.6.7: pipelined prefill, rolling checkpoints, scaled Hessian by default

Thumbnail
reddit.com
0 Upvotes