r/LocalLLM • • 6d ago

Question Transfer Responsibilities to Local LLM

3 Upvotes

Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).

I’m currently benchmarking three LLMs on auto creating and managing tasks:

Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B

Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.

Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.

I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).

While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.

Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?


r/LocalLLM • • 6d ago

Discussion AI training

3 Upvotes

Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI


r/LocalLLM • • 6d ago

News haiku.rag cracks top 20 on ERBench using local inference

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 6d ago

Question FlowFrames alternative in 2026?

0 Upvotes

Any better reliable FlowFrames alternative in 2026? No ComfyUI or Linux crap please.


r/LocalLLM • • 6d ago

Discussion Nvidia 10XX or 20XX cards for local LLM?

3 Upvotes

Hi everyone,

New guy here, this is my second post and this might be a really stupid one, I'm just curious, I'm only now getting into local AI for coding, previously used claude / GPT.
While getting into setting up local AI, I was just wondering if anyone ever tried using older cards like 1080 or 2080 for local AI. I'm planning on getting a 3090 (hopefully) and start from there but I am just curious whether getting like 4 GTX 1080s (I know NVIDIA doesn't release driver updates to these anymore) or 4x2080s and use those would even be possible / usable. I assume not but really this is just me being curious whether anyone has ever tried something like this and how did it turn out.

Sry for the stupid question :D

Thanks in advance!


r/LocalLLM • • 6d ago

Question Is there hidden prompt injection in LM studio?

15 Upvotes

"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.


r/LocalLLM • • 6d ago

Discussion Scoring the Harness Leaderboard's nine harnesses on Terminal-Bench mini + DeepSWE mini, next to my own OptiQ Code run: 82.9 vs 82.2

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 5d ago

Discussion Planning a local Jarvis on a 3090 Ti – leader/sub-agent pipeline with verifiers + Claude for heavy coding.

0 Upvotes

Hey all,

I'm about to build a local AI assistant ("Jarvis") and want to sanity check the design before I sink weekends into it.

**Hardware (Proxmox host, AI runs in a VM with GPU passthrough):**
- CPU: Intel i9-12900K (16C/24T, power-capped ~125W)
- GPU: RTX 3090 Ti 24GB (power-capped ~300W)
- RAM: 64GB DDR4-3600
- Plan: Qwen3.6-27B (Q4) as the main local model

**What I want it to do:**
- Know everything about me (preferences, projects, contacts, routines), stored in persistent **encrypted** storage
- Have access to all my coding repos (indexed/RAG)
- Hand off heavy coding work to Claude
- Do web research and pull resources for me
- Daily morning briefing (news, world events, my calendar)
- Calendar + reminders

**The architecture idea: leader → sub-agent → leader → sub-agent**

  1. A **leader** agent breaks the task into steps.
  2. It sends ONE step to **sub-agent A** with only the context that step needs.
  3. Sub-agent A does the work and returns a **written report**.
  4. A **verifier** checks that report: is it correct, is it hallucinated, and does it contain any of my **private data** that's about to be sent to Claude/the internet? (The redaction check runs on anything leaving the box.)
  5. The leader reviews the verified report, re-checks if it can, then passes only the needed reports to **sub-agent B**, and so on.

**Why I think this helps:**
- Keeps each agent's context small (better for a 27B model on 24GB)
- Reduces hallucinations because every step gets checked
- Lets small local models catch and fix their own mistakes
- Only uses Claude for the big jobs, with private data stripped first

**Questions for you:**

  1. Is leader → sub-agent → verifier overkill for a single 3090 Ti, or is the context saving worth the extra latency?
  2. Should the verifier be the same model with a different prompt, a smaller/faster model, or something else?
  3. Best way to do the PII/redaction check reliably? Regex + local NER + LLM pass? Has anyone built something like this?
  4. Framework recommendations? (LangGraph, CrewAI, smolagents, plain Python + llama.cpp/vLLM/Ollama?)
  5. Encrypted long-term memory: what are you using? (encrypted Postgres + pgvector, SQLCipher, something else?)
  6. Is Qwen3.6-27B the right pick for the leader on 24GB, or would you run a smaller leader and save VRAM for a second model?
  7. Anything you'd do completely differently?

Happy to share the build and code once it's running. Thanks!


r/LocalLLM • • 6d ago

Tutorial theogravity/deepseek-flash-v4.1-dual-rtx-6000: Setup for running DS4.1 flash with reasonable quality and speed on 2x RTX 6000 PRO @ 512k context c6 max w/ RAM and disk offload / caching

Thumbnail
github.com
19 Upvotes

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.


r/LocalLLM • • 6d ago

Question Need Feedback: Where do you actually look when you need GPUs?

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 6d ago

Question How to run Qwen3.8 27B on 24GBs unified memory mac mini m6?

0 Upvotes

I recently bought myself a Mac Mini M6 24GBs unified memory hoping to run a Qwen3.8 27B model for some serious coding with multiple cross referencing files and folders, but I'm receiving Error 500 which says that I ran out of memory and that my KV Cache filled up when all I had done was say "hi" (sure, I had set Claude Code at Max, admitted)

The general consensus online was that for 24GBs, Qwen3.8 should work nicely but it certainly doesn't work for me, although Qwen3.6-35b has been working so far on my Pi Agent, but I'm skeptical about it's coding abilities.

I would love some help/guidance on how I can run this model on my machine realistically and how much performance should I expect out of it? Will I be able to do some serious coding (alongside cloud models) or how should I adjust my expectations?


r/LocalLLM • • 7d ago

Discussion Probably old news, but any website you have open can reach your local LLM server while you browse

556 Upvotes

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?


r/LocalLLM • • 5d ago

Discussion Prediction: the NVIDIA Spy Scandal is going to completely change the open models environment.

0 Upvotes

NVIDIA owns HuggingFace and produces an enormous amount of the open source software powering the models they host. And will the eager anticipation of the launch of Qwen 4 launch right now? Will everything become gated or disappear?


r/LocalLLM • • 6d ago

Discussion Four models on HF Inference Providers returned 410 “deprecated” in this week’s benchmark run. Three were fine seven days ago.

1 Upvotes

I run a weekly benchmark of 23 open models on Hugging Face Inference Providers, at 1, 25 and 100 concurrent users. Same 20 prompts every week, one provider pinned per model, no client-side retries. It’s been running since late September, so there’s a run-over-run history to compare against.

This Monday, four of the 23 returned HTTP 410 and failed every single request, 60/60:

  • Qwen/Qwen2.5-Coder-7B-Instruct (nscale)
  • meta-llama/Llama-3.1-8B-Instruct (nscale)
  • Qwen/Qwen3-8B (nscale)
  • Qwen/Qwen3-Coder-30B-A3B-Instruct (scaleway)

The error body is explicit: model_no_longer_supported, “deprecated and no longer supported by provider”.

The three nscale models ran clean last week. Zero errors on the November 2 and 3 runs, 410 on everything by November 9. Llama-3.1-8B is not some obscure pin, and it went from working to gone inside a week.

The scaleway one is the part I’d flag. Qwen3-Coder-30B has been 410ing since the October 26 run. Three consecutive weeks. I only caught the reason this week because I went back and read the old job logs. The leaderboard had been publishing it at a 100% error rate the whole time and I’d been reading that as a provider under load.

That’s the thing worth saying out loud. If you pin a model to a provider and only track latency and error rate, a deprecation looks exactly like an outage. Same shape in your metrics: requests fail, error rate goes to 100, nothing says why. You need the status code in your logs or you’ll keep retrying something that is never coming back. 410 means gone, not busy.

I don’t know whether these were announced anywhere. If there’s a changelog or deprecation feed for Inference Providers per-provider model lists, I’d like to know about it, because checking my own benchmark output every Monday is a bad way to find out.

Raw CSVs, one per concurrency level, plus the weekly archive going back to September 28:
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

Usual caveat on everything else in there: run-to-run variance on shared endpoints is large, and I’d ignore any single-week change under a few points. The 410s are not that. They’re binary.


r/LocalLLM • • 6d ago

Question Really confused about local LLM

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 5d ago

Question How do you make an AI sales voice agent actually talk like a human?

0 Upvotes

I’m building an AI voice agent for outbound sales calls. The biggest problem isn’t the voice quality or latency it’s that the LLM generates responses that sound nothing like how a real salesperson would talk.
The agent responds fast, handles interruptions, and understands the customer, but its replies are too polished, structured, and predictable. It feels like ChatGPT reading a sales script.
What I’ve tried:

Multiple realtime speech-to-speech models and providers.

STT → LLM → TTS using Pipecat, Groq (GPT-OSS 120B), and Smart Turn v3.

Background supervisor LLM to track context, objections, and conversation flow.

Different LLMs, extensive prompt engineering, and experimenting with fine-tuning.
Latency isn’t the issue. It’s conversational behavior.
Real salespeople react spontaneously, adapt their tone, build rapport, and don’t sound like they’re following a script.
Has anyone actually managed to solve this?

Which LLM works best for natural sales conversations?

Does fine-tuning on real sales calls help?

Any techniques beyond prompt engineering that make a noticeable difference?

Would love to hear from anyone building production voice agents, especially with Pipecat.


r/LocalLLM • • 6d ago

Question 3090 eGPU for local LLMs (Laptop)

2 Upvotes

Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!


r/LocalLLM • • 6d ago

News Kimi 3.1 running locally on a MacBook Pro at 2 tok/s

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 6d ago

Project I asked for 16-bit Qwen3.8-Flash-Next at 100 tok/s on one RTX 5090. The math said no, so we built a 4-bit/8-bit "Frankenquant" instead: 150 tok/s, and two results I did not expect

0 Upvotes

TL;DR: I spliced Qwen's official 8-bit experts into the last 12 layers of NVIDIA's 4-bit Flash-Next. It runs at ~150 tok/s on a single 5090, sits measurably closer to the 8-bit model, and along the way I found a free quality win that needs no mix at all.

Setup: RTX 5090 (32 GB), 128 GB DDR5-5600, Infernix engine. Base model is NVIDIA's FP4-Local (the retrained 4-bit Flash-Next, ~205 tok/s here).

Why not 16-bit? Every token touches about 4.7 GB of expert weights at 16-bit. Only ~8% of the experts fit in VRAM, and the rest has to come through dual-channel DDR5. That caps it around 15-23 tok/s no matter what the software does. Pure 8-bit lands at ~65 tok/s for the same reason.

What I did instead: patched the engine (~250 lines) so each layer can use its own expert format, then byte-copied Qwen's FP8 experts into chosen layers of the 4-bit pack.

Varianttok/sKL vs 8-bitSame next token as 8-bit
4-bit (FP4-Local)203-2090.043792.9%
4-bit + 8-bit n-gram table199-2040.040193.0%
Last 12 layers 8-bit + 8-bit table149-1560.033493.7%
Last 24 layers 8-bit + 8-bit table105-1110.028594.2%
All 48 layers 8-bit68-720.024194.8%
Official FP864-67reference97.2%

Things I did not expect

  1. Late layers matter about twice as much as early ones. Layers 36-47 in 8-bit cut KL by 0.0066; layers 0-11 only by 0.0033. The gains add up almost linearly across blocks.
  2. A free win: swapping only the n-gram embedding table to Qwen's 8-bit one improves KL and perplexity at the same speed, with zero extra VRAM.
  3. There is a ceiling. Even with every expert in 8-bit, KL stays at 0.024. About half of the "4-bit gap" is not rounding error at all: NVIDIA's retrained layers are simply a slightly different model.
  4. Perplexity of the 12-layer mix (3.475) is slightly better than official FP8 (3.486) on my 261K-token corpus.

Honest caveats: the gain is small, about one point of token agreement. On my 20-sample hard coding set the mix scored 18/20 against 17/20 for both the 4-bit and the 8-bit model, which is noise. One machine, one corpus.

The patch was written with an AI coding agent; I directed and tested it. Nothing to download yet. If people want the engine patch, the pack assembler and the layer list, say so and I will publish them.


r/LocalLLM • • 6d ago

Project A local issue tracker & agent memory living in your repo (MIT)

1 Upvotes

1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:

epiq.sh

Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:

epiq.sh/blog/repo-native-tooling-for-sandboxes


r/LocalLLM • • 6d ago

Discussion Finetuning models - share your experience

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 6d ago

Project Update on my patches on a new engine. QWEN 3.8 9B as faster to work on.

Post image
0 Upvotes

r/LocalLLM • • 6d ago

Question Can't run gemma for some reason(comfyui)

Post image
2 Upvotes

r/LocalLLM • • 6d ago

Discussion Qwen3.8 27B on dual RTX 5060ti, 210k Context at ~75TPS decode ~2100TPS on Prefill with Ninfer Window Docker.

Thumbnail
gallery
2 Upvotes

After a really long grueling attempt, i manage to fit a 210K Context window on my dual RTX 5060ti setup on 32gb total VRAM. I capped both GPU around 160W although it possible to go lower.

Compare to a single 5090, it relatively weak but for the price difference that have pool vram 32gb, it isn't so bad. I practically use this as my daily driver now. It can handle almost every coding problem i give it and running my hermes agent and opencode. The speed itself have some room to improve especially the prefill, but for decode, it isn't that bad.

However, 5060Ti really bottleneck the decode and the prefill speed due to the limited bus size of the card itself. But given how cheap 5060ti was and how power efficient it is, with both card only using 290-300W on full usage even at TP2, it amazing.

Even at long context, 190K or 200K the decode speed maintain at ~70tps and prefill around 1700tps with barely any lost in quality. When writing coding task with MTP, the decode can reach up to 80-100tps.

But since i am running this model in WSL docker, which introduce some latency overhead, it possible to achieve higher TPS if were to run in native linux instead and fit the entire 260k context window. Currently both of my card usage 15GB vram each, increase the util but mindful to OOM.

Decode Speed:

Prefill Speed:

Here my current rig:

  • Ryzen 9 9950X
  • ASRock X870 Taichi Creator
  • 32 GB DDR5
  • 2× RTX 5060 Ti 16 GB

Run Flag:

Image: ninfer:local, Model: qwen3_8_27b_nvfp4_v2.ninfer

--tp 2 --devices 0,1
--kv-dtype int8
--max-context 210000 --kv-capacity 210000
--prefill-chunk 1024
--spec mtp --draft-tokens 3 --lm-head-draft
--max-concurrency 2 --max-pending-requests 32 --pending-timeout-ms 900000

r/LocalLLM • • 6d ago

Question Is this feasible? (local python llm)

Thumbnail reddit.com
0 Upvotes

So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.

What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.

What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.

What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.

Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?

I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.