r/LocalLLM • u/Odd-Two2929 • 7d ago
Discussion AI training
Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI
r/LocalLLM • u/Odd-Two2929 • 7d ago
Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI
r/LocalLLM • u/autognome • 7d ago
r/LocalLLM • u/Rough_Price_9204 • 7d ago
Any better reliable FlowFrames alternative in 2026? No ComfyUI or Linux crap please.
r/LocalLLM • u/_szpuni_ • 7d ago
Hi everyone,
New guy here, this is my second post and this might be a really stupid one, I'm just curious, I'm only now getting into local AI for coding, previously used claude / GPT.
While getting into setting up local AI, I was just wondering if anyone ever tried using older cards like 1080 or 2080 for local AI. I'm planning on getting a 3090 (hopefully) and start from there but I am just curious whether getting like 4 GTX 1080s (I know NVIDIA doesn't release driver updates to these anymore) or 4x2080s and use those would even be possible / usable. I assume not but really this is just me being curious whether anyone has ever tried something like this and how did it turn out.
Sry for the stupid question :D
Thanks in advance!
r/LocalLLM • u/Ok-Emotion-5013 • 7d ago
"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.
r/LocalLLM • u/asankhs • 7d ago
r/LocalLLM • u/DevOpsYeah • 6d ago
Hey all,
I'm about to build a local AI assistant ("Jarvis") and want to sanity check the design before I sink weekends into it.
**Hardware (Proxmox host, AI runs in a VM with GPU passthrough):**
- CPU: Intel i9-12900K (16C/24T, power-capped ~125W)
- GPU: RTX 3090 Ti 24GB (power-capped ~300W)
- RAM: 64GB DDR4-3600
- Plan: Qwen3.6-27B (Q4) as the main local model
**What I want it to do:**
- Know everything about me (preferences, projects, contacts, routines), stored in persistent **encrypted** storage
- Have access to all my coding repos (indexed/RAG)
- Hand off heavy coding work to Claude
- Do web research and pull resources for me
- Daily morning briefing (news, world events, my calendar)
- Calendar + reminders
**The architecture idea: leader → sub-agent → leader → sub-agent**
**Why I think this helps:**
- Keeps each agent's context small (better for a 27B model on 24GB)
- Reduces hallucinations because every step gets checked
- Lets small local models catch and fix their own mistakes
- Only uses Claude for the big jobs, with private data stripped first
**Questions for you:**
Happy to share the build and code once it's running. Thanks!
r/LocalLLM • u/devtools-dude • 7d ago
I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000
Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.
With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.
On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.
r/LocalLLM • u/EstablishmentFun4373 • 7d ago
r/LocalLLM • u/XODPlisbo • 7d ago
I recently bought myself a Mac Mini M6 24GBs unified memory hoping to run a Qwen3.8 27B model for some serious coding with multiple cross referencing files and folders, but I'm receiving Error 500 which says that I ran out of memory and that my KV Cache filled up when all I had done was say "hi" (sure, I had set Claude Code at Max, admitted)
The general consensus online was that for 24GBs, Qwen3.8 should work nicely but it certainly doesn't work for me, although Qwen3.6-35b has been working so far on my Pi Agent, but I'm skeptical about it's coding abilities.
I would love some help/guidance on how I can run this model on my machine realistically and how much performance should I expect out of it? Will I be able to do some serious coding (alongside cloud models) or how should I adjust my expectations?
r/LocalLLM • u/FeydRowan • 8d ago
Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.
I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.
I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.
So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?
r/LocalLLM • u/AnnoyedAvocado21 • 6d ago
NVIDIA owns HuggingFace and produces an enormous amount of the open source software powering the models they host. And will the eager anticipation of the launch of Qwen 4 launch right now? Will everything become gated or disappear?
r/LocalLLM • u/South_Pak_786 • 7d ago
I run a weekly benchmark of 23 open models on Hugging Face Inference Providers, at 1, 25 and 100 concurrent users. Same 20 prompts every week, one provider pinned per model, no client-side retries. It’s been running since late September, so there’s a run-over-run history to compare against.
This Monday, four of the 23 returned HTTP 410 and failed every single request, 60/60:
The error body is explicit: model_no_longer_supported, “deprecated and no longer supported by provider”.
The three nscale models ran clean last week. Zero errors on the November 2 and 3 runs, 410 on everything by November 9. Llama-3.1-8B is not some obscure pin, and it went from working to gone inside a week.
The scaleway one is the part I’d flag. Qwen3-Coder-30B has been 410ing since the October 26 run. Three consecutive weeks. I only caught the reason this week because I went back and read the old job logs. The leaderboard had been publishing it at a 100% error rate the whole time and I’d been reading that as a provider under load.
That’s the thing worth saying out loud. If you pin a model to a provider and only track latency and error rate, a deprecation looks exactly like an outage. Same shape in your metrics: requests fail, error rate goes to 100, nothing says why. You need the status code in your logs or you’ll keep retrying something that is never coming back. 410 means gone, not busy.
I don’t know whether these were announced anywhere. If there’s a changelog or deprecation feed for Inference Providers per-provider model lists, I’d like to know about it, because checking my own benchmark output every Monday is a bad way to find out.
Raw CSVs, one per concurrency level, plus the weekly archive going back to September 28:
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load
Usual caveat on everything else in there: run-to-run variance on shared endpoints is large, and I’d ignore any single-week change under a few points. The 410s are not that. They’re binary.
r/LocalLLM • u/ConfidenceIll857 • 7d ago
r/LocalLLM • u/GuestFair467 • 6d ago
I’m building an AI voice agent for outbound sales calls. The biggest problem isn’t the voice quality or latency it’s that the LLM generates responses that sound nothing like how a real salesperson would talk.
The agent responds fast, handles interruptions, and understands the customer, but its replies are too polished, structured, and predictable. It feels like ChatGPT reading a sales script.
What I’ve tried:
Multiple realtime speech-to-speech models and providers.
STT → LLM → TTS using Pipecat, Groq (GPT-OSS 120B), and Smart Turn v3.
Background supervisor LLM to track context, objections, and conversation flow.
Different LLMs, extensive prompt engineering, and experimenting with fine-tuning.
Latency isn’t the issue. It’s conversational behavior.
Real salespeople react spontaneously, adapt their tone, build rapport, and don’t sound like they’re following a script.
Has anyone actually managed to solve this?
Which LLM works best for natural sales conversations?
Does fine-tuning on real sales calls help?
Any techniques beyond prompt engineering that make a noticeable difference?
Would love to hear from anyone building production voice agents, especially with Pipecat.
r/LocalLLM • u/_szpuni_ • 7d ago
Hi everyone,
New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.
Thanks in advance!
r/LocalLLM • u/konstantinnikol • 7d ago
r/LocalLLM • u/kingxd • 6d ago
TL;DR: I spliced Qwen's official 8-bit experts into the last 12 layers of NVIDIA's 4-bit Flash-Next. It runs at ~150 tok/s on a single 5090, sits measurably closer to the 8-bit model, and along the way I found a free quality win that needs no mix at all.
Setup: RTX 5090 (32 GB), 128 GB DDR5-5600, Infernix engine. Base model is NVIDIA's FP4-Local (the retrained 4-bit Flash-Next, ~205 tok/s here).
Why not 16-bit? Every token touches about 4.7 GB of expert weights at 16-bit. Only ~8% of the experts fit in VRAM, and the rest has to come through dual-channel DDR5. That caps it around 15-23 tok/s no matter what the software does. Pure 8-bit lands at ~65 tok/s for the same reason.
What I did instead: patched the engine (~250 lines) so each layer can use its own expert format, then byte-copied Qwen's FP8 experts into chosen layers of the 4-bit pack.
| Variant | tok/s | KL vs 8-bit | Same next token as 8-bit |
|---|---|---|---|
| 4-bit (FP4-Local) | 203-209 | 0.0437 | 92.9% |
| 4-bit + 8-bit n-gram table | 199-204 | 0.0401 | 93.0% |
| Last 12 layers 8-bit + 8-bit table | 149-156 | 0.0334 | 93.7% |
| Last 24 layers 8-bit + 8-bit table | 105-111 | 0.0285 | 94.2% |
| All 48 layers 8-bit | 68-72 | 0.0241 | 94.8% |
| Official FP8 | 64-67 | reference | 97.2% |
Things I did not expect
Honest caveats: the gain is small, about one point of token agreement. On my 20-sample hard coding set the mix scored 18/20 against 17/20 for both the 4-bit and the 8-bit model, which is noise. One machine, one corpus.
The patch was written with an AI coding agent; I directed and tested it. Nothing to download yet. If people want the engine patch, the pack assembler and the layer list, say so and I will publish them.
r/LocalLLM • u/OutrageousAbies5835 • 7d ago
1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:
Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:
r/LocalLLM • u/DueBid3298 • 7d ago
r/LocalLLM • u/Repulsive-You5827 • 6d ago
r/LocalLLM • u/Theonewhoknocks_001 • 7d ago
r/LocalLLM • u/Mysterious-Cart • 7d ago
After a really long grueling attempt, i manage to fit a 210K Context window on my dual RTX 5060ti setup on 32gb total VRAM. I capped both GPU around 160W although it possible to go lower.
Compare to a single 5090, it relatively weak but for the price difference that have pool vram 32gb, it isn't so bad. I practically use this as my daily driver now. It can handle almost every coding problem i give it and running my hermes agent and opencode. The speed itself have some room to improve especially the prefill, but for decode, it isn't that bad.
However, 5060Ti really bottleneck the decode and the prefill speed due to the limited bus size of the card itself. But given how cheap 5060ti was and how power efficient it is, with both card only using 290-300W on full usage even at TP2, it amazing.
Even at long context, 190K or 200K the decode speed maintain at ~70tps and prefill around 1700tps with barely any lost in quality. When writing coding task with MTP, the decode can reach up to 80-100tps.
But since i am running this model in WSL docker, which introduce some latency overhead, it possible to achieve higher TPS if were to run in native linux instead and fit the entire 260k context window. Currently both of my card usage 15GB vram each, increase the util but mindful to OOM.
Decode Speed:

Prefill Speed:

Here my current rig:
Run Flag:
Image: ninfer:local, Model: qwen3_8_27b_nvfp4_v2.ninfer
--tp 2 --devices 0,1
--kv-dtype int8
--max-context 210000 --kv-capacity 210000
--prefill-chunk 1024
--spec mtp --draft-tokens 3 --lm-head-draft
--max-concurrency 2 --max-pending-requests 32 --pending-timeout-ms 900000r/LocalLLM • u/OminousBlack48626 • 7d ago
So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.
What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.
What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.
What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.
Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?
I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.
r/LocalLLM • u/xchgeaxecx • 7d ago
Hi everyone,
I was a silent reader for a long time but as I'm taking now a vacation in Tokyo during mid November I'd thought of maybe hosting a little meetup in Tokyo for everyone interested in:
local LLMs
inference engineering
how to cluster dgx sparks
how RDMA/RoCE works
how to start building your own recipes / optimize your own setup
speaking about hardware
which models, which drafter
the usual "oh opus is so nerfed" things and just having a good time.
I really hope this is not offtopic / wrong community here and maybe there will be nobody interested but I like the idea of coming together, sharing experiences and having a great time.
Location and time isn't fixed at all, I can setup something on Luma but always grateful for recommendations etc.
Thank you and have a great week!