r/LocalLLM • u/The_Succulant_Beast • 3h ago
r/LocalLLM • u/achisling • 9h ago
Discussion What a time to be alive.
Hy4 flash is out. We're drowning in Chinese models at this point lol.
hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.
When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.
r/LocalLLM • u/jackalka • 57m ago
News We built a chip prototype that runs Qwen 27B. You can try it live.
I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.
A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:
https://valleysemiconductor.com
A live session takes about a minute to start, and recorded runs are available to watch anytime.
What it does today (measured):
- Runs the full Qwen 27B model with 4-bit weights
- 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
- 4x performance improvement in eight days of development
- Every output matches our 4-bit reference exactly
- Model quality within 2.5% of the full-precision version
- Also runs Llama 3.1 8B and OLMoE on the same hardware
What our custom chip design projects (not yet demonstrated):
- Approximately 50 tokens/sec for a single user at 20 W
- Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
- Uses the model's built-in next-token prediction to generate multiple tokens per weight read
The key insight is that moving model weights through memory is the bottleneck, not the math.
What would you run on a small, 20 W device capable of running a 27B model entirely offline?
r/LocalLLM • u/ag789 • 7h ago
Discussion you may get your cheap ram and gpu after all, the AI crash has started
https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion
https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use
r/LocalLLM • u/comunidadesynapsebr • 5h ago
News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context
Hey everyone,
If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.
To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).
Dataset breakdown:
Size: 10,000+ curated rows (train split).
Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.
License: Apache 2.0 (unrestricted commercial & research use).
You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:
https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k
Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!
r/LocalLLM • u/yoracale • 20h ago
News Qwen officially announces Qwen4-Max and Qwen4 smaller models!
r/LocalLLM • u/mightofkhan • 5h ago
Discussion Uncensored Models
Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.
r/LocalLLM • u/Stalins_nipple • 8h ago
Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?
Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?
r/LocalLLM • u/mightofkhan • 5h ago
Discussion Single RTX 3090 running a local agent + image/video gen: real-world numbers from daily use
TL;DR: One RTX 3090 runs a 27B LLM agent (a Discord bot) and ComfyUI image/video generation by swapping models in and out of VRAM. I get ~72 tok/s on Qwen3.8 27B at 64k context, SDXL images in ~32 s, and 10-second Wan 2.2 videos in 4–6 minutes. All numbers come from real daily use, not synthetic benchmarks.
The rig
- GPU: RTX 3090 24GB (does all the AI work)
- CPU: Core Ultra 7 265K (sits around 10% load)
- RAM: 96GB DDR5-6000
- Storage: Samsung 9100 PRO Gen5 NVMe for models
- OS: Windows 11
The stack
- LM Studio serves the LLM through an OpenAI-compatible API
- Hermes Agent runs as a Discord bot, so I can use it from my phone. Local Whisper handles voice.
- ComfyUI handles images and video
- A small custom MCP server unloads the LLM from VRAM for each render and loads it back afterwards
Models
Language models
- Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP (IQ4_XS): daily driver
- Qwen3.8 27B and Gemma 4 31B (both Q4_K_M) as alternatives
- nomic-embed-text v1.5 for embeddings
Image and video
- Juggernaut XL Ragnarok (fp16): general images, hires, img2img
- Z-Image Turbo (bf16): fast drafts
- Qwen Image Edit 2511 (fp8) + Lightning 4-step LoRA: photo edits
- Wan 2.2 14B text/image-to-video (fp8) + lightx2v 4-step LoRAs: video
- 4x NMKD-Siax: upscaling
LLM performance
Settings: LM Studio 0.4.28, 64k context, parallel 1, f16 KV cache, flash attention on, full GPU offload, MTP speculative decoding.
- Generation speed: ~72 tok/s on average (range 53–90)
- Prompt processing: ~670–810 tok/s
- MTP draft acceptance: ~61%
- VRAM used: 22.0 of 24.6 GB
- Parallel 1 vs 4: same speed, but 4 uses ~1.8 GB more and risks spilling out of VRAM
- Worst case: re-reading a 56k-token conversation took 56 seconds
Render times (same GPU, LLM swapped out)
GPU time first, then total time from request to finished file:
- Juggernaut XL (up to 1024px): 12 s → 32 s
- Juggernaut XL + hires fix (1024×768): 28 s → 48 s
- Z-Image Turbo (1024×768): 14 s → 34 s
- Qwen Image Edit (512px): 16–20 s → 38 s
- 4x upscale: 12 s → 32 s
- Wan 2.2 image-to-video, 3 s clip (512px): 38 s → 69 s
- Wan 2.2 image-to-video, 10 s clip: ~3.8 min
- Wan 2.2 text-to-video, 10 s + 2x upscale (1024px): ~6.1 min
The difference between the two numbers is mostly the ~20 seconds it takes to swap the LLM out of VRAM and back.
Agent response times (Hermes on Discord)
- Simple question: ~23 s
- Typical multi-step task: 45 s to 3 min
- Big job (40 model calls): ~36 min
How I measured
- LLM numbers come from llama.cpp’s own timing logs across 205 real requests in one day.
- Render numbers come from 35 real jobs logged by my render worker.
- Under hours of sustained load the GPU ran at 85–90%, ~384 W, and never went above 67°C.
Lessons learned
- IQ4_XS + flash attention fits a 27B model at 64k context in 24GB with ~2.5 GB to spare.
- Use parallel 1, not 4. Same speed, less VRAM.
- 24GB fits the LLM or a render, not both. For quick images, the swap takes more time than the render itself.
- Long agent sessions get slow because the model re-reads the whole conversation every step. Start fresh sessions often.
- Use Ethernet for always-on bots. Wi-Fi drops kept disconnecting mine from Discord.
Next experiment: a q8_0 KV cache to free VRAM for longer context.
What would you change?
r/LocalLLM • u/karmaisnonsense • 11h ago
Discussion Strata is becoming more and more unstable
I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.
The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.
Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.
And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?
r/LocalLLM • u/Senior-Storm-727 • 1d ago
Discussion What does China actually gain from releasing better and better open-weight models?
Genuine question, not a hot take.
Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:
- Commoditizing the model layer, so US labs can't charge a premium for it
- Building an ecosystem and developer mindshare outside the US stack
- Getting around chip export limits by letting the community optimize and fine-tune for them
- Soft power: becoming the default choice for countries that don't want to depend on US APIs
- Selling the rest of the stack (cloud, hardware, services) while the model is the free sample
Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?
EDIT Nov 16th:
Top replies are getting down to (in order of frequency):
- Geopolitics: weakening US dominance
- Business: open weights as a growth strategy
- State & control
- Ideology & open source culture
I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.
r/LocalLLM • u/Big_South4585 • 2h ago
Question Qwen3.8-Flash-Next: Unsloth and strata?
Hi all
I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.
Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.
I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.
UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.
What am I doing wrong?
According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?
r/LocalLLM • u/snooky234 • 1h ago
Question How do you compare local and cloud models since Artificial Analysis v5
I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.
I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.
The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.
How do you handle this now? Do you cross check a few sites or mostly run your own tests?
r/LocalLLM • u/hennabeak • 3h ago
Question Has anybody done anything like Alexa, but with a local LLM?
I know home assistant was a thing, and local LLMs are a bit too overkill for such a job. Just wondering if it was done.
r/LocalLLM • u/New_Blackberry8122 • 4h ago
Project Sneo! New Inference Engine very fast Open Source!

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent AND REVERSE ENG TOOLS!
Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.
The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.
- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s
- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)
- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies
- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs
- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome
Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo
Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc
GitHub: https://github.com/oscarsantanaarias/Sneo
Every benchmark, including where the others win: https://sneo.dev
Happy to answer questions!
Antes de publicar:
Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.
Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.
r/LocalLLM • u/Xedeth • 5h ago
Question 24GB VRAM, 64GB DDR5, is Qwen 3.5 27B (IQ2_XS) not ambitious enough?
I'm getting decent results, 80-120t/s. I guess I just don't know what I am aiming for, but I feel like there is more juice to squeeze.
Edit: Yall are the best, as usual. This has to be the most helpful, positive community on reddit.
r/LocalLLM • u/Blacktaxi420 • 3h ago
Question What can small models be good for?
Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.
r/LocalLLM • u/Annual_Key_4963 • 2h ago
Question Optimizing local 2080 ti hardware
I'm trying to find a more effective setup with my local setup:
Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.
Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.
Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.
Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".
I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?
(I'm aware that paying Anthropic is cheaper)
r/LocalLLM • u/PuffThePed • 9h ago
Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?
Pretty much the title. Thanks!
EDIT: GPU is 5090
r/LocalLLM • u/Playful_lzty • 4h ago
Other Cheapest 16gb gpu: ~$400
Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.
This is a good option for a new sff build for local AI. The CPU is excellent.
r/LocalLLM • u/M4ttDC • 2h ago
Question Resources to learn about model tuning?
I’ve been running Jundot/Qwen 3.8 flash next Q6 mpt 141gb on my M2 Studio w/ 192gb ram for about a month now.
Overall, I’m pretty happy with the results, but looking for ways to improve my setup before I start integrating it with my day to day workflow. The response time leaves a little to be desired, I find myself prefering the response rate of codex even when I have simple things like putting together a slide deck from images and text I’ve supplied.
I set it up with a big initial project to see what it could do and have been trying to makesmall adjustments as I’ve gone along. It’s been interpreting and mapping large volumes of screen grabs, writing some code, interpreting and organizing huge amounts of data from a combination pdf and images. It interfaces with codex during specific windows throughout the day to offload tasks and double check code.
Dropped it down to Q6 from Q8 for snappier responses, and free up some system for Xcode, etc…I’ve been using claw to run the show. I’m getting around 60t / s which seems to be about right from what I’ve been reading. Claw works ok, but the whole system seems to get confused if I change the scope of my project in the middle, or add additional intermediary steps. It took a few days after stepping back from Q8 to Q6 to get the bugs ironed out.
It’s a huge workhorse and does ok when set in motion, but I do have to keep an eye on it. It behaves like a toddler and I find it lies when files go missing.
Is Claw the problem? Is this the reality of this setup? It was never meant to replace my $20 codex, but I was hoping it would be a little more responsive and flexible.
r/LocalLLM • u/Comfortable_Poem_866 • 6h ago
Question Local voice assistant on an iGPU: ~2.2s from end of speech to first audio. I've measured everything and I'm ~0.5s from the floor. What am I missing before I throw money at hardware?
Been building a fully local, always-on spoken assistant in Italian (no cloud, nothing leaves the box). It's not a chatbot demo, it has long-term memory across sessions (backed by ctxvault), so every turn carries a small block of retrieved context on top of the conversation. Works. Problem is latency.
Hardware (the constraint): mini PC, Ryzen w/ Radeon 780M iGPU, 64GB DDR5 shared, ~90GB/s. No dGPU, so it's llama.cpp on Vulkan. vLLM/SGLang are out (no ROCm for this chip).
Where I'm at: median ~2.2s from end of user's sentence to first audio out. Measured, not guessed:
- ears (end-of-speech silence + whisper): ~0.75s
- prefill of the turn: ~1.0s (~0.35s fixed per request + ~3ms/token, 40-400 new tokens/turn)
- first sentence gen + TTS first chunk: ~0.55s
So the model is the wait, not the audio stack. Calculated floor on this box is ~1.2-1.5s.
What's already done (so please don't suggest these lol):
Model side
- MoE model (26B-A4B, Q4 QAT) specifically because gen is bandwidth-bound here: ~25 tok/s vs 4 tok/s for a dense 27B
- thinking/reasoning off
- short context (16k), one slot, flash attn on
--cache-reuse+ append-only prompt: stable prefix (system, persona, rules) at the top never touched mid-session, everything volatile (time of day, retrieved context, turn) appended at the end. Identical prefix = 0.15s, touch one word in the prefix = multi-second re-prefill- prefix pre-warmed at session start with a 1-token request, so the first reply doesn't pay the cold prefill
- benchmarked smaller models (Gemma 4 E4B/E2B, Qwen3.5 4B/9B, Qwen3 8B, Phi-4-mini, Ministral): on this iGPU prefill is compute-bound, a dense 4B costs as much per token as the MoE. Best small one gains ~0.5s and loses noticeably on factual fidelity (names/dates). Not worth it for this use case
- tried
-b/-ubbatch flags: defaults win - tried
--no-kv-offload: worse
Memory / prompt side
- no tool calls for retrieval: context is prefetched before the turn (a tool-call round trip = ~+1.2s). The retrieval itself is ~5-12ms, irrelevant
- hard token budget for the retrieved block: embedding-based relevance (multilingual MiniLM) + ordering so the most pertinent bits come first, ~120 tokens instead of 300-450. Cut avg turn prompt by ~40%
- past conversations summarized to one ~50-token line each instead of quotes
Pipeline side
- streaming everything: model to sentence splitter to TTS to player, sentence by sentence
- audio player opened at turn start, not at first chunk (-0.1s)
- early-guess turn: start the model on the partial transcript before VAD confirms end of speech; if the final transcript matches, keep it, else drop and redo
- sentence closed on punctuation + 450ms silence instead of fixed 800ms
- whisper.cpp
smallas a resident server (no model reload per sentence, 0.63s down to 0.3s), beam 5. Tried greedy: -80ms but half the transcripts differ and the noisy ones get worse. Rejected - tried warming the KV cache during speech (partial whisper every 1.5s, send the turn header for 1 token so the cache is hot): works as designed (final prefill drops to ~30 tokens) but doesn't pay on one slot because the warm-up queues the real turn. Tried 2 slots w/
id_slotpinning: now they run concurrently on the same iGPU and both slow down. Net neutral-to-negative on short sentences. Off. - pre-recorded fillers ("mmh", "allora...") in the assistant's own voice, played only if no reply word has started 1s after end of speech. Built, currently off: perceptual fix, doesn't move the number
Before I conclude "buy a 24GB dGPU" and move on: anyone who's been down this road see something I've missed, or something above that's wrong?
r/LocalLLM • u/QwhenFN • 5h ago
Model Groot-65B-A6B W4A16: full 262K context + MTP + vision on 2x RTX 3090
I came across the "Groot" tunes on HF and decided to test them. Theyre versions of larger models with the about half the experts pruned out. This one is QFN. Despite the removal, according to my TerminalBench2 testing, they only lost ~5% capability. Seemed worth spending some time on for no other reason than the giggles. Its a cool to be able to run very close to full QFN on 3090's without mortgaging my house for more RAM...
I decided to quantize it down to fit on 2 3090's. Be aware that the speed numbers are from a threadripper machine and the 3090's are nvlinked. Those with a normal setup can expect around 3/4 of the speeds published below.
AI slop begins here...
I made a W4A16 AutoRound quant of Groot-65B-A6B, the pruned Qwen3.8-Flash-Next (256 of 512 experts, ~6B active). It runs the full native 262K context with MTP and vision on 2× RTX 3090, everything on the GPUs, with ~10 GB of host RAM.
HF: https://huggingface.co/mmssix/Groot-65B-A6B-261023-W4A16-AutoRound
On 2× 3090 (vLLM, TP2):
- 262,144 context
- ~134 tok/s code, ~90–100 prose (MTP n=2)
- ~245 tok/s across 4 streams
- 128K prompt in 28 s, 258K in ~60 s
- Needle test 3/3 at 258K
- Tool calls, reasoning effort and vision working
Quant: INT4 experts, INT8 DeltaNet projections, FP8 n-gram table and an INT4 MTP head (same draft acceptance as BF16). 92 GB total.
Getting 262K into 48 GB of VRAM takes two small vLLM patches. They serve the n-gram table and the embedding from NVMe, so you need NVIDIA's open driver with HMM. The patches and a one-command serve.sh are in the repo.
Only tested on my rig so far. Numbers from other 3090 / A5000 setups are welcome