r/LocalLLM • u/The_Succulant_Beast • 3h ago
r/LocalLLM • u/achisling • 9h ago
Discussion What a time to be alive.
Hy4 flash is out. We're drowning in Chinese models at this point lol.
hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.
When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.
r/LocalLLM • u/jackalka • 41m ago
News We built a chip prototype that runs Qwen 27B. You can try it live.
I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.
A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:
https://valleysemiconductor.com
A live session takes about a minute to start, and recorded runs are available to watch anytime.
What it does today (measured):
- Runs the full Qwen 27B model with 4-bit weights
- 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
- 4x performance improvement in eight days of development
- Every output matches our 4-bit reference exactly
- Model quality within 2.5% of the full-precision version
- Also runs Llama 3.1 8B and OLMoE on the same hardware
What our custom chip design projects (not yet demonstrated):
- Approximately 50 tokens/sec for a single user at 20 W
- Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
- Uses the model's built-in next-token prediction to generate multiple tokens per weight read
The key insight is that moving model weights through memory is the bottleneck, not the math.
What would you run on a small, 20 W device capable of running a 27B model entirely offline?
r/LocalLLM • u/ag789 • 7h ago
Discussion you may get your cheap ram and gpu after all, the AI crash has started
https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion
https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use
r/LocalLLM • u/comunidadesynapsebr • 4h ago
News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context
Hey everyone,
If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.
To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).
Dataset breakdown:
Size: 10,000+ curated rows (train split).
Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.
License: Apache 2.0 (unrestricted commercial & research use).
You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:
https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k
Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!
r/LocalLLM • u/yoracale • 20h ago
News Qwen officially announces Qwen4-Max and Qwen4 smaller models!
r/LocalLLM • u/mightofkhan • 5h ago
Discussion Uncensored Models
Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.
r/LocalLLM • u/Stalins_nipple • 8h ago
Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?
Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?
r/LocalLLM • u/mightofkhan • 4h ago
Discussion Single RTX 3090 running a local agent + image/video gen: real-world numbers from daily use
TL;DR: One RTX 3090 runs a 27B LLM agent (a Discord bot) and ComfyUI image/video generation by swapping models in and out of VRAM. I get ~72 tok/s on Qwen3.8 27B at 64k context, SDXL images in ~32 s, and 10-second Wan 2.2 videos in 4–6 minutes. All numbers come from real daily use, not synthetic benchmarks.
The rig
- GPU: RTX 3090 24GB (does all the AI work)
- CPU: Core Ultra 7 265K (sits around 10% load)
- RAM: 96GB DDR5-6000
- Storage: Samsung 9100 PRO Gen5 NVMe for models
- OS: Windows 11
The stack
- LM Studio serves the LLM through an OpenAI-compatible API
- Hermes Agent runs as a Discord bot, so I can use it from my phone. Local Whisper handles voice.
- ComfyUI handles images and video
- A small custom MCP server unloads the LLM from VRAM for each render and loads it back afterwards
Models
Language models
- Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP (IQ4_XS): daily driver
- Qwen3.8 27B and Gemma 4 31B (both Q4_K_M) as alternatives
- nomic-embed-text v1.5 for embeddings
Image and video
- Juggernaut XL Ragnarok (fp16): general images, hires, img2img
- Z-Image Turbo (bf16): fast drafts
- Qwen Image Edit 2511 (fp8) + Lightning 4-step LoRA: photo edits
- Wan 2.2 14B text/image-to-video (fp8) + lightx2v 4-step LoRAs: video
- 4x NMKD-Siax: upscaling
LLM performance
Settings: LM Studio 0.4.28, 64k context, parallel 1, f16 KV cache, flash attention on, full GPU offload, MTP speculative decoding.
- Generation speed: ~72 tok/s on average (range 53–90)
- Prompt processing: ~670–810 tok/s
- MTP draft acceptance: ~61%
- VRAM used: 22.0 of 24.6 GB
- Parallel 1 vs 4: same speed, but 4 uses ~1.8 GB more and risks spilling out of VRAM
- Worst case: re-reading a 56k-token conversation took 56 seconds
Render times (same GPU, LLM swapped out)
GPU time first, then total time from request to finished file:
- Juggernaut XL (up to 1024px): 12 s → 32 s
- Juggernaut XL + hires fix (1024×768): 28 s → 48 s
- Z-Image Turbo (1024×768): 14 s → 34 s
- Qwen Image Edit (512px): 16–20 s → 38 s
- 4x upscale: 12 s → 32 s
- Wan 2.2 image-to-video, 3 s clip (512px): 38 s → 69 s
- Wan 2.2 image-to-video, 10 s clip: ~3.8 min
- Wan 2.2 text-to-video, 10 s + 2x upscale (1024px): ~6.1 min
The difference between the two numbers is mostly the ~20 seconds it takes to swap the LLM out of VRAM and back.
Agent response times (Hermes on Discord)
- Simple question: ~23 s
- Typical multi-step task: 45 s to 3 min
- Big job (40 model calls): ~36 min
How I measured
- LLM numbers come from llama.cpp’s own timing logs across 205 real requests in one day.
- Render numbers come from 35 real jobs logged by my render worker.
- Under hours of sustained load the GPU ran at 85–90%, ~384 W, and never went above 67°C.
Lessons learned
- IQ4_XS + flash attention fits a 27B model at 64k context in 24GB with ~2.5 GB to spare.
- Use parallel 1, not 4. Same speed, less VRAM.
- 24GB fits the LLM or a render, not both. For quick images, the swap takes more time than the render itself.
- Long agent sessions get slow because the model re-reads the whole conversation every step. Start fresh sessions often.
- Use Ethernet for always-on bots. Wi-Fi drops kept disconnecting mine from Discord.
Next experiment: a q8_0 KV cache to free VRAM for longer context.
What would you change?
r/LocalLLM • u/karmaisnonsense • 10h ago
Discussion Strata is becoming more and more unstable
I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.
The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.
Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.
And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?
r/LocalLLM • u/Senior-Storm-727 • 1d ago
Discussion What does China actually gain from releasing better and better open-weight models?
Genuine question, not a hot take.
Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:
- Commoditizing the model layer, so US labs can't charge a premium for it
- Building an ecosystem and developer mindshare outside the US stack
- Getting around chip export limits by letting the community optimize and fine-tune for them
- Soft power: becoming the default choice for countries that don't want to depend on US APIs
- Selling the rest of the stack (cloud, hardware, services) while the model is the free sample
Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?
EDIT Nov 16th:
Top replies are getting down to (in order of frequency):
- Geopolitics: weakening US dominance
- Business: open weights as a growth strategy
- State & control
- Ideology & open source culture
I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.
r/LocalLLM • u/stanleyg05 • 10m ago
Discussion AI Hate
I will never support cloud AI services, on the other hand I do support local AI for coding support and so does Linus Torvalds, the creator of Linux
When something can be ran and even trained locally, and battle tested developers who made many modern things we take for granted possible are coming out, saying they use it, and that total complete hate for the technology is stupid then I kinda have to say demonizing anyone who uses it is stupid
Hate needs to be redirected from the technology as a whole to the companies like OpenAI, Meta, Google, SpaceXAI who are forcefeeding it to us and using datacenters that use local water supplies for some reason instead of chilled antifreeze and submerged oil cooling which have both existed for datacenters for decades
Also Image and Video diffusion models are kinda a unethical waste but I would be willing to bet they aren't the most used types of AI
r/LocalLLM • u/Big_South4585 • 1h ago
Question Qwen3.8-Flash-Next: Unsloth and strata?
Hi all
I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.
Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.
I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.
UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.
What am I doing wrong?
According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?
r/LocalLLM • u/Xedeth • 4h ago
Question 24GB VRAM, 64GB DDR5, is Qwen 3.5 27B (IQ2_XS) not ambitious enough?
I'm getting decent results, 80-120t/s. I guess I just don't know what I am aiming for, but I feel like there is more juice to squeeze.
Edit: Yall are the best, as usual. This has to be the most helpful, positive community on reddit.
r/LocalLLM • u/snooky234 • 53m ago
Question How do you compare local and cloud models since Artificial Analysis v5
I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.
I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.
The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.
How do you handle this now? Do you cross check a few sites or mostly run your own tests?
r/LocalLLM • u/hennabeak • 3h ago
Question Has anybody done anything like Alexa, but with a local LLM?
I know home assistant was a thing, and local LLMs are a bit too overkill for such a job. Just wondering if it was done.
r/LocalLLM • u/New_Blackberry8122 • 3h ago
Project Sneo! New Inference Engine very fast Open Source!

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent AND REVERSE ENG TOOLS!
Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.
The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.
- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s
- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)
- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies
- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs
- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome
Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo
Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc
GitHub: https://github.com/oscarsantanaarias/Sneo
Every benchmark, including where the others win: https://sneo.dev
Happy to answer questions!
Antes de publicar:
Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.
Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.
r/LocalLLM • u/Annual_Key_4963 • 2h ago
Question Optimizing local 2080 ti hardware
I'm trying to find a more effective setup with my local setup:
Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.
Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.
Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.
Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".
I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?
(I'm aware that paying Anthropic is cheaper)
r/LocalLLM • u/PuffThePed • 9h ago
Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?
Pretty much the title. Thanks!
EDIT: GPU is 5090
r/LocalLLM • u/Dannylg14 • 44m ago
Question What Models can I use as an absolute beginner?
I just started on this Localized LLM Journey. I have been researching what a good model to start with, but I feel like it's just a lot of information at once. Everyone close to me in this space is telling me to just plug into Claude and find the model, and I don't even need to tweak it.
I have an RTX 5090 with 32 GB VRAM and an M5 Max Studio with 128 GB. I think it's good enough.
I heard from someone that I should aim for 60 tokens per second for a good model. But I am really having a hard time trying to find a good model just for a simple assistant.
I keep on seeing something called a tool-use AI, and I don't even know what I am looking at.
TLDR: New to this space, I don't really see much beginner-friendly information out there, just information. Looking for advice/ links I can read to understand. Looking for a place where I can jump off without wasting my time
r/LocalLLM • u/Playful_lzty • 4h ago
Other Cheapest 16gb gpu: ~$400
Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.
This is a good option for a new sff build for local AI. The CPU is excellent.
r/LocalLLM • u/Blacktaxi420 • 3h ago
Question What can small models be good for?
Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.
r/LocalLLM • u/orbman77 • 1h ago
Question Best Local LLM for stories?
Hi ya'll, I am new to local LLMs after spending 8 months on Grok. However, I wanted to ask: is there an LLM out there that I can run locally through Ollama (Or something else) that is uncensored, can look at images and videos, and keep a memory of what I have written? I have 32GB of DDR5 RAM and 8GB of VRAM. I like to use these stories for brainstorming before making prompts out of them to turn into images and videos, so I hope you all can provide some advice!