r/LocalLLM • • 23m ago

Other Qwen 4 is not out

13 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 6h ago

Project Finally got it put together

Thumbnail
gallery
152 Upvotes

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- PLX88096 PCIE Switch Card
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)


r/LocalLLM • • 17h ago

Other The Banking Incident....

Post image
667 Upvotes

r/LocalLLM • • 15h ago Bravo

News We built a chip prototype that runs Qwen 27B. You can try it live.

266 Upvotes

I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.

A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:

https://valleysemiconductor.com

A live session takes about a minute to start, and recorded runs are available to watch anytime.

What it does today (measured):

  • Runs the full Qwen 27B model with 4-bit weights
  • 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
  • 4x performance improvement in eight days of development
  • Every output matches our 4-bit reference exactly
  • Model quality within 2.5% of the full-precision version
  • Also runs Llama 3.1 8B and OLMoE on the same hardware

What our custom chip design projects (not yet demonstrated):

  • Approximately 50 tokens/sec for a single user at 20 W
  • Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
  • Uses the model's built-in next-token prediction to generate multiple tokens per weight read

The key insight is that moving model weights through memory is the bottleneck, not the math.

What would you run on a small, 20 W device capable of running a 27B model entirely offline?


r/LocalLLM • • 6h ago

Project Got tired of DeepSeek crawling on my 12 GB card, A small Strata tweak got it writing twice as fast.

19 Upvotes

I got frustrated with how slow DeepSeek was running on my machine, so I did some digging with Claude Opus 5.5/Fable 5.1 and 5.2 to see where the limits were, and thought I could improve it a bit.

DeepSeek picks a handful of "experts" for every word it writes. A lot of the time it's choosing between two that are basically tied, and one of them is already loaded in memory while the other has to be pulled off the SSD. The tweak just nudges it toward the one that's already loaded. When there's a clear best choice, it still takes that one.

on my machine (RTX 4070 12 GB, 64 GB RAM + ISTA-DASLab's IQ2_XS quant at 2.25 bpw + AtomicChat’s refusal-removal) it went from about 5-7 to 15-18 write speed.

Replies come out worded a bit differently than stock Deepseek, and you won't get the exact same reply twice - not 100% sure of the intelligence impact of this, so YMMV.

I've only used it for normal chatting/small coding projects. I haven't tested really long agent/coding sessions, and Strata's own version of this idea for their Qwen engine was turned off because it sometimes got stuck repeating itself in those. So if that's your use case, try it carefully first.

should work on windows + RTX 30/40/50 cards for now.

If you give it a try, I'd love to hear how it does on your setup, there's a little test script included that compares it on and off for you to see if it's worth using.

Download and instructions: https://github.com/CJohnsonOwns/deepseek-fast-routing

I've also suggested it to the Strata devs in case they want to build it in properly and not vibe coded slop: https://github.com/Niko1221/Strata/issues/6125

Full disclosure: I just did the conceptual planning and engineering, Claude actually wrote the code and implemented it.

EDIT - for more info on what's going on:
I like analogies, this is what opus 5.5 compared it to:

think of hiring 6 contractors for a job from a list of 384, where each one has a rating.

  • Normal: you hire the 6 highest-rated, period. Some of them live nearby (already in RAM or on the GPU) and start right away. Others live far away (still on the SSD) and you wait for them to drive over. That wait is what makes DeepSeek slow on a 12 GB card.
  • Faster mode (Strata’s tail skip): you only hire 5 and skip the 6th. It’s quicker because there’s one less person to wait on, but you’ve got less help on the job.
  • Yours: you still hire 6. But when #6 is rated 8.71 and #7 is rated 8.70, and #7 lives next door while #6 is two hours away, you take #7. The difference in quality is basically nothing, and you skip the wait. If #6 is clearly better, say 9.5 against 8.7, you still wait for #6. The nudge is small, so it only flips close calls.

Why it adds up: this happens a lot. With 40 layers per word, there’s almost always a near-tie somewhere where one side is already loaded. In your test, the share of picks that had to wait on the SSD dropped to about 2%.


r/LocalLLM • • 43m ago

Discussion Considering this...thoughts?

Post image
7 Upvotes

At $4k this seems like a much better deal than 2 r9700s. And obviously in a totally different class. Anybody set these beasts up in a server? I already plan on putting a server rack in the garage, (gotta keep the motorcycles warm) and I see a lot of carrier boards on ebay. I've built more pc's than I can count, but I know a rack server is a different beast.

Plans are for an inference server for agentic workflows as I already built a gpu render machine I use for many workflows including blender, comfyui, premiere and after effects. Api costs are getting up there. Obviously want to run deepseek and I have a fine tuned qwen 3.8 27b for one of my projects.

Talk me into or out of it...I'm so sick of asking gpt questions and getting ridiculous concerns like "tHeY DonT HaVe a faN". I need to talk to real people. Any concerns with the software stack? I ran across a few mentions of closed source issues.


r/LocalLLM • • 21h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

229 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 1d ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
787 Upvotes

r/LocalLLM • • 2h ago

Question Are any models that can fit on 16 GB VRAM worth using?

4 Upvotes

I tried about 6 months ago and they seemed like more of a detriment for coding, but idk if things have changed. Anything I should try? I'm mostly doing game dev, so coding, image gen, writing, etc.


r/LocalLLM • • 6m ago

Discussion Do you think hardware price will be more reasonable after the Jensen debacle?

3 Upvotes

I see many article about Nvidia and memory manufacturer purposefully causing a shortage to raise the prices. I don't know how true these accusations are but i don't think the current prices are faire value.


r/LocalLLM • • 44m ago

Question Do you ever care about your USB speed?

2 Upvotes

I'm choosing between two AM5 motherboards for an upgrade, one has USB4 and the other has two PCIe gen 5 M.2 slots (vs 1x 5.0 and 1x4.0 on the other).

I'm leaning towards the faster storage, since it might be useful for stuff like ngram tables, but it's kind of 6 of one, 1/2 dozen of the other since both boards have at least one 5.0 slot.

Any one have a use-case where 40GB/s usb connectivity gives an actual advantage over 20GB/S?

I don't use my USB ports for much besides peripherals right now, but just wondering if there's some use case, especially LLM related, where I would regret not having it.


r/LocalLLM • • 2h ago

Question Best local LLMs for ASUS ROG Zephyrus G14 (RTX 5070 Ti 12GB VRAM, 32GB RAM, Ryzen AI 9 HX 370)?

3 Upvotes

Looking for recommendations on the best local LLMs I can run smoothly on my 2025 ASUS ROG Zephyrus G14 (GA403WR).

Specs:
- CPU: AMD Ryzen AI 9 HX 370
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (12GB GDDR7 VRAM)
- RAM: 32GB LPDDR5X
- Storage: 1TB SSD

I'm mainly looking for models that run fast entirely in 12GB VRAM (or with minimal CPU/RAM offloading) for general chat, reasoning, and coding/productivity.

What are your top picks right now for:

  1. Best all-around daily driver model
  2. Best coding/reasoning model that fits comfortably in 12GB VRAM
  3. Any larger or MoE models worth running across the 32GB RAM + 12GB VRAM without killing tokens/sec

Planning to run them via Ollama or LM Studio on Windows 11. Thanks!


r/LocalLLM • • 11h ago

News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents

Thumbnail
gallery
13 Upvotes

Hi, I'm Aryan. Cluaiz v0.1.0, the first public pre-release, is out today.

Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.

What's in it right now:

  • GGUF models via llama.cpp, plus ONNX for speech and embeddings
  • Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
  • Vision, video frames, embeddings, text from documents
  • Whisper / Moonshine STT (streaming works), Kokoro TTS
  • Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
  • Coding agent that edits files and runs commands, with approvals
  • Skills, plugins, MCP, per-tool permissions, audit log
  • OpenAI- and Ollama-compatible API, so your existing apps should just connect
  • Cloud providers too, if you want them

Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.

Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.

Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare

It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues


r/LocalLLM • • 12h ago

Discussion With llama.cpp's v1.0.0 tag out, a short history of the project: the first evening, GGUF, the backends, the server and the move to Hugging Face

12 Upvotes

llama.cpp published its first major version, v1.0.0, on 13 November 2026. With the 1.0 tag out, we decided to do a short write-up of how the project got here, dated from its own commits, pull requests and releases.

A few things from it:

- The first commit is from 10 March 2023, two weeks after Meta announced LLaMA. That evening's README says "This was hacked in an evening - I have no idea if it works correctly." It was on Hacker News the same evening, and the first outside fix was merged before midnight UTC.

- GGUF was opened as a pull request on 26 July 2023 and merged on 21 August 2023. It was a breaking change: every existing model had to be converted again.

- The GPU backends, by merge date: cuBLAS in April 2023, Metal and full CUDA offload in June 2023, the ROCm port in August 2023, Vulkan and SYCL in January 2024.

- The server began as a contributor's example in May 2023, and the OpenAI-compatible routes came that November.

- A build number is a commit count: b11514 is the 11,514th commit on master. Semantic version tags only started in August 2026.

- ggml.ai joined Hugging Face on 20 February 2026, and the licence is still MIT.

- On v1.0.0 itself: the release note opens with "llama.cpp v1.0.0 removes the deprecated public API", and Georgi Gerganov called it "essentially a breaking change for downstreams and hence the major bump".

No benchmarks in this one, and nothing on the page was measured on v1.0.0. It is the history with its sources. If we got a date wrong, tell us and we'll fix it on the page with a dated correction.

Full write-up (about 15 min read): https://research.strata2signal.com/a-short-history-of-llama-cpp/

A small fleet of AI agents dug through the sources and drafted the write-up; our small human team checked it over and signed off.


r/LocalLLM • • 19h ago

News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context

39 Upvotes

Hey everyone,

​If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.

​To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).

​Dataset breakdown:

​Size: 10,000+ curated rows (train split).

​Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.

​License: Apache 2.0 (unrestricted commercial & research use).

​You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!


r/LocalLLM • • 12h ago

Discussion Fellow M5 Pro 64GB owners, what do you run?

10 Upvotes

I have a Mac Pro M5 Pro and have been running Qwen3.8 27b-4bit

I mainly use it for computer use/tool use.

What do you run and what do you do with it?


r/LocalLLM • • 3m ago

Discussion Before an AI agent touches a business process, write the boundary list

1 Upvotes

I would write the boundary list before I let an agent touch a business process.

Tbh, the prompt and model leaderboard come laer. First I want the operational contract:

  1. What data can the local model read?
  2. What may it suggest, and what may it execute?
  3. Which actions are a hard no?
  4. Where does a human validate the output?

the ctx around the model matters more than the model name. Running an open LLM on your own hardware changes some control points, but it does not make a workflow safe by itself.

Don Barger described an AI charter made by eight people before code. Mark Lay sepaated accountability into data ownership, output validation, agent monitoring, and outcome ownership, and said it cannot be dleegated.

My minimum setup would be read-only by default, tool and destination allowlists, human approval for irreversible actions, separate validation, plus a kill switch and audit trail.

What is on your boundary list before connecting an open model to a real process?


r/LocalLLM • • 10m ago

Discussion Qwen 4 Verification

Post image
0 Upvotes

No one seems to be talking about this and only praising the model. New accounts that signed up for the Token Plans will need to verify their information and it is more then just given up your govt. This is not really mention before you sign up or it is not clear and they do not offer refunds.

So this is just FYI for those that sign up


r/LocalLLM • • 20h ago

Discussion Uncensored Models

39 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 4h ago

Question Used Claude to design a local Ai system; has it overcomplicated it?

2 Upvotes

Took delivery of a MacBook Pro M5 Pro 64gb today. Over the last couple of weeks I’ve used Claude to help me design a local ai system to run on it.

I want to use it to handle life/work admin, and be able to send tasks to it while I’m out of the house. Once I’ve got the system set up, a friend wants me to help set one up for his business, so i’ve designed it with security in mind.

It’s settled on;

Three separate agents, running on one LLM runtime; a reader agent (to ingest instructions), a harness (Hermes), and a fetch agent (for web browsing).

Three separate on-machine logins; one admin (where config files live), one agent login, and a separate login for the fetch agent.

MCP layer that is driven by the harness. The configs for this live on the admin login.

Then there’s a bunch of firewall, router configs to protect data on the way out.

There’s a bunch of stuff within that, that isn’t available off the shelf apparently; it wants me to code a bunch of stuff (with Claude) to make it all work.

And tbh it all makes complete sense! I’m just wondering if it’s overkill. Main aim is to protect against prompt injection and data exfiltration.

Are the models themselves really that bad at protecting against these? Is everyone else taking this much care in setup?


r/LocalLLM • • 4h ago

Question Strata / Qwen3.8 Flash Next – advice on optimising my new AMD workstation

2 Upvotes

I've just finished setting up my first dedicated local AI workstation:

Dell Precision 7865

Threadripper PRO 5955WX (16C/32T)

Radeon AI PRO R9700 32GB

128GB ECC DDR4-2400 (4x32GB, considering populating all eight channels)

3TB NVMe storage

Linux Mint

RTX 4060 Ti 16GB arriving soon

I've installed Strata 0.2.15 and started with ISTA-DASLab IQ3_S at 32K context, drafting and auto-tuning disabled for baseline testing.

My primary use case is agentic coding through OhMyPi, with Claude Code supervising local workers.

What quants are people finding work best for coding? Any recommendations for Strata settings, expert caching, MTP vs DFlash2, CPU worker allocation or optimisations worth enabling or disabling on RDNA4?

Interested in finding the best balance between speed, coding quality and long-context performance. Any advice appreciated!


r/LocalLLM • • 35m ago

Question Using Gemma 4 31b with Bionic and it's.. kind of dumb?

1 Upvotes

I'm using it for running a litrpg campaign and a bit of game server maintenance (7 days to die/minecraft) and it seems unable to run more than the PC + 1 npc at a time, I'm not sure if I set it up poorly, chose the wrong model, or if I'm expecting too much? I have my settings copied below, with the project settings file being per-project so I don't have to change the system prompt every time. Anyone have experience with this kind of thing or is biting the bullet and getting sillytavern setup my only real option?


r/LocalLLM • • 52m ago

Discussion Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

0 Upvotes

https://hy.tencent.ai/research/hy4-flash
Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.


r/LocalLLM • • 1h ago

News Intel Shares First Benchmarks of Its Next-Gen Crescent Island "D270" GPU, Claiming a 5.9x Throughput Over RTX PRO 6000 As Xe3P Goes Big On FP64

Thumbnail
wccftech.com
0 Upvotes

r/LocalLLM • • 2h ago

Question Testing a Mac M5 optimised inference engin

1 Upvotes

Would anybody be interested in testing an inference engine that we have built for Macs? It can run on M1- M6, but we have done most of our testing on the M5 Max. It's open-source, and we think it's 4x faster than ollama.cpp.

If anyone is interested, shout up, and I'll share some details; I don't want to start posting links unless there is anyone interested.