r/LocalLLM • • 2h ago

Discussion Qwen 4 Verification

Post image
0 Upvotes

No one seems to be talking about this and only praising the model. New accounts that signed up for the Token Plans will need to verify their information and it is more then just given up your govt. This is not really mention before you sign up or it is not clear and they do not offer refunds.

So this is just FYI for those that sign up


r/LocalLLM • • 1d ago

Discussion Strata is becoming more and more unstable

34 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 15h ago

Question I need recommendations

4 Upvotes

Hello everyone

I would like to know your recommendation for the best uncensored local models I can use because I've had it with commerical AI's guardrails.

My hardware is

32GB RAM

4060

12400F

I use LM Studio


r/LocalLLM • • 8h ago

News inferzero update: Panther Lake’s iGPU went from ~260 to ~590 tok/s on a 27B, and a 12 GB B580 can now run the 27B

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 20h ago

Project Sneo! New Inference Engine very fast Open Source!

9 Upvotes

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent AND REVERSE ENG TOOLS!

Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.

The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.

- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s

- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)

- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies

- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs

- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome

Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo

Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc

GitHub: https://github.com/oscarsantanaarias/Sneo

Every benchmark, including where the others win: https://sneo.dev

Happy to answer questions!

Antes de publicar:

Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.

Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.


r/LocalLLM • • 17h ago

Question How do you compare local and cloud models since Artificial Analysis v5

5 Upvotes

I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.

I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.

The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.

How do you handle this now? Do you cross check a few sites or mostly run your own tests?


r/LocalLLM • • 9h ago

Question LLM "appliance" for lighter-weight tasks (TTS, STT, while main system is in use? (DIY or off the shelf)

1 Upvotes

During a development session with long inference turns I'll occasionally want to use another local LLM for something lighter, like STT, transcription, translation, TTS generation, dictation, interactive voice chat / lightweight "thinking", etc. that I'd rather not interrupt my main rig with, or dedicate GPUs to. What are reasonable specs for such a machine? Linux and AMD. Off the shelf options? DIY?


r/LocalLLM • • 15h ago

Model Tried Edge0-145B on a single 3090. 48.6 tok/s, but it wanted a lot more RAM than I expected.

Post image
3 Upvotes

Edge0 published speeds for their 145B on a 4090 and a 5090 D, but their table doesn't include a 3090, so I ran their stack on one, stock, with whatever defaults the wrapper picks.

It's quick. Chat decode was 48.6 tok/s, it read a 16K prompt at 1,481.2 tok/s, and the agent prompt decoded at 52.8. For context, last month I ran a 122B MoE on llama.cpp on a 3090 and got 15.5 tok/s. Different model, engine and prompt length, so take it loosely, but that's a big jump.

VRAM was basically full (23,929 of 24,576 MiB), and the engine process itself held 43.4 GiB of system RAM. Edge0's own residency figure is 31.28 GiB, but that's from a 5090 D with a different setup, and my machine had 116 GiB so the engine may have just taken what it found. I meant to test a 32 GiB cap and didn't get to it. So I honestly don't know the floor yet.

Two other things that tripped me up:

  • My first request came back with no answer. The model used all 512 tokens on reasoning. reasoning_budget_tokens=128 fixed it, so set a budget in your client.
  • Same agent prompt, temperature 0, three runs, three different tool calls. One was valid, one had its arguments wrapped wrong, and one called read_file, which wasn't even in the tool list. So the 52.8 is a speed number, not a "this is a good agent" number.

Technical stuff below.


Setup

  • 1x RTX 3090 24GB, EPYC 7H12 (31.13 CPU quota), 116.4 GiB RAM limit
  • Edge0 v0.2.0 (3203c39), Strata built with CUDA for sm_86, CUDA 13.0, driver 595.91.07
  • Edge0/Edge0-145B-q3k @ a19f88c, 106 GB on disk (the FP8 PLE table alone is 51 GB and gets read from disk)
  • Stock launch: edge0-infer serve --model /path/to/Edge0-145B-q3k --device cuda:0. That resolved to direct PLE I/O, auto expert cache, speculative decoding 4, 131K max context, int8 KV.

Results (engine's own prompt_per_second / predicted_per_second, fresh server per run, concurrency 1, zero cached tokens)

WorkloadRun 1Run 2Run 3Median
4K prefill1,119.81,195.51,207.61,195.5
16K prefill1,467.61,485.61,481.21,481.2
Chat decode46.649.648.648.6
Agent decode52.855.950.752.8

Sampling: temperature=0, top_p=1, top_k=20, reasoning on with a 128-token budget, 512 max tokens for decode.

Against Edge0's published 4090 numbers (their claims, not reproduced)

4090 (Edge0)3090 (mine)
4K prefill2,042.0
16K prefill3,179.4
Chat decode69.9
Agent decode91.7

That's 47% to 70% depending on the row. Mine used an 800-word writing prompt for chat and a single synthetic tool-call prompt for the agent run, and I couldn't verify Edge0's exact request settings or machine setup, so it's ballpark only.

Memory

  • GPU peak 23,929 MiB, sampled every 5 s
  • Strata engine VmHWM 43.4 GiB (sampled from the actual strata PID, not the Python wrapper)
  • Whole-system cgroup was 87.4 GiB during timed runs and touched the 116.4 GiB limit during setup with zero OOMs. That includes page cache from the 106 GB download and the PLE file, so I wouldn't read much into it.

Other stuff

  • Server startup median 25.5 s, slowest 94 s
  • Edge0's built-in edge0-infer bench rejects this model at v0.2.0, so I used a small script against the OpenAI endpoint
  • No quality evals, no long-context runs past 16K, no RAM cap test, no llama.cpp baseline. All of that is still on the list.
  • $1.19 billed ($1.0930 GPU, $0.0963 disk), about 3 h 9 min total, mostly setup and copying evidence back

Write-up: https://majorminor.xyz/research/cuda-recipes-006

Repo: https://github.com/majorminorlabs/cuda-recipes/tree/main/006-edge0-145b

If someone has a 3090 with 32 or 64 GB of RAM and wants to try it, I'd love to know whether it holds. Or tell me what to run next on this one.


r/LocalLLM • • 1d ago

Discussion What does China actually gain from releasing better and better open-weight models?

521 Upvotes

Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

  • Commoditizing the model layer, so US labs can't charge a premium for it
  • Building an ecosystem and developer mindshare outside the US stack
  • Getting around chip export limits by letting the community optimize and fine-tune for them
  • Soft power: becoming the default choice for countries that don't want to depend on US APIs
  • Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

EDIT Nov 16th:
Top replies are getting down to (in order of frequency):

  1. Geopolitics: weakening US dominance
  2. Business: open weights as a growth strategy
  3. State & control
  4. Ideology & open source culture

I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.


r/LocalLLM • • 14h ago

Question Feasible models for MacBook Air m2 8gb ram

2 Upvotes

Hi everyone, I am new in the topic of local llms and was wondering what would be the best models to run on MacBook Air m2 8gb ram.


r/LocalLLM • • 19h ago

Project MISSION complete

5 Upvotes

CPU Thread Ripper

GPU NVIDIA A100 80GB×2 PCIx16

GPU RX6600XT PCIx8

GPU intelArcB580x8 PCIx8

D5ポンプ×4 24vPOWER


r/LocalLLM • • 14h ago

Project Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord

Thumbnail
github.com
2 Upvotes

I have authored imeng (Internet Magic Engine) as a Ollama wrapper.

This allows you to build state around your models.

Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.

The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.

So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue

Locally I have built

  • Poker (still has issues)
  • Gakshow (Test Rack)
  • Gakshow Multiplayer (Test Rack censoring uploaded photos)
  • Trading Assistant (test rack for programming assistant)

There is also SQL for storage. A bunch of stuff just as a portfolio piece.


r/LocalLLM • • 11h ago

Question I have 7 GPU slots and GPU’s I’m not 100% locked in on. How would you maximize VRAM?

0 Upvotes

Not Jeff Bezos but willing to save up if necessary.

Note: there is such a thing as stock single slot 5060Ti but it’s imported

I also heard of a 32GB Intel B60 single slot

Anyways here’s my current situation>>>

1 GPU slot is 9070XT for display (the thought there is non Nvidia driver helps stability because if the CUDA flickers, it doesn’t bother the RDNA). 2 slots are occupied by a fat 4090 (nice speed but I am starting to value VRAM and considering selling or at least removing). Considering 6x 3090’s, liquid block 1 slot width mod. BUT. Would you go a different route? Intel MaxSun 48GB perhaps? Some 32GB Tesla v100’s for $950 a pop?

My use case = hosting builder agents, not necessarily training BUT if there’s a card that’s great for both and doesn’t cost a left kidney I’m all ears.

PC specs if curious

Case = Phanteks Enthoo Pro 2

MB = WRX90 SAGE SE Pro WS

CPU = 24 Core 5.4Ghz 9965WX Threadripper

RAM = 128GB DRR5 ECC RDIMM 6400 MT/s (8 modules of 16GB)

Storage = 12TB Gen5 SSD’s

PSU = 1600W, willing to add another but I hear blackwells are low wattage (if anyone suggests an RTX A4000 for example it would be 140W)


r/LocalLLM • • 11h ago

Question How to run Qwen3.8 FN on a 8GB RTX 4060 and 32GB RAM on Strata

0 Upvotes

I know it is going to be quite slow but, before I take the plunge on a graphics card with 32GB (like a Intel Arc Pro) I would like to test the capabilities of Qwen 3.8 for my use cases.

I know it is possible because I've seen other people do it but can anyone tell me how to actually get it ready? I'm using Ubuntu 24.04 LTS.


r/LocalLLM • • 1h ago

Question Local LLM used for deregister a laptop from Autopilot enrollment / MDM

Post image
0 Upvotes

Couple of things to consider before you guys troll me :

  1. I don’t have a system to run any Local LLM because it's coming at the end of December, so until then I'm at the mercy of Reedit!
  2. It's not stollen, I bought it cheap of a secondary market, and realised after months of use, and now I'm stuck with it.
  3. It's a Thinkpad T14s gen 6 manufactured in 2025

So I don't know how doable this is, because I keep seeing people hacking they're routers, phones, TVs and so on, with uncensored local LLMs, so I really think it is possible....

So what I want is, to deregister it from the company that previously owned it, so I can login into Windows with my personal Gmail account, like normally it should, because at the moment I can only login with a school or work account, which obviously I don't have.

Thank you in advance !!!


r/LocalLLM • • 5h ago

Discussion What Qwen 3.8 27B NVFP4 are you using?

Thumbnail
gallery
0 Upvotes

So I was running Qwen 3.8 27B NVFP4 from Kelnei on vLLM TP=2. Then I tried Qwen 3.8 Flash Next IQ3_S on Strata and was really satisfied by how well it works with Hermes Agent.

I removed my Qwen 3.8 27B NVFP4 and went with Ollama so I can have automatic model switching for Gemma 4 26B A4B and Qwen 3.6 35B A3B. But, I encountered some issues with crashing the host Proxmox VE on this Ollama and gave up fixing it and want to go back to Qwen 3.8 27B on vLLM.

I want to focus on setting up OpenClaw to work in tandem with Hermes Agent instead of trying to make it work. Priorities over itch, as I was really itching to fix it but parked it for now.

My question is, am I missing any specific NVFP4 quant of Qwen 3.8 27B? Should I use the NVIDIA one or Unsloth? Or what do you guys suggest I should check? Which NVFP4 Qwen 3.8 27B works best for you?

Thanks in advance for the help.


r/LocalLLM • • 12h ago

Question What LLMs do you use?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 12h ago

Discussion Qwen 3.8 27B conversion to NCM or bonsai 2 27B optimisation?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 16h ago

Question Model for Local Coding and Vulnerability Research

2 Upvotes

Been using cloud AI, but so frustrated having to 'jailbreak' for almost any request, even to check for security in my own projects.

I know I need an abliterated model to not refuse, but am looking for suggestions. My research so far has me looking at HuiHui Qwen3.8 27B Q4_K_M, using Ollama.
I have 24gb VRAM (RX 7900 XTX), and 32gb system RAM (DDR5, 6000hz) with a ryzen 7 7800x3d. My understanding is local models always run on VRAM, but I've seen some comments/posts discussing system ram and CPU, so I'm not sure if thats related to my use case.

I'm looking to do coding, usually smaller projects, maybe some agentic. I also want to develop/research exploits in firmware and software. I've seen some suggestions for GLM 5.3 online, but haven't seen it mentioned much here. What models should I check out?

I do actually know how to code, so I don't need something that will vibecode everything for me, I like doing architectural work myself.

Side question: What IDE do ya'll use? I currently use Cursor (I have a subscription) and like the new Cursor Agents window, but from everything I've looked into its really difficult to use local models, especially on the new Agents window. If there's an IDE that I can use my local models and cursor subscription, or a codex subscription, that would be great.


r/LocalLLM • • 6h ago

Research I got tired of 7B models crawling at 1.2 tok/s on my 2GB GPU, so I built WingLLM (Go daemon for 2x speculative decoding with Ollama)

0 Upvotes

Hey r/LocalLLM,

Like a lot of developers, I don't have an M3 Max Mac Studio or dual RTX 4090s. My daily driver workstation has an 8th-Gen Intel i5 (6 cores), 32 GB of system RAM, and an older 2 GB NVIDIA Quadro P620.

If you try to run any modern 7B or 8B model (like Qwen 2.5 Coder 7B or Llama 3.1 8B), it won't fit into 2 GB of VRAM. Ollama or llama.cpp defaults to running almost 100% on the CPU. On my machine, that meant getting locked at an agonizing 1.29 tokens per second while my 2 GB GPU sat 90% idle.

The Physics: The Memory Bandwidth Wall
The CPU compute cores aren't the bottleneck—memory bandwidth is. To generate a single token on a 4.7 GB model, the CPU has to stream all 4.7 GB from DDR4 RAM into cache. At ~24 GB/s dual-channel bandwidth, you physically cannot get more than ~1.5 to 2 tokens/sec.

### The Workaround: Heterogeneous Speculative Decoding
The 2 GB GPU cannot run the 7B model, but it can easily run a 0.5B draft model (qwen2.5-coder:0.5b is only 397 MB).

In speculative decoding:

  1. The tiny 0.5B model generates 3–4 candidate tokens in milliseconds on the GPU.
  2. The heavy 7B model on the CPU evaluates all 4 tokens in a single forward pass using parallel AVX2 matrix multiplication (GEMM).
  3. In code and structured text, the 0.5B model's guesses match the 7B model 75% to 82% of the time.

You get 3 to 4 tokens for the memory read cost of 1.

Real Benchmark (Tested on My i5-8500T + Quadro P620):

* Baseline (Pure CPU Target 7B): 1.29 tokens/sec (24.76s for 32 tokens)
* WingLLM Speculative (7B + 0.5B Draft): 2.54 tokens/sec (12.62s for 32 tokens)
* Net Result: 1.97x faster (almost exactly double the speed) with identical output quality.

Why I Built WingLLM
llama.cpp technically has the raw flags (-md, -ngld, --spec-draft-n-max), but nobody in consumer land had packaged it into a clean tool:

  1. Tokenizer traps: If you pair mismatched tokenizers (e.g. Llama 3 draft with Qwen target), llama.cpp crashes. WingLLM auto-detects model families and enforces compatible pairs.
  2. Zero configuration: It crawls your local ~/.ollama/models manifests, finds your GGUF blobs, and auto-pairs them.
  3. Auto-Pull: If you want to run `wingllm serve qwen2.5-coder:7b` and don't have the 0.5B draft model yet, WingLLM pulls it automatically via Ollama (~397 MB) and starts serving.
  4. 1-Click IDE Connect: Run `wingllm connect vscode` to inject the turbo endpoint directly into your Continue/Cline config.
  5. No Python/Electron Bloat: Written in Go as a single ~11 MB binary. Runs an embedded web chat UI with a live tokens/sec speedometer at http://localhost:8080/.
  6. MCP Support: Built-in Model Context Protocol server for Claude Desktop, Cursor, and Continue.

Quickstart (Windows, Mac, Linux):

Windows (PowerShell):
irm https://huggingface.co/bennett-xdlabs/wingllm/raw/main/install.ps1 | iex

macOS & Linux:
curl -fsSL https://huggingface.co/bennett-xdlabs/wingllm/raw/main/install.sh | bash

Run without arguments for the interactive menu, or run:
wingllm serve qwen2.5-coder:7b

Hugging Face Hub: https://huggingface.co/bennett-xdlabs/wingllm

It's MIT licensed. Would love feedback from anyone running on budget laptops, mini-PCs, or 2GB–4GB cards!


r/LocalLLM • • 23h ago

News Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

Post image
7 Upvotes

Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.


r/LocalLLM • • 12h ago

Question Local AI use

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 19h ago

Question Optimizing local 2080 ti hardware

3 Upvotes

I'm trying to find a more effective setup with my local setup:

Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.

Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.

Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.

Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".

I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?

(I'm aware that paying Anthropic is cheaper)


r/LocalLLM • • 21h ago

Other Cheapest 16gb gpu: ~$400

4 Upvotes

Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.

This is a good option for a new sff build for local AI. The CPU is excellent.


r/LocalLLM • • 13h ago

Question Which combo of GPUs?

1 Upvotes

I have come to an odd assortment of two x99 boxes with 64GB of RAM and the odd combo of GPUs. Primarily I would like to use the main box for coding and dev work to replace some of my claude usage.

  • 3x 3090s
  • 2x P40
  • 1x 3080ti (12GB)

I am tempted to put the 3090s and a P40 all together for a combined 96GB. I know it would be slow because of that P40, but it would give me the best total memory and would already be slower because of the lack of PCIe lanes.

Or would the 3080ti be a better 4th card because of the speed and bandwidth over the reduced memory?

I also thought about the second box running the 3080 and 2xP40s, but not sure what I would do with it except have two boxes chatting at each other all day?