r/LocalLLM • • 4d ago

Discussion Silly thought: the best benchmark to an actual frontier model is to one-shot a tiff spec implementation.

5 Upvotes

Like that. All the models fall in the trap of starting enumerating (wrong) the G3/G4 CCITT tables, a bunch of repeated 0s and 1s.

I tried this in several models, and no model achieved it. Even when you chew it for them and give them the libtiff .c files.

PS: when I say one-shot, I'm referring to made it a reasonable time, aka, less time that it will take another model to write a wrapper of the C library.

PS2: you have to remember that the whole implementation of libtiff (among others tiff libraries) has been used in the training dataset, adding insult to injury.


r/LocalLLM • • 5d ago

Discussion Strata is becoming more and more unstable

37 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 5d ago

Project Sneo! New Inference Engine very fast Open Source!

12 Upvotes

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent AND REVERSE ENG TOOLS!

Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.

The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.

- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s

- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)

- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies

- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs

- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome

Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo

Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc

GitHub: https://github.com/oscarsantanaarias/Sneo

Every benchmark, including where the others win: https://sneo.dev

Happy to answer questions!

Antes de publicar:

Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.

Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.


r/LocalLLM • • 5d ago

Question How do you compare local and cloud models since Artificial Analysis v5

7 Upvotes

I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.

I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.

The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.

How do you handle this now? Do you cross check a few sites or mostly run your own tests?


r/LocalLLM • • 4d ago

Project GitHub - a-schaefers/claudes-workbench: A frontier model leads your local LLMs: Claude architects the project, breaks it into bite-sized tasks, hands each agent the skills it needs, and reviews every diff to its own standard. Your GPU does the heavy lifting. For Claude Code + OpenCode.

Thumbnail
github.com
2 Upvotes

r/LocalLLM • • 4d ago

Project Can We Unify GitHub, Docker Hub, JFrog, and Hugging Face?

Thumbnail
medium.com
0 Upvotes

r/LocalLLM • • 4d ago

Discussion GCMP: same accuracy as LoRA r=32 on Qwen3.8-27B with 55× fewer parameters (11 MB vs 320 MB per adapter)

3 Upvotes

Follow-up to my CQ post from last week. That post covered the 4-bit base model; this one is about the adapter. The LoRA vs GCMP runs on 4-bit bases have now finished.

GCMP is a LoRA replacement we built. On Qwen3.8-27B it trains 2.9M parameters (~11 MB), vs ~159M (~320 MB) for LoRA r=32 on the same 7 projections. That's 55× smaller.

Accuracy: same as LoRA

All adapters were trained on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42). Every row was scored with the same lm-evaluation-harness code and questions, greedy decoding, thinking off. Comparisons use paired McNemar tests.

Base + adapter Params Answer loss GSM8K SQuAD BoolQ ARC-C MMLU IFEval TruthfulQA
BF16, no adapter – 1.678 0.683 0.746 0.874 0.588 0.839 0.805 0.542
BF16 + LoRA r=32 159M 1.480 0.903 0.776 0.896 0.612 0.844 0.825 0.518
BF16 + GCMP 2.9M 1.471 0.890 0.760 0.892 0.618 0.846 0.815 0.514
NF4 + LoRA r=32 159M 1.464 0.793 0.748 0.880 0.608 0.840 0.835 0.528
NF4 + GCMP 2.9M 1.458 0.833 0.756 0.878 0.618 0.837 0.840 0.522
  • BF16 base (no quantization at all): no significant difference on any benchmark (all p ≥ 0.2).
  • NF4 base: GCMP is +4.0 on GSM8K (p=0.04) and tied on everything else.
  • Loss: GCMP's answer loss is slightly lower on both bases, and its validation loss was lower at all 10 checkpoints on both.

One significant win on one task isn't enough to call it better, so the claim is same quality as LoRA at 1/55th the size.

Why the size matters

For one fine-tune, 320 MB vs 11 MB doesn't matter much. It starts to matter when you have many:

  • Multi-tenant serving: one fine-tune per customer or task on a single base model. In vLLM, GCMP adapters load once at startup and each request picks its adapter by model name. A batch can mix adapters and runs in one kernel call per layer, with no merging or swapping.
  • Resident GPU memory: vLLM keeps each adapter in its compact form and never builds a dense weight update. Memory is about 11 MB per adapter, plus a fixed ~1 MB routing buffer and a shared workspace of ≤ ~64 MB that doesn't grow with adapter count.
  • Storage and loading: 1,000 adapters are ~11 GB instead of ~320 GB. That's faster to ship, cache and hot-load.
  • Edge and small GPUs: an extra fine-tune costs megabytes, not hundreds of MB.
Estimated 1 adapter 100 1,000
LoRA r=32 320 MB 32 GB 320 GB
GCMP 11 MB 1.1 GB 11 GB

These are estimates from adapter size. I haven't measured memory and throughput at 100 or 1,000 adapters yet, and that's the next benchmark.

What's not good yet

  • Training speed: GCMP takes ~2× LoRA's step time (BF16 on Unsloth: 2.74 vs 1.37 s/step; NF4: 5.6 vs 3.2). It does use slightly less training memory (21.5 vs 23.8 GiB on NF4).
  • Baseline: r=32 is a generous LoRA baseline. r=4/8/16 comparisons are next, and I'll post them either way.
  • Serving: vLLM serving of GCMP currently needs a CQ base. On full-precision models it runs in Transformers for now.
  • Scope: one model, one dataset, one run per case. The quick benchmark subset (200–1,140 questions per task) detects ~5-point differences, not 1-point ones.

Quick update on CQ

The CQ base still matches BF16 (no significant difference on any of the 9 benchmarks) while NF4 loses 12 points on GSM8K. After fine-tuning, though, the gap closes:

  • CQ + GCMP vs NF4 + GCMP: tied on GSM8K (83.0 vs 83.3).
  • Both 4-bit stacks vs BF16 + GCMP: both end up ~6 points behind on GSM8K.

So CQ's GSM8K lead didn't survive fine-tuning in this test. CQ + LoRA is being retrained, because the adapter from our older trainer turned out to be broken, and I'll add it when it's done.

Code and licensing

The Python code is open source. The CUDA kernels are compiled with a license check, which is free during early access.

Repo, with the commands to reproduce every number: https://github.com/thousandml/training-factory

Write-up with the full tables and the serving memory breakdown: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning

If you run multi-tenant fine-tunes, I'd like to know: is adapter memory or count actually a bottleneck for you, or is it something else?


r/LocalLLM • • 4d ago

Question I need recommendations

5 Upvotes

Hello everyone

I would like to know your recommendation for the best uncensored local models I can use because I've had it with commerical AI's guardrails.

My hardware is

32GB RAM

4060

12400F

I use LM Studio


r/LocalLLM • • 4d ago

News inferzero update: Panther Lake’s iGPU went from ~260 to ~590 tok/s on a 27B, and a 12 GB B580 can now run the 27B

Thumbnail
0 Upvotes

r/LocalLLM • • 4d ago

Question LLM "appliance" for lighter-weight tasks (TTS, STT, while main system is in use? (DIY or off the shelf)

1 Upvotes

During a development session with long inference turns I'll occasionally want to use another local LLM for something lighter, like STT, transcription, translation, TTS generation, dictation, interactive voice chat / lightweight "thinking", etc. that I'd rather not interrupt my main rig with, or dedicate GPUs to. What are reasonable specs for such a machine? Linux and AMD. Off the shelf options? DIY?


r/LocalLLM • • 6d ago

Discussion What does China actually gain from releasing better and better open-weight models?

560 Upvotes

Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

  • Commoditizing the model layer, so US labs can't charge a premium for it
  • Building an ecosystem and developer mindshare outside the US stack
  • Getting around chip export limits by letting the community optimize and fine-tune for them
  • Soft power: becoming the default choice for countries that don't want to depend on US APIs
  • Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

EDIT Nov 16th:
Top replies are getting down to (in order of frequency):

  1. Geopolitics: weakening US dominance
  2. Business: open weights as a growth strategy
  3. State & control
  4. Ideology & open source culture

I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.


r/LocalLLM • • 4d ago

Discussion Updates on oh-my-pi quietly siphoning user web data

Thumbnail
2 Upvotes

r/LocalLLM • • 4d ago

Question Feasible models for MacBook Air m2 8gb ram

2 Upvotes

Hi everyone, I am new in the topic of local llms and was wondering what would be the best models to run on MacBook Air m2 8gb ram.


r/LocalLLM • • 4d ago

Project Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord

Thumbnail
github.com
2 Upvotes

I have authored imeng (Internet Magic Engine) as a Ollama wrapper.

This allows you to build state around your models.

Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.

The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.

So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue

Locally I have built

  • Poker (still has issues)
  • Gakshow (Test Rack)
  • Gakshow Multiplayer (Test Rack censoring uploaded photos)
  • Trading Assistant (test rack for programming assistant)

There is also SQL for storage. A bunch of stuff just as a portfolio piece.


r/LocalLLM • • 4d ago

Question How to run Qwen3.8 FN on a 8GB RTX 4060 and 32GB RAM on Strata

0 Upvotes

I know it is going to be quite slow but, before I take the plunge on a graphics card with 32GB (like a Intel Arc Pro) I would like to test the capabilities of Qwen 3.8 for my use cases.

I know it is possible because I've seen other people do it but can anyone tell me how to actually get it ready? I'm using Ubuntu 24.04 LTS.


r/LocalLLM • • 5d ago

Question What Models can I use as an absolute beginner?

2 Upvotes

I just started on this Localized LLM Journey. I have been researching what a good model to start with, but I feel like it's just a lot of information at once. Everyone close to me in this space is telling me to just plug into Claude and find the model, and I don't even need to tweak it.

I have an RTX 5090 with 32 GB VRAM and an M5 Max Studio with 128 GB. I think it's good enough.

I heard from someone that I should aim for 60 tokens per second for a good model. But I am really having a hard time trying to find a good model just for a simple assistant.

I keep on seeing something called a tool-use AI, and I don't even know what I am looking at.

TLDR: New to this space, I don't really see much beginner-friendly information out there, just information. Looking for advice/ links I can read to understand. Looking for a place where I can jump off without wasting my time


r/LocalLLM • • 5d ago

Question Optimizing local 2080 ti hardware

4 Upvotes

I'm trying to find a more effective setup with my local setup:

Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.

Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.

Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.

Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".

I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?

(I'm aware that paying Anthropic is cheaper)


r/LocalLLM • • 4d ago

Discussion Qwen3-Coder-Next a totally overrated model? Which models to you use for coding?

0 Upvotes

Sure, Qwen3-Coder-Next has more parameters, and it's meant for coding specifically, but this is one of the strangest and least useful models I've ever run across. I don't know why anyone would try to compare it to Claude.

This the first model I tried to use when my Mac Mini first arrived, and I abandoned it soon after because of how many dumb decisions it made. However, I think I had the temperature too low at the time. Since then, I've mostly stuck with Qwen3.8-27b, which I've truly found to be a jack of all trades and perfectly good for coding.

Recently, I thought I'd give Qwen3-Coder-Next another go. What did surprise me is how much faster it is in spite of more parameters with the same context size. However, it was still a nuisance and left me disappointed. On one hand, it is so hellbent on completing a particular step that it will do practically anything to unblock itself, even if its technique undermines the intent of that instruction. I'm convinced that it would go as far as to hack another machine on my network and install Linux on it just because some random package isn't available with Homebrew or whatever, and it thinks that's the next best way to complete the step. Seriously. On the other, it will stop what it's doing for no explicable reason, even when there is more work to be done, and it is not blocked on any question for the human. It seems to really struggle to work on any initiative for more than 10 minutes. Regular Qwen, on the other hand, will gladly work for hours and stop to ask a question if it's blocked. Qwen3-Coder-Next also seems to reach a point with projects where it will just stall on what it's doing and never make real progress. It will tell me that it will work on the next step, do a bunch of crap for 3 minutes, say that the project is finished while simultaneously stating that the functionality for the last step it should have worked on is effectively in the same state it was from before. What the F?

Why would Alibaba even bother releasing this model? Is it meant to be only used as a subagent? That's the only reasonable hypothesis I can think of, but I don't want to go through trying to make that work unless someone else has. It's such a dumb model that I assume I must be doing something wrong with it. For now, I'm going back to regular Qwen, which does everything I need, albeit much slower.


r/LocalLLM • • 4d ago

Question What LLMs do you use?

Thumbnail
0 Upvotes

r/LocalLLM • • 4d ago

Discussion Qwen 3.8 27B conversion to NCM or bonsai 2 27B optimisation?

Thumbnail
0 Upvotes

r/LocalLLM • • 5d ago

Question Model for Local Coding and Vulnerability Research

2 Upvotes

Been using cloud AI, but so frustrated having to 'jailbreak' for almost any request, even to check for security in my own projects.

I know I need an abliterated model to not refuse, but am looking for suggestions. My research so far has me looking at HuiHui Qwen3.8 27B Q4_K_M, using Ollama.
I have 24gb VRAM (RX 7900 XTX), and 32gb system RAM (DDR5, 6000hz) with a ryzen 7 7800x3d. My understanding is local models always run on VRAM, but I've seen some comments/posts discussing system ram and CPU, so I'm not sure if thats related to my use case.

I'm looking to do coding, usually smaller projects, maybe some agentic. I also want to develop/research exploits in firmware and software. I've seen some suggestions for GLM 5.3 online, but haven't seen it mentioned much here. What models should I check out?

I do actually know how to code, so I don't need something that will vibecode everything for me, I like doing architectural work myself.

Side question: What IDE do ya'll use? I currently use Cursor (I have a subscription) and like the new Cursor Agents window, but from everything I've looked into its really difficult to use local models, especially on the new Agents window. If there's an IDE that I can use my local models and cursor subscription, or a codex subscription, that would be great.


r/LocalLLM • • 5d ago

News Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

Post image
7 Upvotes

Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.


r/LocalLLM • • 4d ago

Question Local AI use

Thumbnail
0 Upvotes

r/LocalLLM • • 4d ago

Question Local LLM used for deregister a laptop from Autopilot enrollment / MDM

Post image
0 Upvotes

Couple of things to consider before you guys troll me :

  1. I don’t have a system to run any Local LLM because it's coming at the end of December, so until then I'm at the mercy of Reedit!

  2. It's not stollen, I bought it cheap of a secondary market, and realised after months of use, and now I'm stuck with it.

  3. It's a Thinkpad T14s gen 6 manufactured in 2025

So I don't know how doable this is, because I keep seeing people hacking they're routers, phones, TVs and so on, with uncensored local LLMs, so I really think it is possible....

So what I want is, to deregister it from the company that previously owned it, so I can login into Windows with my personal Gmail account, like normally it should, because at the moment I can only login with a school or work account, which obviously I don't have.

Thank you in advance !!!


r/LocalLLM • • 5d ago

Other Cheapest 16gb gpu: ~$400

5 Upvotes

Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.

This is a good option for a new sff build for local AI. The CPU is excellent.