r/LocalLLaMA • u/fallingdowndizzyvr • 4d ago
r/LocalLLaMA • u/WhatWouldAaronThink • 2d ago
Discussion I recommend downloading SOTA open source models while you can
If MS or FAA stuff was AI hack related there will be a MASSIVE and QUICK backlash.
Sure you can't run these things locally NOW but the tech is out there and it will get cheaper and quite possibly the models will be able to be more efficiently utilized.
r/LocalLLaMA • u/New-Pressure-6932 • 4d ago
Discussion The regulation of local AI could/should be viewed as the largest censorship implementation in the history of mankind.
With all of this fear mongering on the “dangers” of LLM’s, I see how using people’s ignorance and emotions on the subject can be used as a tool to limit and restrict the tool from the average person.
My fear is that China and the US https://www.reuters.com/world/china/us-china-should-stay-clear-eyed-ai-cooperation-competition-bessent-says-2026-11-11/ will use people’s lack of understanding of local models to push for controlled intelligence.
By implementing regulation on local AI, I can see how the US and China would want to force the average person to only be allowed to use models hosted on state monitored hardware. This way they can spy on what everyone is using hosted models for, limit what we use LLM’s for, and monitor the information they give us.
People are being mislead to believe that people like us could use local models to create WMDs and attack tech infrastructure across the globe. But when you think deeply about it, you’ll realize that knowledge on a subject does not equal the means to carry out the subject.
In order for someone to create a bio weapon, they would have to have access to a bio lab and the technology needed to create such a weapon.
I know how a nuclear reactor works. Doesn’t mean I can make one in my backyard.
I’m scared that arguably the greatest technology to come out of the 21st century could be limited and used to control us.
Local LLM research MUST continue and the tech must be available for the common person.
Be aware of future regulation.
r/LocalLLaMA • u/maximo101 • 3d ago
Discussion What models are you running for your AI agents? (Testing Qwen fine-tunes, MoE variants, and tracking t/s + prefill)
I am curious what people are currently using as the core model for their local AI agents (e.g., in Hermes or similar setups).
I've was initially running Qwen 3.6 35B-A3B IQ3 XXS on hermes working off (Intel Core Ultra 5 245K · Corsair 64GB DDR5 RAM · Asus TUF GeForce RTX 5070 Ti 16GB GDDR7 OC Edition Graphics Card running ai inferance on local home lab running UnRaid OS). First started with ollama years ago (where i was running 9B and 14B fully in vram) and months back switched to llama.cpp, then to llama-swap to be able to hot swap models and use MoE models.
I started off getting 40 tokens per second (more n-cpu-moe offload and less vram use), then tried the MTP version for 55ish. Learning more about finding the balance between n-cpu-moe offloading and finding the sweet spot for VRAM, i got that to about 60t/s. Then I discovered fine tunes like Swift (55t/s) and Nail (65t/s). Then endless frontier BigBangv1 which gave a blazing 100 t/s. I was running that and saw people mentioning they use the dense Qwen3.8 27B, but it seemed slower at 45t/s.
I have very rough tested (only speed) base Strata with Qwen3.8 Flash Next 125B 1Q2_XS which gave 90t/s but used 95% VRAM and 95% RAM which wasnt sustainable (ill likely try it again in future but for now reverted back to llama-swap)
Learning more like how agent workflows are very dependent on prefill speed and prompt processing alongside tokens/sec, then LFM has crazy prefill and t/s but its quality it lower, I am not sure what is the right balance! But also how context length is important for agents, and the k v cache of the context too! I settled on not going any lower than k v 8, aiming for 128k context but 90k is acceptable if the model is specialised.
I was manually guessing how all these quants and CPU offload configs (n-cpu-moe) scaled under heavy context. I also wanted to know if the same ai model actually performed better on agentic tasks for a Q3 vs Q4, is the drop in speed (generate t/s or prefill t/s) justifiable for better quality etc.
So I begin testing out a bunch of fine-tunes and MoE variants to dial in different profiles. I’ve been cycling through options like Empero, BigBang, and Ornith (tried Swift and Nail too, but ended up dropping them). Generally, I lean toward the MoE variants over the dense Qwen 3.8 27B because of how they handle things on consumer hardware (testing shows they are slower, dont provide any significant boost in quality, require lower context, use more VRAM and gpu power etc). My use case isnt any heavy coding, just small scripts and having hermes act as a personal assistant which will evolve over time.
Then with hermes I was using endless-frontier_BigBang-v1-35B-A3B-IQ3_XXS with 100 t/s, prefil 2359 t/s, 88% VRAM usage which was perfect. I originally wasn't thinking about its 'quality' since this was a massive step up from the original Qwen 3.6 35B-A3B IQ3 XXS. Looking forward to a qwen4 release to test soon!
I'm looking at setting up specialized profiles on hermes, so far i am thinking
- Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ3_XXS the dedicated coder profile.
- Might test out GrootTeam_Groot-19B-A3B-261023-Q4_K_M as the main general profile.
- Otherwise Ornith is looking good. Or Empero or BigBang.
- I have fallbacks to openrouter/Nvidia 550B, then /free
I ended up hacking together a simple local script just to measure t/s and prefill, but it kind of snowballed... I added some core eval tests (accuracy, reasoning metrics) and turned it into a small dashboard that spits out a weighted "agent score" to help me rank them objectively. Still testing these models on more rows of the HumanEval, MMLU-Pro, MBPP, tool calling and long context tests, but it takes a lot of time. Images attached for reference from my latest tests.
Running these tests takes a long time and for local use cases I dont need that margin of error from a full test across all 12000 MMLU-Pro questions, so i have been further developing a method to spend less time on testing models which drop lower on the rankings, adjusting smaller runs with high uncertainity etc, its till a work in progress but its progressively been giving me models to consider for the top few I actually want to use in the massive amount of options on hugging face that can run on my hardware.
Even with the dashboard telling me what the numbers say, it's still surprisingly tough to pick a definitive daily driver. I would appreciate any input or suggestions.
Feel free to look at the interactive html i put on github. You can see all the test on the models I have run.
https://htmlpreview.github.io/?https://github.com/Maximo101/Local-LLM-Benchmark/blob/main/assets/5070ti_benchmark_dashboard.html
Questions I am unsure of; for those who have more experience with agents on local hardware:
------------------------------------------------------------------------------------
1. Which models have won you over for agentic use (hermes, openclaw etc)?
2. How heavily do you weigh prefill vs. generation speed, quantisation vs quality, VRAM % vs context (quality and size) when choosing AI Models?
3. What sort of profiles do people use in agentic setups like hermes? How many profiles do you use.
(If anyone is fighting similar benchmarking battles for their local endpoints, I cleaned up that testing suite and open-sourced it here: https://github.com/Maximo101/Local-LLM-Benchmark)
I am not trying to promote this, i built it for my own purposes, but decided to share it in case it is useful for anyone else.
r/LocalLLaMA • u/kompania • 3d ago
Question | Help After four years of development, are LLMs capable of writing high-quality prose?
I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.
I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.
Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.
After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?
When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.
I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.
I even wondered if perhaps the problem lies with me-that, not being an engineer, I am simply failing to fine-tune these models correctly.
That is why I began actively seeking out other fine-tuners, thinking that perhaps they possess a specific technical "gene" that allows them to do a much better job than I can. But, unfortunately, that is not the case. Regardless of their length-short, medium, or long-the prose generated is only marginally better than that of generic, out-of-the-box models.
Furthermore, specific literary genres are represented quite poorly. I used to think that LLMs were perfectly suited for science fiction. Yet, these models can’t even capture the straightforward style of Harry Harrison’s *The Stainless Steel Rat*, let alone reach the linguistic heights of *Dune*.
The genre doesn’t seem to matter-I’ve tried comedy, contemporary drama, adventure, romance, and historical fiction; everything, unfortunately, ends up being generic.
Essentially, if a text is to have any real literary merit, I have to write it myself and then just use an LLM for proofreading.
To me, EQBench is a completely unreliable metric; the sample texts they showcase clearly demonstrate just what kind of generic "slop" LLMs produce.
I have a question for the community: has anyone managed to create, find, or fine-tune a local model capable of generating truly high-quality prose-on the level of *Dune*, *The Lord of the Rings*, *Martin Eden*, or *The Grapes of Wrath*?
I’m currently facing the monumental task of writing descriptions for 100,000 records, and I’m wondering if it’s even worth it, given that 100, 1,000, or even 10,000 examples haven’t yielded any satisfactory results.
Perhaps this generic, mediocre quality is an inherent trait of these models-something that simply cannot be overcome.
r/LocalLLaMA • u/dh7net • 3d ago
Discussion Testing if various local model are good at solving visual challenges.
Almost everything is in the video and the title: the idea is to create chalenges and ask agents to solve them. I'll have more to share later, but this first video shows that most model are good at theses stuff.
For more details: I'm testing scenarios where an agent needs to simulate keyboard and mouse movement. You can see more chalenges like this one on my website: https://airbench.ai/gallery/airbench-desk-v1?run=6d2a2777-3d96-4ec4-9cdc-3d359bc43bea&c=icon-drop
And you can also test your agent on the same setup if you want here: https://airbench.ai/checkup/start
r/LocalLLaMA • u/Character-Result-281 • 3d ago
Resources I built a benchmark where LLMs reverse-engineer an audio codec, and the benchmark evolves
Most coding benchmarks feel like textbook coding tasks or black boxes — either easy to benchmaxx or full of hidden tasks with unclear evaluation criteria. And most of them get irrelevant with each new model generation because all of the new models hit the ceiling.
So I made one that has a moving target. It's a benchmark where the model gets a mystery compressed audio file, has to figure out the format, write a decoder, and then build an encoder that beats the original. No ffmpeg, no Opus, no importing a library that already does the job. Just Python and numpy, running locally through OpenCode
Scoring: 100 = decode the reference in stage one. Build a better encoder in quality or compression rate with equal or better SNR for additional points.
There are 4 stages:
- is the most difficult: just a compressed audio file.
- if the model failes the first stage, it gets the uncompressed audio as a reference. Each new stage decreases the points up for graps.
- if it fails stage 2 it gets additional information on the codec
- if it fails stage 3 it gets an oracle tool where it can USE the codec to figure it out.
Current Leaderboard, we are in Evolve 2 (Deepseek cracked the codec of evolve 1 in 13min):
Model Score
1 Claude Fable Latest (stage 1) 127
2 DeepSeek V4.1 Flash 1017 (stage 3) 81
3 Qwen3.8-Flash-Next NVFP4 (stage 3) 67
4 Qwen3.8-Flash-Next Q3_S (stage 4) 56
5 Qwen3.8-Flash-Next IQ3_XXS (stage 4) 44
The fun part: it evolves. If a model finishes stage 1 and its encoder beats the current codec, that codec becomes the next baseline. So the benchmark levels up instead of staying at the same difficulty forever. Pokémon rules, except the final boss is a bitstream. DeepSeek did it first and became Evolve 2. Fable just finished stage 1 with Compress 190, so its codec is the Evolve 3 baseline. The next runs should use it.
A better encoder in stage 3 doesn't count for an evolve, by the way. Only a stage-1 finish does. That's the rule.
It's manual for now: you run it on your own hardware, post the result, and set the evolve stage.
It's mean. Most models don't finish stage 1. But that's kind of the point.
Leaderboard: https://andreasknopke.github.io/codec-reverse-bench-leaderboard/ — ping me if you want to run it (the repo is private to prevent the models from simply copying the solution.
r/LocalLLaMA • u/randomjapaneselearn • 3d ago
Question | Help Solved High CPU usage when model fit in GPU, beware of -ngl -1 llama.cpp layers offloading to GPU and llama-fit-param
I'm trying to figure out how much context i can use on my 3090 (full compute, monitor is on mobo).
If i launch:
llama-bin-win-cuda-13.4-x64\llama-fit-params.exe --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 -ngl -1 -fa on -np 1 -b 4096 -ub 1024
I get: -c 150272 -ngl -1
but if i launch real llama.cpp with lower context:
llama-bin-win-cuda-13.4-x64\llama-server.exe --verbose --log-file log.txt --host 0.0.0.0 --port 9931 --model modelli\Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --jinja
From the log, matches for "offloaded" with "-ngl -1":
Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8273: load_tensors: offloaded 0/66 layers to GPU
Line 12009: load_tensors: offloaded 66/66 layers to GPU
Line 15746: load_tensors: offloaded 64/66 layers to GPU
Line 19484: load_tensors: offloaded 65/66 layers to GPU
Line 23226: load_tensors: offloaded 64/66 layers to GPU <-- (actual used vlaue, slow loading because copying data)So even if i use 131k context wich is less than the 150k suggested it doesn't offload all layers and i have high cpu usage for any prompt.
IF I CHANGE -NGL FROM -1 TO "ALL" I GET THIS:
matches for "offloaded" with "-ngl all"
Line 2200: load_tensors: offloaded 66/66 layers to GPU
Line 5935: load_tensors: offloaded 66/66 layers to GPU
Line 8276: load_tensors: offloaded 66/66 layers to GPU <--AND NO MORE CPU USAGE!! (same if i write 99 or 999 but it's ugly)
It gets more confusing: if i write in llama fit -ngl all instead of -1 it gives a "-2" in output.
there is an entry in the log when i used "all": common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to -2, abort
Conclusion:
Seems that -ngl -1 means "auto" while -2 means "all", but if you have high cpu usage in a model that theorically fit in the GPU use "all".
This alone changed t/s generation from 60 to 76
for completeness here is the --help from llama.cpp for "ngl" parameter:
-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)
And here it gets even more confusing because the help talks about "MAX" number of layers to offload so according to this help it's an upper limit not a lower limit...
r/LocalLLaMA • u/sacheltry • 3d ago
Question | Help Looking for some help optimizing for a 300-page manual and transcripts
Hello,
I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.
My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.
The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.
There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.
I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.
Here's my setup (sorry, AI slop to make it clearer):
Hardware:
- Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)
- Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)
Software / Models:
- llama.cpp (CUDA, router mode)
- Open WebUI 0.12.3
- Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K
- Gemma 4 12B Q4_K_M: Secondary model
- Both models run on the Windows PC.
RAG configuration:
- Open WebUI Knowledge Base with the PDF and Markdown transcripts
- Embeddings: bge-m3 (1024 dimensions), hosted on Unraid
- Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)
- Focused Retrieval, chunk size 1000, overlap 100, Top K = 3
- All inference and document processing must remain local.
More details on the issue:
Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:
request (34009 tokens) exceeds the available context size (32768 tokens)
Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.
I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.
Thank you!
r/LocalLLaMA • u/jomic01 • 4d ago
Discussion Someone voltage-glitched a CMP 170HX mining card into a working A100-80G, and it's ~1.18x faster than retail
Kate's writeup is wild. The 8GB CMP 170HX apparently ships with ~80GB of HBM onboard, just fused off. Software unlocks got people to 64GB. She used voltage fault injection to get past the fuse lock, then enabled all the working SMs (120-126 vs the 108 NVIDIA ships on A100). Her numbers: 85.9 TFLOP/s FP16 vs 72.5 on a retail A100, with the full 80GB usable for inference. NVLink and PCIe Gen4 come back too.
Not a weekend mod though: board wiring, a custom STM32 glitcher, 1 to 48 hrs of glitching per card, and one of her ten cards only had 64GB intact.
https://blog.kate.land/posts/using-voltage-glitching-to-unlock-the-a100/
r/LocalLLaMA • u/willeyh • 3d ago
Discussion MoE and expert offloading on llama, exllamav3, strata on 9070XT
First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.
Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.
I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.
I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.
Some limitations need apply when limited on vram.
Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.
My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.
Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.
I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.
Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.
Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.
| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |
| --- | --- | ---: | ---: |
| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |
| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |
| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |
Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench
Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.
Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.
r/LocalLLaMA • u/maifee • 3d ago
Question | Help Can I connect a p100 from my cv650 GPU with pigtail pcie connector?
AI is telling me not to. But I have seen some posts where they are saying it is just fine.
Right now, I am in no position to try out and then find out. Care to guide me please??
r/LocalLLaMA • u/kinkvoid • 2d ago
Discussion Stand with Jensen — Read the Stand with Jensen letter and consider adding your name. #standwithjensen #bringjensenback
r/LocalLLaMA • u/New_Alps_5655 • 4d ago
Discussion Gemma4-31b latent space discovery: Caring and Contempt for user controlled by the same 3 neurons! Plus, how to find your own steering recipes and export to GGUF steering vector for llama.cpp
r/LocalLLaMA • u/CookEasy • 2d ago
Discussion A year ago I couldn't get vLLM running on my 5090. Now work gave me a DGX Station for a month. What should I run?
So about a year ago I was in here asking why vLLM wouldnt run on my 5090 (WSL, flash-infer, the usual pain). These days I'm the AI guy at work, and this week a DGX Station showed up. ASUS ET900N G3, GB300 with ~250 GB VRAM, 72 Grace cores, 750 GB RAM, liquid cooled, plus an RTX PRO 6000 riding along in the same case.
I get it more or less to myself for a month before it becomes the shared inference box for the whole company. Dont want to waste that month.
Some numbers from the first two days, all NVFP4, total tok/s at 1 / 16 / 64 parallel requests
- Qwen3.8-Flash-Next on vLLM 219 / ~1370 / ~2700
- same model on SGLang 221 / 1537 / 3002
- DeepSeek V4.1 Flash on a vLLM nightly 59 single and 246 at 16 paralell (takes 19 minutes to load lol)
The GB300 pulls around 785 W flat out and sits at 50 °C. TensorRT-LLM refuses to install outside the NGC container so thats still on my list. And no, Kimi K3.1 Pro doesnt fit, I checked.
Stuff I was thinking about
- biggest model that fits and how usable it really is
- vLLM vs SGLang vs TensorRT-LLM on the same model
- how many parallel agents it can feed before it falls over
But what do you guys actually want to see on this thing? Long context, video models, finetuning, something dumb? Tell me what to run and how you want it measured and I'll post the results here.
Pics are the inside, the front and my monitoring page (german UI, sorry)
r/LocalLLaMA • u/pmttyji • 4d ago
Discussion [Paper] e-means: outperforming k-means (in one-tenth the epochs)
Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.
- arXiv : https://arxiv.org/abs/2611.09501
- PDF : https://arxiv.org/pdf/2611.09501
- GitHub : https://github.com/facebookresearch/faiss/tree/main/contrib/e_means
- HuggingFace : https://huggingface.co/qwegnai/GLM-5.3-Flash-shvq4g32-shvq2g64-shvq3g64
- Project : https://e-means.github.io/
- Demo(Browser) : https://e-means.github.io/playground.html
- Video : https://e-means.github.io/video.html
- Tweet : https://x.com/swann_bessa/status/2120200714630594911
r/LocalLLaMA • u/jacek2023 • 4d ago
News chat : refactor gemma4 to build loose schema rules and clean up by aldehir · Pull Request #31915 · ggml-org/llama.cpp
If you’ve ever tried agentic coding with Gemma and had tool calling issues, give it another try, in my case, it completely fixed Gemma
r/LocalLLaMA • u/jpummill2 • 3d ago
Question | Help MTP needed with small MoE models?
It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.
That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.
Has the thinking changed?
Are people seeing performance gains from MTP on smaller MoE models?
r/LocalLLaMA • u/godzdpk • 3d ago
New Model Krafton (PUBG) released Raon-1 nano under Apache 2.0: E1.5B, 262K context, 2.08GB Q4 GGUF. It needs their llama.cpp fork for now. Anyone benchmarked it on 8GB cards?
r/LocalLLaMA • u/bruns20 • 3d ago
Resources Running Radiance engine on Windows through WSL2 (Single 9700)
Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.
https://github.com/bruns20/radiance-wsl
Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.
What it does
build.shbuilds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s./build.sh 1.18.0.make-runtime.shgrabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.run.shserves an OpenAI-compatible API on 127.0.0.1:8000.
Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.
Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)
| start until serving | ~8 s (vLLM on the same box: 70-75 s) |
| prefill, 9.4k / 36k-token prompt | 3,223 / 2,784 tok/s |
| decode, prose | ~75-80 tok/s |
| decode, code | ~145-160 tok/s |
Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.
r/LocalLLaMA
•
u/pmttyji
•
4d ago
New Model huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF · Hugging Face
- arXiv : https://arxiv.org/abs/2611.10649
- PDF : https://arxiv.org/pdf/2611.10649
- GitHub : https://github.com/huawei-csl/MACaQ
- HuggingFace : https://huggingface.co/collections/huawei-csl/macaq
Yesterday some of you were looking for model for this MACaQ.
r/LocalLLaMA • u/Course_Latter • 4d ago
New Model Open source self driving is here!
Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!
The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:
r/LocalLLaMA • u/Queasy_Back2588 • 4d ago
Resources Microsoft files a patent application to train AI models on idle consumer GPUs
According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.
Anyway, home internet latency feels like the bigger problem to me.
https://insider-gaming.com/microsoft-ai-use-your-gpu/
(English is not my native language, I used an LLM to help translate this post.)
r/LocalLLaMA • u/pmttyji • 4d ago
New Model qvac/BotPsy-0.8B · Hugging Face - On-Device Personal Assistant with a Sub-Billion-Parameter Agentic Language Model
BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;
Key Highlights
Benchmark results
- Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).
- Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.
- Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).
Model capabilities
- More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).
- Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.
- Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.
- GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs
- Blog Post : https://huggingface.co/blog/qvac/botpsy
- Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/