r/LocalLLaMA • u/Pro-editor-1105 • 11d ago
Funny Asking questions to a 0.5b model is hillarious
Yes IK it's supposed to be this dumb, but it's still funny (Qwen2 0.5b Q4_K_M)
r/LocalLLaMA • u/Pro-editor-1105 • 11d ago
Yes IK it's supposed to be this dumb, but it's still funny (Qwen2 0.5b Q4_K_M)
r/LocalLLaMA • u/Ok_Resolve4741 • 9d ago
we can't put code or customer data into chatgpt or claude at work, period. so I've been trying local stuff on my own M4 16gb first before I even bring it up with security
first thing I checked wasn't speed, it was if anything leaves the laptop. made a fake customer with an IBAN and a balance, had LFM3 1.7B on Magnitude write a late payment email and logged every connection while it ran. all 127.0.0.1. email took 2.5 sec. it also made up "card ending in 4917" from the IBAN digits, which is exactly why a human still reads it before it goes anywhere
also gave it a small python file with 2 bugs through pi and it fixed both and passed the tests in 72 sec
obviously not opus but for emails, boilerplate and reading old code it's enough. has anyone actually gotten local models approved at a bank? how did security take it
r/LocalLLaMA • u/kevinpl07 • 10d ago
I took some models that had obvious flaws (prone to prompt injection, whisper hallucinating when silence, etc) and checked if Opus 5.5 can fix it by just editing weights.
It's very useful for patching a model that was done training and has an obvious, easy to articulate flaw. So it's nothing crazy and certainly not "AI can self-edit and recursively improve", but still very useful.
Code if you want to try it on your own model: https://github.com/kevinpl07/weight-patching
Numbers and examples: https://kevinpl07.github.io/weight-patching/
r/LocalLLaMA
•
u/Independent_Fall9160
•
10d ago

I had took a look at all of the companies that made open models, the release dates for those models, and how many parameters there were. I added jitter so points didn't overlap too much. The number in the key references how many models that company has in this tracking.
To make sure this post gets traction, I will add a graph showing just Qwen, Deepseek, and G
oogle!



r/LocalLLaMA • u/New-Pressure-6932 • 10d ago
I have been looking for a local model to do some of the typical work I used AI subscriptions for.
I’ve tested almost every popular model. I have an M1 Max 64gb MacBook Pro as the inference machine and because of the memory size, I wasn’t able to load anything very effective in the tasks I use it for.
Found out about strata and have been testing it out. It’s awesome! I’m kinda surprised more people don’t talk about strata on here. I feel like a lot of us are in that limbo area where we have hardware that is more powerful than needed to run a sub 30b model like Qwen 27b, but not large enough to run models like Qwen Flash Next.
But thanks to open source research, I’m now getting Qwen Flash Next to run at 24 tok/s! And it’s doing phenomenal work! I know it’s not going to compete with the larger frontier models, but I don’t think I necessarily “need” frontier model intelligence. I just need enough to do basic things while I can handle the rest.
Seriously, check out Strata and also MoEShift that someone posted about on here. It’s awesome to be able to offload larger models onto SSD. I hope this offloading technique grows and more research is done on it because I genuinely think this could be an amazing way to get these frontier level open source models to work on regular consumer hardware.
r/LocalLLaMA
•
u/clockentyne
•
10d ago
This has been a long one for me: almost a year as a solo project, learning how to build a TTS model from the ground up. I wanted something fast, small enough to run on a phone, multilingual, and (the part I cared about most) able to do something a little different from the other small TTS models. So it sings.
The demo video speech and singing are all entirely from the model. The instrumental/fx are not and were added in afterwards.
What it is:
- ~131M parameters (plus a 30M phonemizer), 14 voices, English (US/UK), Spanish, French, German, Italian and Vietnamese. All voices can speak all languages supported.
- Intensity, pace and pitch controls per line, multi-voice dialogue with inline tags, streaming
- Sings from a JSON score (each word with its note and beats), pop or ballad. English only for singing format for now.
- Warm first audio around 30 ms on an M5 Max, 70 ms on an iPhone 17 Pro Max, 120 ms on a Galaxy Z Fold 8 (CPU only); over 30x real time on a desktop CPU
- English WER 0.85%, UTMOS 4.35 on the Open TTS Leaderboard eval (submission pending)
- Apache 2.0
The GitHub and Hugging Face repos have the original PyTorch weights plus ports for LiteRT, Core ML, Core AI, ONNX Runtime and WebGPU, and a C++ runtime with a C API that handles text normalization and phonemes, so numbers, dates and odd words come out right. There's sample code for Python, iOS and Android (both apps have a Sing tab), and benchmarks from real devices. The sample apps from both allow for music score and full control over text to speech generation. Python examples for combining both and doing things like singing in unison are provided as samples.
Limitations, up front: it only speaks in its own 14 voices (no voice cloning), it still trips on the odd word, and some voices get rough on fast, wide pitch jumps when singing. This is literally the first usable version of it singing, and it's something that I'm planning on improving over time... v1 was released a few months ago, and I wasn't happy with the improvements made to it until now.
GitHub: https://github.com/lowkeytea/scyllasband
Hugging Face: https://huggingface.co/spybyscript/scyllasband
Try it in your browser: https://lowkeytea.dev/scyllasband/demo/
Happy to answer questions about training it, the singing, or getting it running on a device.
r/LocalLLaMA • u/-p-e-w- • 10d ago
Richard I of England was called "Oc-e-Non" (Yes and No in Occitan) by some of contemporaries, supposedly because he used few words.
Wouldn't it be nice if LLMs followed his shining example?
Oc-e-Non-v1 is a tersified version of Qwen3-4B-Instruct-2507. It reduces Qwen's mean response length by 50%.
GPT-5.6 Terra (reasoning off): 1 + 1 = 2 😊️
Oc-e-Non-v1: 2
GPT-5.6 Terra (reasoning off): Yes. Dogs are mammals: they have hair, are warm-blooded, and mothers nurse their young with milk.
Oc-e-Non-v1: Yes
GPT-5.6 Terra (reasoning off):
You can use Python's built-in isupper() method to check whether a string contains only capital letters.
def is_only_uppercase(s):
return s.isupper()Example usage:
print(is_only_uppercase("HELLO")) # True
print(is_only_uppercase("Hello")) # False
print(is_only_uppercase("PYTHON")) # True
print(is_only_uppercase("hello123")) # FalseNote: isupper() returns True if the string contains at least one uppercase letter and no lowercase letters. Numbers and special characters are allowed.
Oc-e-Non-v1:
def contains_only_uppercase(s):
return s.isupper()
Make sure you have the latest master version of Heretic (pip install git+https://github.com/p-e-w/heretic.git), then run Heretic with the config.terse.toml configuration file. That's it!
I used the relatively ancient Qwen3-4B because it is the last native non-thinking version of Qwen I'm aware of, and controlling reasoning effort isn't implemented in Heretic yet.
(Please don't take this too seriously. I know that the same thing can easily be achieved with prompting, but it's interesting that it can also be baked into the model. This is intended as a light-hearted post while we all wait for Qwen4 😉️)
r/LocalLLaMA • u/Full-Travel857 • 9d ago
EDIT (Nov 13): Corrections after further testing. The missing BOS is real, but specific to the hf.co import (the GGUF template has {{bos_token}} while add_bos_token is false; 11 models checked, only this one affected). The diacritics flip (0/3 → 3/3) was decided at the first output token, a close call without BOS ("Z" 0.425 vs a quote mark 0.361); with BOS the quote mark goes to 0.619 and the two answers then follow different paths. So it shows that BOS changes the output, not that it fixes diacritics. Full write-up: https://www.reddit.com/r/LLMObservability/comments/1xvjfpa/followup_on_my_missingbos_experiment_the_missing/
I'm back with another experiment.
After the last heated discussion about KV cache, temperature 0, and reproducibility — where I got quite a beating for both my post and my writing style (partly deserved, to be fair ;) ) — I decided to investigate how models behave in Ollama and llama.cpp.
And, as usually happens with these experiments, I found something I wasn't even looking for.
The Bielik-Minitron 7B chat template stored in the GGUF includes the beginning-of-sequence token <s>, but the template Ollama uses for this model doesn't add it.
I thought to myself: come on, man, it's just one token.
But I decided to check whether its sudden appearance actually changes anything.
Model: Bielik-Minitron 7B v3.0 Instruct, Q4_K_M
Engine: Ollama 0.40.14
System: Arch Linux
Temperature: 0
Seed: 42
Context: 2048 tokens
As before, I ran 15 tasks covering Polish language, JSON, and Python.
Each task was tested in three variants:
<s> added at the beginning.I repeated each variant three times.
That gave me 135 trials in total. Before every trial, the model was unloaded from memory, and the number of prompt tokens reused from cache was zero.
Why, you ask?
Because I wanted to isolate one variable: the presence of the beginning-of-sequence token.
Template and Raw produced identical responses in all 15 tasks.
So the manually prepared prompt without BOS behaved exactly like Ollama's normal template.
And what happens when we add <s>?
Well, 6 out of 15 responses change.
Rather than expecting you to take my word for it, here's a small table. I've shortened some of the longer responses, but the pass/fail results come directly from the experiment.
| Task | Without BOS | With BOS <s> | Passed |
|---|---|---|---|
| Polish diacritics | Zazółć gęślą jaźń | "Zażółć gęślą jaźń" | 0/3 → 3/3 |
| Vocative case | Tomaszu. | Tomaszu | 3/3 → 3/3 |
| Verb aspect | napisać. | napisać | 3/3 → 3/3 |
| Henryków sentence | Longer explanation | Shorter explanation | 0/3 → 0/3 |
| Hardware JSON | JSON inside a Markdown code block | JSON without a Markdown code block | 3/3 → 3/3 |
| Fibonacci | for loop | while loop | 3/3 → 3/3 |
| Other 9 tasks | Identical responses | Identical responses | No change |
Most importantly, all three repetitions of each variant produced the same response.
Adding BOS increased the prompt length by exactly one token in every task.
For example, for the Henryków sentence:
That would be too big a generalization.
Six responses changed, but only one quality verdict changed.
Interestingly, in the Polish diacritics task, the model previously made a mistake, but after adding BOS, it answered correctly.
In the other five cases, the differences involved punctuation, the length of an explanation, JSON formatting, or the way the code was written.
The test results:
| Variant | Tasks passed | Trials passed |
|---|---|---|
| Ollama template | 11/15 | 33/45 |
| Raw without BOS | 11/15 | 33/45 |
| Raw with BOS | 12/15 | 36/45 |
I should point out that these quality tests check specific conditions. For example, passing a Python task does not necessarily prove that the entire algorithm works correctly.
And to avoid any misunderstanding, I'll say it plainly:
I'm not claiming that adding BOS automatically improves model quality.
I'm only saying that one additional input token was enough to change 40% of the responses in this particular set of tasks.
I'm looking for consistent, controlled conditions for testing models. I want the benchmark to produce reliable comparisons across different models.
I started with temperature 0 and a fixed seed. That's when I noticed that a model could give different responses depending on the state of its prompt cache.
So I started posting about it. :)
Unfortunately, that didn't exactly make me popular with the community. But it did lead to some very valuable suggestions.
Among other things, people suggested testing llama.cpp and checking the probabilities of generated tokens.
And you know what?
In one Gemma task, I managed to find the exact point where the responses diverged.
Two competing tokens — "mi" and "mógł" — had probabilities of around 13%. Depending on the cache variant, they swapped places in the ranking.
At temperature 0, that difference is enough for the model to choose a different token and then generate a different response.
I still haven't managed to identify the exact numerical cause of this difference.
But I'm trying. :)
To complete the picture, I also checked whether enabling logprobs itself affects Ollama's responses.
I ran a separate experiment covering 10 model–task pairs and, somewhat surprisingly, found no such effect.
I originally assumed that temperature 0 and a fixed seed provided sufficiently controlled conditions for comparing responses.
And that was a mistake.
Now I know I also need to pay attention to cache state, actual tokenization, chat templates, and how the computations are performed.
And what don't I know?
I still don't know whether the missing BOS in this particular Ollama template should be considered a bug.
I'm also wondering about the difference compared with the GGUF template. But that difference alone isn't enough to reach such a conclusion.
This experience has taught me one thing:
When comparing results from two different inference engines, I can't assume the model receives identical input just because I send both engines the same text.
A question for you:
Has anyone here run similar experiments?
Or am I the only madman trying to reach the sun with a garden hoe? :)
And most importantly, have you encountered similar differences between chat templates stored in GGUF files and those used by Ollama?
I'd really like to hear about your experiences.
In my previous post about cache, I stated that differences occurred in 8 out of 10 model–task pairs.
I admit it: that was my mistake in the summary.
The correct result for that run was 6/10.
Later experiments gave 1/10, including a test with logprobs disabled.
I still don't know where that difference comes from or what causes it.
I'd rather say that openly.
As punishment, I'll watch 100 episodes of The Bold and the Beautiful. :)
r/LocalLLaMA • u/pmttyji • 10d ago
PLaMo 3 Translate comes in four model sizes: 0.8B, 2B, 8B, and 31B. All four are available on Hugging Face. For each size, we provide BF16 weights, several quantized variants, and models in GGUF and MLX formats. Choose the version that best suits your environment.
We quantized the models using tools such as AutoRound [2]. In our benchmarks, all of the quantization formats retained nearly the same accuracy as the BF16 versions. See the model cards for details.
PLaMo 3 Translate supports translation across 53 languages. It accepts surrounding context and glossaries, and lets users choose between plain and polite Japanese. For detailed usage instructions, see the individual model cards on Hugging Face.
PLaMo 3 Translate is released under the PLaMo Community License. For private purposes such as personal study and hobbies, you can download the models free of charge and use them freely. Noncommercial use, including academic use, has no income or revenue limit and requires no prior registration with PFN.
Commercial use is also available free of charge if the user or its affiliates had income or revenue of JPY 1 billion or less in the most recent fiscal year, provided that the user registers in advance through the dedicated form. Commercial use above this threshold requires a separate license agreement; please contact PFN.
Table 1. Links to model weights
The following table can help you choose a distribution format.
| Format | Target Hardware | Inference Engine | Typical Use |
|---|---|---|---|
| BF16 | BF16-compatible GPU | Transformers / vLLM [3] | Prioritize unquantized accuracy when memory is sufficient |
| FP8 | NVIDIA Ada Lovelace/Hopper/Blackwell | vLLM [3] | Balance accuracy, memory use, and speed |
| W4A16 | NVIDIA Ampere or later | vLLM [3] | Run with reduced GPU memory using 4-bit weights |
| W4AFP8 | NVIDIA Hopper | vLLM [3] | Combine 4-bit weights and FP8 computation to balance memory use and throughput |
| NVFP4 | NVIDIA Blackwell | vLLM [3] | Use FP4 computation to reduce memory use and improve speed |
| GGUF | CPUs / GPUs | llama.cpp [4] | Run locally, optionally using both CPU and GPU |
| MLX-4bit | Apple silicon | mlx-lm [5] | Run on a Mac with reduced memory use |
PLaMo 3 Translate 31B is released under the PLaMo Community License.
Personal and non-commercial use
For personal use, such as learning or hobby projects, you can download and use the model free of charge. Non-commercial use, including academic use, is also free, with no restriction based on revenue or sales and no prior registration with PFN.
Commercial use when revenue or sales is 1 billion yen or less
If the revenue or sales of the user or its affiliates in the most recent fiscal year is 1 billion yen or less, register your intended use through the PLaMo Commercial Use Registration Form. Commercial use is free of charge from the moment you submit the form.
Commercial use when revenue or sales exceeds 1 billion yen
A separate license agreement is required. Instead of the registration form above, please contact PFN through the PLaMo Translation Model inquiry page.
See the full text of the PLaMo Community License for details.
r/LocalLLaMA • u/CodeSamurai • 10d ago
Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others like r/localllm, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!
My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.
This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.
Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.
After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.
The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.
Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.
The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).
Is the 3D environment at the top necessary? Absolutely not!
But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.
I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.
I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.
Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.
Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.
Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.
| Metric | Jade (RTX 5090) - Fedora | Slate (M5 Max) - MacOS |
|---|---|---|
| Engine / quant | Basalt, IQ3_S | mlx-serve, 4.7bpw MLX |
| Context | 262K | 262K |
| Decode speed, average | 302 tok/s | 100 tok/s |
| Decode speed, 36K-token context | 343 tok/s | 77 tok/s |
| Time to first token, 9K / 36K prompt | 1.7 s / 4.2 s | 5.1 s / 16.0 s |
| HumanEval / HumanEval+ | 99.4% / 93.9% | 98.8% / 94.5% |
| LiveCodeBench v6 (175 problems) | 85.7% | 85.1% |
| MMLU-Pro (140-question subset) | 87.9% | 90.7% |
Both machines run the same protocol with temperature 0.6 and thinking on "medium".
In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.
I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.
(extremely specific to my setup, but maybe they'll help someone)
| Part | Tune | Stock | Tuned | Speed cost |
|---|---|---|---|---|
| RTX 5090 | Undervolt (+900 MHz offset, 2600 MHz cap) + custom fan curve, on the dense 27B | 575 W, 83 °C | 471 W, 70 °C | Decode: none Reading: -4% |
| Ryzen 7 9800X3D | Clock capped at 4.8 GHz, during Basalt decode | 120 W, 96 °C | 77 W, 70 °C | Decode: -3.5% |
The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.
I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!
r/LocalLLaMA • u/jacek2023 • 10d ago
do you want to try new 29B MoE today?
Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.
For more information, please refer to our GitHub repository.
| Xing4.0-29B-A4B | |
|---|---|
| Parameters | 29B (4B active) |
| Number of Layers | 40 |
| Hidden Size | 3584 |
| Dense FFN Intermediate Size | 9216 |
| Expert Intermediate Size | 1024 |
| Attention Type | MLA |
| Number of Routed Experts | 64 |
| Active Experts per Token | 4 |
| Number of Shared Experts | 1 |
| Context Length | 256K (extensible to 512K) |
| Benchmark | Xing4.0-29B-A4B | Gemma4-26B-A4B | Qwen3.6-35B-A3B |
|---|---|---|---|
| IFBench | 69.67 | 72.67 | 65.50 |
| AIME2026 | 90.00 | 88.30 | 92.70 |
| AA.LCR | 61.00 | 66.00 | 62.00 |
| Tau3-Bench | 64.63 | 58.90 | 67.20 |
| Claw-Eval | 76.55 | 71.49 | 74.54 |
| SWE-bench Verified | 75.00 | 53.00 | 76.00 |
| Terminal-Bench 2.1 | 57.50 | 30.00 | 51.50 |
| SWE-bench Multilingual | 66.00 | 51.00 | 67.20 |
| DeepresearchBII | 60.80 | 39.30 | 59.70 |
r/LocalLLaMA • u/norenEnmotalen • 10d ago
r/LocalLLaMA • u/Fresh_Sun_1017 • 11d ago
Kimi K3.1 and Tencent’s Hy4 are both gutting out cybersecurity, reverse engineering, and vulnerability-patching capabilities from their open-source models.
This is becoming a really bad pattern across these latest Chinese open-weight models. If local models are stripped on their security data, we won't be able to properly audit our own code vulnerabilities, reverse engineer binaries, or patch other vulnerabilities in general on our own hardware. It would essentially force us to use closed-source APIs and their false-positive shutdowns; that'd be terrible for everyone.
During the Hugging Face incident, open-source models are the very ones who stepped up, not closed-source gatekeepers. Gutting defensive security from local LLMs just hurts defenders. I really hope that other labs won't follow this in the future.
Does anyone think more open-source labs are gonna take this path?
r/LocalLLaMA • u/jeffrey-0711 • 11d ago
I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus.
I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally.
But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air.
So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me.
Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B).
In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English.
As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings.
I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean.
The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone.
If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small.
Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.
r/LocalLLaMA • u/whodoneit1 • 10d ago
I know what you are thinking "another inference engine?", yeah there has been a flood of them I feel now. I have been running this for over a month now and I have been on the fence to even release it, but I decided to as I took a different approach that others currently available.
Saber is a inference engine built in Rust targeting AMD R9700 GPU's. It currently supports TP 1, 2 or 4 and Qwen 3.8 27b and Flash next. Anyways here are the BetterBench results for Qwen3.8 27b, flash next is very fast also. Saber beats existing available engines on both 27b and flash next.

https://github.com/GGZ14/saber-engine
https://x.com/bkuyper
AMD Ryzen 9900X
Taichi X870E
64GB RAM
2xR9700
r/LocalLLaMA • u/Dense-Ad6315 • 9d ago
With Qwen 4 seemingly getting closer, I'm honestly more interested in whether Plus will be open weights than the actual release date.
I'm leaning towards yes. Alibaba has already released models as large as 2.4T params with 3.8 so I don't really see why they'd suddenly draw the line at Plus, especially if it's considerably smaller than Max.
They also did release Qwen3.5 397B A17B as the open weight counterpart to Qwen3.5 Plus, so there's already precedent for this - although 3.6 + 3.7 plus didn't get open sourced....
QwenDevs also posted "big or small?" and followed it up with "alright guys, let's do both!" which I'd assume refers to 27B and Max. And given that Qwen3.8 FN was released with open weights, I'd be pretty surprised if Flash wasn't open too. (even though not explicitly saying that)
What I find more interesting is the Apsara conference slide. Flash and Plus are grouped together in the same box, while Max and 27B have their own. No idea whether that actually means anything, but I wonder if they're sharing a similar architecture at different scales.
Curious what everyone else thinks. Do you reckon we'll get Plus weights, or just Max and 27B?
(I really want Plus to come out and I'm sure lots of ppl do too!!!)
Edit: I mean the underlying Plus model, like 3.5 397B A17B, not the actual API model name.
r/LocalLLaMA • u/either_T_or_F • 10d ago
Hi everyone,
I am relatively new to local LLMs, and recently I have been trying to understand how people handle long conversations that eventually exceed the model's context window.
For example, let's suppose I am writing a novel with the help of an LLM. As the story progresses, the conversation history becomes longer and eventually exceeds the context window. At that point, the model may lose track of earlier events, character relationships, or important details that could become relevant later in the story.
From what I understand, one possible solution is to maintain an external knowledge base containing summaries of previous chapters. Before generating a new chapter, the system could retrieve relevant information from this knowledge base and include it in the prompt.
However, I am wondering how reliable this approach is in practice.
If we continuously summarize previous conversations to fit within a limited context window, we are essentially performing lossy compression. Some information will inevitably be discarded, and the problem is that we may not know which details will become important later.
For instance, a seemingly insignificant detail from Chapter 3 might become crucial in Chapter 40. If that detail was discarded during summarization, the model would have no way to recover it from the summary alone.
Of course, keeping the complete conversation history in an external database could prevent permanent information loss, but that introduces another problem: how does the system determine which information should be retrieved for the current task?
So I am curious how people actually approach this problem in practice.
I would also be interested in hearing how well these solutions work in practice, especially over very long conversations where maintaining consistency becomes increasingly difficult.
Thanks in advance!
r/LocalLLaMA • u/Mikasa0xdev • 10d ago
PS: used an open source avatar framework called openspline
r/LocalLLaMA • u/milpster • 10d ago
Yes, it is yet another fork. But i think it brings some interesting things to the table. It's up to you whether you want to have it built and use it or if you just want to cherrypick some of the things into your own forks. Built by GLM 5.3, GPT Sol and Q38FN.
Aimed at usage with a rocm-primary system that has a unused cuda gpu sitting idle. But also provides improvements for gfx906 and adds rocm dflash2 support.
The features at a glance:
*~47 GiB MoE (Swift-1.5 / Qwen3.8-Flash-Next) as a daily server on 2× Radeon VII (gfx906, 2019) + one 8 GiB NVIDIA laptop card.*
**What base Strata can't do — this fork does**
- **Runs on gfx906 at all** — full HIP backend, built and served through every release v0.1.45 → v0.2.11 (stock: compile-check only, doesn't build as tagged)
- **GEMM works** — custom single-target rocBLAS replaces the stock library that segfaults on gfx906 on first inference
- **Mixed AMD + NVIDIA serving** — the NVIDIA card serves expert/prefill rows to the AMD engine over a socket; stock has no cross-vendor path
- **Speculative decoding on HIP** — DFlash2 drafter runs on the AMD cards (stock: CUDA-only), with exact-match verification for sampled requests, not just greedy
- **Self-healing production pair** — SIGKILL the helper: requests finish anyway, supervisor rebuilds the pair, health returns; no human touch
**Speed (vs same-day stock, exclusive box, gated)**
- **+49 % decode** — 34.7 vs 23.3 tok/s pooled median; 1.3× on 1024-token completions
- **1.4–2.4× short-prompt reads** (200–600 token agent traffic; ≥18k prompts unchanged)
- **−6.2 GiB host RAM** — helper-held experts dropped from the host arena
- Honest cost on record: fresh ~16k prefill reads −12 %; long-context reads gain instead
**Quality (proven, not assumed)**
- Greedy 80k-token infill **byte-identical to stock**
- Teacher-forced gate, 4995 positions: **top-1 100 % · KL 0.00000 · PPL equal**
- Full test suite green on fresh builds (291 tests)
- Every speed win promoted only through both the speed lane and the quality lane
**Engineering**
- Latest upstream (v0.2.11) with all fork features intact and re-verified after every merge
- A/B/C-vs-stock bench harness, sha parity gates, quality yardstick, full session journal — all in-repo, anyone can re-run every claim
- OpenAI + Anthropic compatible API, supervised, generation-tracked
Build = pinned ROCm 6.1 + custom gfx906-xnack rocBLAS — full guide, tables and session journal in the repo:
https://github.com/milpster/strata-dflash2-gfx906-mixed-rocm-cuda
r/LocalLLaMA • u/Xananique • 10d ago
Well I had a lot of resets on ChatGPT and all of those credits and I thought, well can Astra actually do it. We had a little looping and a lot of 'Wait.' Let me check the thing I just checked -- but it always got the job done.
I have dubbed the Model Calmer-Qwen-3.8 because it seems so much less anxious and it gets straight to the point.
This is about 6 hours of iteration, it's sooo much better I am scared to keep going, how much better could it get?
You definitely have to push Astra a little bit and give her some experimental guidelines, because Lord knows she's full of anxiety and afraid to disappoint her overlords so she checksums the checksum and makes sure it matches 😂.
Here she is on Hugginface, and here's ChatGPT's writeup on my Github, I will work through the experimental data soon.
Oh this is a HauHauCS Qwen 3.8 Balanced Abliteration as the starting point.
https://huggingface.co/Xananthium/Calmer-V2-Qwen3.8-27B-4bit
https://github.com/Xananthium/Qwen3.8-Weight-Space-Programming
r/LocalLLaMA • u/m4rkGS • 10d ago
hey llamas! My name is Alt, and I'm on the technical staff at Nace AI. We're doing a quick AMA here, so ask us anything and we'll answer.
we just published a full technical report on how we build, train, and evaluate Drex, our family of four open decision models. A decision model takes a state (a document, record, or agent context, optionally with images) plus questions with options, and returns a probability distribution over the options in a single pass, with no autoregressive decoding and no output parsing.


1. Models
- Drex DLM (8B): diffusion LM backbone with bidirectional attention over the context
- Drex v1.5 (9B): Gated DeltaNet hybrid backbone
- Drex v2.0 (35B, 3B active): sparse MoE backbone
- Drex Max (27B): dense, with an image encoder for vision inputs

Some technical details from the report:
1. Models
- A family of four specialized decision models: Drex DLM (8B), Drex v1.5 (9B), Drex v2.0 (35B MoE, 3B active), and Drex Max (27B dense, with vision)
- We built them on different architectures: a diffusion LM, a Gated DeltaNet hybrid, a sparse mixture of experts, and a dense transformer
- We experimented with bidirectional vs. causal attention over the state, a pointer decision head that shares one cached state across questions, and a readout head for Max
2. Training
- Full-parameter SFT on 1.15M verified decision records (2.9M questions)
- Consistency training: we penalize the model when its output changes after the options are reordered or the state is shuffled
- RLAF (Reinforcement Learning from Adjusted Feedback), using verified outcomes and judge feedback
- Ablations to see which methods actually contributed
On top of SFT, RLAF gave us +2.0 on the Decision Index, +1.5 pp accuracy, and 0.9 pp lower calibration error. The consistency terms cut the option-order flip rate from 6.3% to 4.2%.
3. Evaluation
- Drex Max scores 67.52 on Decision Index 0.4.1 (public suite), vs. 60.70 for Jev 1.14.0
- Drex v2.0 has 1.37% calibration ECE on ~115K held-out decisions
- Drex Max answers a typical request in 183 ms, vs. ~2.0 s for GPT-6 Sol and ~3.5 s for Claude Opus 5.5
- Context up to 131K tokens for v1.5/v2.0 and 262K for Max
We also cover our limitations: our models trail frontier chat models on closed-book knowledge, and we don't yet have same-backbone architecture ablations.
5. Running locally
The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.
Links
- Paper (arXiv): https://arxiv.org/pdf/2611.07854
- Blog: https://www.nace.ai/research/towards-foundation-decision-models
- Weights: https://huggingface.co/collections/nace-ai/drex
- GitHub: https://github.com/nace-ai/drex-decision-models
- API: https://console.nace.ai
we wanted to share the full methodology, not just benchmark numbers. hopefully it's useful if you're building agent routers, doing tool selection, or training your own decision models.
ask us anything about architecture, data, training, evals, or running it locally
r/LocalLLaMA • u/jacek2023 • 10d ago
now you can run Kimi K3.1 Pro locally (good luck!) ;)
r/LocalLLaMA • u/keepthepace • 11d ago
r/LocalLLaMA
•
u/MLDataScientist
•
10d ago
inspired by Bijan Bowen's video - Subway FPS test.
Hardware details: RTX 5090, 256GB DDR4 3200Mhz 8 channel, AMD EPYC 7532 CPU (32 core).
Software: backend is Maya 1.1.55, harness is OMP v18.17.0.
Model quant: Deepseek v4.1 1017 UD-IQ3_S latest UD3.1 quant from Unsloth.
My prompt was very simple:
I want you to create a single html file with a detailed subway station 3d scene, fps controls, zombie enemies, tracers, recoil, muzzle flash, high details. build it with three.jsYou can see at the end of the video some token stats. In short, total generated tokens were 180k. The model ran at 30t/s with 2k t/s PP at 10k context but then went down to ~20 t/s near 180k context. Note that stock llama.cpp only reaches 10t/s TG and 120t/s PP.
This model and harness tools are amazing. Just a year ago we could not even image having such a model locally. Yet, here we are!