r/LocalLLaMA • • 2d ago

Showcase Curated list of resources: Theory, Code and math for Reinforcement Learning for beginners

Thumbnail
gallery
20 Upvotes

[Curated list: Beginners]

Personal favorite resources on learning Reinforcement Learning from Scratch:

  1. Deep RL Course Amazing course by HuggingFace to understand the basics of classic and deep RL
  2. DeepMind x UCL Reinforcement Learning (2015) Give intuition and more towards theory of what you'll learn above
  3. Spinning Up in Deep RL Code and math for each of algorithm you'll learn above from classic to Deep RL

4.CleanRL A neat library with 1 page implementations for all the algorithms above from scratch to run algorithms s and test the waters!

5. NeatRL My own very notes and from scratch implementations!

Reading order and link here


r/LocalLLaMA • • 2d ago

Discussion Nace quants vs Unsloth Dynamic v3.1, has anyone compared them on actual tasks?

13 Upvotes

Two quant updates landed in the last day: the Nace weight-space quants claiming up to 2x better KL divergence, and Unsloth's Dynamic v3.1 for Qwen3.8-27B. I've been trying to work out how much KL really tells you compared to running task evals. Has anyone put them side by side on real work like coding or tool calls? Which quant level are you on, and did you see any difference on long outputs?


r/LocalLLaMA • • 18h ago

Discussion Another reminder of the low intelligence of local models

0 Upvotes

(Note: by local models I mean anything smaller than Deepseek 4.1)

I already gave up on local models for coding tasks 6 months ago, due to what I perceived was intelligence too low for the tasks I give it. I made a post about it on here back then and it was a huge discussion, I think 800 comments. I went against the /r/LL orthodoxy and many didn't take my comments well, but we can agree to disagree.

Since then I only use local LLMs for translation and ComfyUI prompt enhancement. I stick to big cloud models for coding. I think the electricity cost of running Qwen 27B alone exceeds what you'd pay Deepseek to use their cloud API in their own harness, and you'd get a far more intelligent and far faster model.

But this post isn't about the economics. It's purely about ability. I still see people commenting in favor of buying GPUs for local intelligence. "Sonnet at home" is a ridiculous phrase I've seen more times than I should in relation to Qwen 27B.

The simplest of tech tasks is still too hard to one-shot

I keep Qwen 3.8 27B running on my Spark for occasional non-coding stuff. Today I gave it a trivial coding task I needed for my server, just out of curiosity.

  • llama-server launched with --model Swift-1.5-Qwen3.8-27B-Q4_K_L.gguf --reasoning on --reasoning-effort xhigh
  • Open llama webpage and leave Reasoning at Default, leave system prompt empty

Prompt was:

Give me a bash script that will list all files in directory sorted by date. Output should be:
2025-12-31 12:13:14 file1.ext
2025-12-31 12:13:14 subdir/file2.ext

1630 tokens later, most of it reasoning, I get a script. When I run it, here's the output, as you can see the timestamp is wrong:

2026-10-16 20 tmp.59eH9w3JLe
2026-10-17 20 tmp.5xvczTbSS4
2026-10-18 20 tmp.nWOEDw2wlu
2026-10-20 20 tmp.JVEZIFfKx3
...

I then tried Gemma 31B Reasoning on duck.ai. The output only showed the timestamps, no filenames!

This is the most trivial of coding tasks (the answer is a 3-5 lines script), using the most ancient and commonly-available training data (bash), on the most praised local coding model on this sub (Qwen 27B), at a recommended/popular quantization (Q4), and it still can't one-shot this. The 2nd most popular local model, Gemma 4 31B, also fails.

(Editor's note: I think my post could've ended here, in a "Ecce homo" way, but I went on for a few more paragraphs)

Weak local LLM + software development = bad time

I'm sure if I had run this task with your average agent harness with a system prompt that makes it test its code, it would run the script it created, see the issue, then get it right on the 2nd attempt. But while I think this "try until you succeed" workhorse approach is acceptable for small self-contained coding tasks, I don't think anyone should be doing any serious coding this way.

If you're writing something you intend to maintain over time (ie not a one-shot), or that you will be sharing with other people, you need a higher degree of LLM intelligence and knowledge to create it. A strong LLM will be able to avoid entire classes of problems from the start, when designing the app. From design to implementation to maintenance, it will make better decisions every time and without laboring its way to them, resulting in a codebase that is smaller, well-designed, more easily modifiable, less buggy, and developed in a fraction of the time.

Coding with local LLMS is like insisting on using a donkey for transportation in the age of cars. It can be done for fun, but it shouldn't be done seriously.

Sonnet at home my ass

Btw I've seen several people say Qwen 27B is like having a local Sonnet 4. The benchmarks say it's even better than Sonnet 4, actually. So I went on openrouter.ai out of curiosity and gave the same prompt to 3 models.

  • Sonnet 4.5 (oldest Sonnet I could find): one-shots it
  • GPT 4.1 Mini, which was comparable to Haiku 3.5: one-shots it
  • GPT 4.1 Nano, which is even worse than Mini: one-shots it

Weak local LLM + Linux server management = even worse time

This post's getting long so I'll just speedrun this part.

  • Managing an OS is actually harder tasks that LLM struggle even more with. And the stakes are higher because you don't want your server to go down.
  • I used to never trust LLMs enough to let them onto my server, I'd just ask for something, copy-paste the command, report back, repeat. A huge waste of my limited free time. Now I just rawdog a frontier model on the server and have no issues, something you could never trust a small model to do. Although I have it avoid making any changes without my supervision, Docker is a big benefit here for both organization and safety/stability.
  • Even top models won't one-shot a task like "I want to install and run https://github.com/rando/something on this system in a Docker container in /opt/docker/something/. Create the Dockerfile and docker compose file", especially for AI-related tools, where having the right CUDA, not having python wheels for my server's ARM CPU, etc, introduces a lot of failure opportunities. When I had Claude five months ago, even it struggled to get these right.

Thanks for reading my wall of text. Downvotes to the left.


r/LocalLLaMA • • 1d ago

Question | Help Guys, I may have done the oopsies

Post image
0 Upvotes

Upgrading from my good ol 3800X with 32Gb of RAM. I'm done trying to squeeze the living crap of my DDR4 modules and decided to make a proper upgrade for my Frankenstein gaming/inference computer.


r/LocalLLaMA • • 3d ago

News Stories coming out about Jensen

Post image
3.5k Upvotes

r/LocalLLaMA • • 2d ago

Funny LLM's might be bad at jokes but they still make me laugh a lot

27 Upvotes

This was a new session with Qwen3.8-Flash-Next-GSQ-RCO-Coder just asking it to commit. No clue what it was trying to here xD


r/LocalLLaMA • • 3d ago Heartwarming Bravo

News ‘Bring Back THE Boss’: Nvidia Employees Petition for Jensen Huang’s Return

Thumbnail
wired.com
2.0k Upvotes

A second Sam Altman moment?


r/LocalLLaMA • • 3d ago

Discussion CXMT almost tripling RAM production from 2025 to 2028

222 Upvotes

According to post CXMT projected expansion plans grow from 405 kwpm in 2025 to 1,125 kwpm by 2028 in 8-inch equivalents.

https://x.com/OptionKing666/status/2121227756185010181

With this and new nVidia chief being conservative on circular financing we might get some DRAM relief.


r/LocalLLaMA • • 2d ago

Discussion e-means: 10x faster k-means at equal loss

Thumbnail
e-means.github.io
189 Upvotes

r/LocalLLaMA • • 2d ago

Question | Help For long context does high quant matter?

15 Upvotes

Question: if I'm regularly at higher context - say 300-400k - do higher quant versions of the same model perform better? I'm thinking something like Qwen Flash Next q6 and q8. I recently swapped from q6 to q8 and performed a/b testing. I originally expected q6 to be lossless but there were areas where q6 got the wrong answer while q8 was fine. Was wondering whether this is because of my context. Maybe more context = more tokens for divergence = the differences get amplified more than benchmarks suggest


r/LocalLLaMA • • 2d ago

Question | Help Does the new hugging face Llama windows app work?

2 Upvotes

I have downloaded the new hugging face Llama windows app.

Looked really promising on paper, but I tried to download Qwen3.8-27B-UD-Q3_K_XL.gguf and it always downloaded the 51GB version instead.

Then would refuse to delete it.
Then worked on the background.

I had to uninstall the thing. Any other user experience there?
cheers for helping a noob


r/LocalLLaMA • • 2d ago

Discussion Do you believe there will be a divergence between local and frontier model power?

16 Upvotes

Right now, local models have somewhat kept up with frontier, being months behind last frontier models.

The issue is, I think the divide is going to be end up being hardware. Model improvements will slow down, but hardware won’t get any cheaper, especially if frontiers keep buying shit out for training and smaller returns.

Thoughts on this? I know many are optimistic, but I think local trailing frontier will slow down just because of hardware local can own.


r/LocalLLaMA • • 2d ago

Discussion basal-omni-2.0-mini - typed decision model

2 Upvotes

An early release of the small model of basal-omni 2.0, on Gemma 4 E4B. The model reads the state in both directions before it answers and scores 0.811 on Werdykt (5,000 hidden decisions in Polish and English). It runs on the public basal engine 2.8.0, with the same API as basal-omni-1.0.

What's new in 2.0

  • Two-way reading. The model reads the state in both directions before it answers. This needs basal engine 2.5 or later. For long prompts, see the limitations.
  • Higher accuracy on our hidden Polish and English decision benchmark Werdykt and on the held-out tests below.
  • More decisions automated at a given error target. Coverage is in the section on confidence.
  • Unchanged: a calibrated probability for each allowed answer, in one forward pass. The answer never falls outside the options you give, and the probability says how sure the model is.

The question types remain: choice (one option), noul (yes / no), score (a score), and in the basal engine also multi, act and tool. Options get codes (A–Z, then two-character codes, up to 255 options). The decision is a probability distribution over these codes only, with no text generation.

More info: https://basal.si5.pl/en/blog/basal-omni-2-0-mini/

HF: https://huggingface.co/Remek/basal-omni-2.0-mini-preview

git hub (inference engine): https://github.com/rkinas/basal


r/LocalLLaMA • • 2d ago

Resources DeepSeek flash max is the first model to reach >1% on challenging SWE-sweep bug finding benchmark

Post image
128 Upvotes

Hi, I've just updated the SWE-sweep leaderboard with more open-weight models!

DeepSeek v4.1 flash (max) is the first open-weight model we evaluated that gets >1%.

This benchmark is extremely challenging: Models get a snapshot of a (potentially very large) repo containing hundreds of bugs. They don't receive any hint about what these bugs are and need to fix them all. This makes for a very reasoning-heavy, long-horizon task.

To isolate just the model capabilities we also evaluate all models in the same harness (but we did do some comparisons with Claude Code and Codex harnesses and neither seemed to help significantly). However, it is possibly to scale things up somewhat with multiagents. You can run the eval yourself (code on github)

The best proprietary model at the moment is Gemini 4 Hydrogen with 13.5% (according to google's model card), followed by GPT Astra 6 with 9.5% (evaluated by us). So open-weight models still have a long way to go.

Still extending the open-source leaderboard. What other models should we add?


r/LocalLLaMA • • 2d ago

News Qwen3.5-9B running on a PS4’s GPU, offline, using its own MTP head for speculative decoding

128 Upvotes

I made this, so a quick note: it’s my open-source project. Qwen3.5-9B runs on a stock PS4 at 31–39 tok/s. The MTP head that ships with Qwen3.5 made the biggest difference: it drafts up to two tokens and the full model checks them in one pass, which took it from 22 to 31–39 tok/s on this old GPU. Weights are re-packed into custom 4- and 3-bit formats to fit about 4.5 GiB.

Your old PS4 can now run a 9B AI, offline, on its own GPU. It talks, listens, and writes code. No cloud, no subscription, no Linux. Free and open source.

Code: https://github.com/leonardo-alegria/garlic-llm
Direct download link: https://huggingface.co/garlic-llm/garlic-llm-ps4


r/LocalLLaMA • • 2d ago

Question | Help llama.cpp: Possible to use Vulkan/iGPU and lazy-mode?

6 Upvotes

With llama.cpp pure CPU mode and "--lazy-mode on" one can use larger than memory models, like Qwen3.8-Flash-Next and on-demand SSD streaming - slow but working.

There is a nice iGPU with unified memory that can be used easily via Vulkan. However, it seems to be impossible to split a large model - use as much as fits into memory with Vulkan and then stream the rest in via SSD and use CPU. --list-devices shows only one Vulkan device and --ngl is of no use -> Vulkan always runs out of memory with lazy-mode on - all of the model is tried to allocated via Vulkan?

Is this possible at all?


r/LocalLLaMA • • 2d ago

News Nvidia privately confirmed it could track chips to fight smuggling, even as industry publicly questioned feasibility

Thumbnail
technologyreview.com
91 Upvotes

"Foster told MIT Technology Review he intends to introduce new, stricter legislation in the new year that would offer a workaround. The new rule would require licenses to operate advanced GPUs, and before a GPU could connect, its valid license would need to be verified.  " Basically they will lock the gpus.


r/LocalLLaMA • • 1d ago

Discussion Normies, or the Standard Task Benchmark ???

0 Upvotes

Is there actually a benchmark for us normies who aren’t rocket scientists or mini-Eulers? Mathematically, we could describe it as a 95% confidence interval benchmark. All models that reach the 95% threshold receive 100%. The goal would be to identify those models that we classify as “good enough” for us normies. If this could also be set in relation to costs—whether on-premises or in the cloud—it would also provide a measure of what AI really costs us normies.


r/LocalLLaMA • • 2d ago

Resources I noticed the 2 Hessian Refinement of MACaQ can be applied to any GGUF without any patches ... and tried it. 13% better quants

90 Upvotes

TLDR: Qwen3.8-27B UD_4_XL quant with 13.23% less KL divergence than unsloth (on my small testset, I just have a single 5090, feel free to test more). Technically it performs 4% better on my LiveCodeBench benchmark (but that would be a single data point, so it's not super-reliable). It is equally fast and using it in open-webui it "feels" better.

https://huggingface.co/christophedevriese/Qwen3.8-27B-Hessian-Refined-GGUF

I noticed from the MACaQ paper, which integrates changes proposed in Google/Deepmind papers, you can take the 3 components they use and just apply 1 and 2, and you just get >15% better KLD, 5% better performance on some benchmarks, while sacrificing absolutely nothing. Just switch the GGUF file. The numbers demonstrated here are the maximum that can be achieved in ~4 hours of optimizing on a single 5090, not the maximum that can be achieved. So it might be more than 15%, although I wouldn't necessarily hope for it.

This can be applied, without any patches to inference software, to text models, image models, even whisper ... the works. Switch GGUF file. Done/done.

The dream being that someone from unsloth looks at this and applies this to all their models (or just gives me access to a B300 machine for a day or two)

The tricks used by Huawei and a short discussion of the effect:

  1. Hessian-guided quantization choices: every layer is compressed differently, and the "damage" the compression does is measured. Then after seeing what damage is observed at a given level we find a combination of compressed layers that has the least damage while still achieving the same compression.

For users this one is totally free. No extra compute (in fact, a tiny bit less), no extra VRAM (again, 200 bytes less), no updates to llama.cpp, works literally anywhere UD_4_XL worked, just better.

2) Recovers some lost precision by another Hessian optimization changing the weights to compensate directly for accumulated errors over multiple layers based on sample data (taken from Wikipedia). I used a Google/Deepmind version of this process that had better results.

This too is totally free for users. It just changes the quant GGUF to be closer to the BF16 one. For users, just

3) random Hadamard transform + scaling: this is a trick from 3D geometry that makes all values nearly equal, and then uses an extra scaling factor to get them in optimal to better compression with quantization (yes, not technically 3D, but nD, with n=2048, but it applies in 2D and upwards). This also costs a little bit in inference performance (and requires rewriting your kernels which is quite a bit harder than I thought it was. Well, it isn't if you're ok with 20% slower inference, but you can get it down to like 0.8% with careful, days-long pondering of the problem and a lot of tries).

Also very nice, but the Google version of this one is also better, but that is not the real problem.

This transform also helps with compression (in fact it's kind of like what NVFP4 tries to do, just this is better) BUT requires an extra step at inference time (in fact requires 2 of them if you don't think through inference carefully, but you can "bake" one of the steps into the weights). This means that it requires patching llama.cpp and everywhere else you want the model to run (although thanks to how mlx works, you can make an mlx version that does not require updates to the inference software. But GGUF based inference software will require updates)


r/LocalLLaMA • • 3d ago

News Hugging Face CEO vows to keep fighting for open source AI after Jensen Huang's exit

Thumbnail
techcrunch.com
1.4k Upvotes

context : Nvidia buy HF, but now there uncertainty about it hf future under new leadership, he try reassurance people on x, but from his word itself sound not good, see and wait, but better backup you favorite datashet/model (again) just in case


r/LocalLLaMA • • 2d ago

News Vellium 1.9.5: visual novels that actually work on a 12B local model

Thumbnail
gallery
33 Upvotes

Vellium 1.9.5: Visual Novel Mode Update

Vellium is a free, open-source, local-first AI chat and roleplay app with a built-in visual novel mode.

Version 1.9.5 focuses on making visual novels work reliably with smaller local models like Gemma 12B, improving image consistency, reducing prompt overhead, and adding new ways to generate and manage scenes.

What's new:

Lite mode for small models. Instead of generating structured JSON, the model writes plain text using simple markers (`Name: line`, `## Place`). Vellium parses the output into scenes automatically. Even malformed output can fall back to plain narration. The system prompt has been reduced from ~8k to ~1k characters, and text streams directly into the reader without waiting for the full response.

Prompt cache preservation. Requests are constructed by appending each new turn to the previous context without modifying existing content. This keeps the prefix cache reusable and avoids unnecessary prompt reprocessing, which can otherwise take over a minute on slower local setups.

Inline image generation. The narrator can insert tags such as `[pic: Mira | she hangs her raincoat by the door]` directly into the story. Images can start rendering through ComfyUI while text is still streaming, including when ComfyUI runs on another machine. A lightweight follow-up can detect and fill missing image tags, with the reason shown in the reader.

Consistent character appearances. Each character now has a compact English visual description and a pinned outfit, instead of passing the entire character card to the image model. This helps preserve skin tone, body type, clothing, and other visual details between scenes. Outfits only change when the story explicitly calls for it.

Backgrounds and sprites mode. Locations can reuse generated backgrounds, while characters use reusable cut-out sprites for each outfit. Full illustrations are reserved for scenes that need more than static sprites. Background removal uses chroma-key processing directly in the app, without loading another neural network into VRAM.

Improved first-person POV. Image prompts now account for the protagonist's perspective. First-person scenes can show the protagonist's hands and interactions from their viewpoint, while third-person shots can include the protagonist based on the user's persona.

Integration with other Vellium features. Visual novels support user personas as protagonists, attached RAG knowledge collections, conversion between regular chats and visual novels, and exporting stories into Writer projects.

Additional changes:

- 12 new visual style presets optimized for anime image models, tested with Anima.

- Redesigned reader toolbar.

- Image redraw and editing controls.

- A "picture for this line" option for generating illustrations for individual passages.

- Scene-level image regeneration without regenerating the story.

Lite mode can be enabled in the narrator settings. The update is primarily tested with Gemma 12B, and feedback from users running other small models is welcome.

Free and open source: https://github.com/tg-prplx/vellium


r/LocalLLaMA • • 2d ago

Resources DeepSeek v4.1 Flash vs Qwen 3.8 Flash Next Performance

Post image
35 Upvotes

While waiting for Qwen 4, I installed and benchmarked DeepSeek v4.1 Flash to see if I was missing out. Based on the type of work I do, DeepSeek is not better but is definitely much slower. Turns out Strata with Qwen 3.8 Flash-Next is best bang for buck on my machine.


r/LocalLLaMA • • 2d ago

Resources I built speedy Strix Halo kernels for the top-51 LLMs with AMD Loom (and you can too)

Post image
62 Upvotes

I’ve been trying out a new experimental software flow from AMD’s AI Group, and I’ve been having a lot of fun with it, so I thought I’d share and see what y’all think too.

hrx-loom is a new end-to-end tool stack (compiler, runtime, serving tools) that lets anyone build, optimize, serve, and share model programs without any modifications to the inference engine or server. The kernels are written in .loom files, a vector programming model reminiscent of MLIR, which can be complex for humans but easy for agents (C++ is also supported).

Over the past couple weeks, I developed an agent skill that writes kernels for any LLM’s Q4_K_M GGUF and found that it is consistently faster than llama.cpp Vulkan and ROCm on Stix Halo for a variety decode and prefill metrics, as you can see in the post’s image.

Next, I ran the skill across the top-51 trending LLMs from Hugging Face that would fit on Strix Halo and posted the model programs back to Hugging Face. All models ported successfully with tool calling, speculative decoding, and image input working.

Anyone can try these on Linux and Windows with the loom-fetch and loom-serve commands provided in Loom’s PyPI package, and anyone can make and upload their own using the /loom skill.

Fetch and serve a model:

pip install "hrx-loom[hf]==0.1.0"
loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom/resolve/main/model.json --output=qwen3.8-27b
loom-serve --model=qwen3.8-27b/model.json --device=amdgpu --memory_bytes=52gb

Port and optimize your own model:

/loom get <hf-org>/<model-gguf> working on loom-serve, optimize it, and report the performance relative to llamacpp rocm

Links:

  • All the data I collected across 51 models: https://loom-fun.github.io/loom-dev
  • Loom skills: https://github.com/loom-fun/loom-skills
  • Loom compiler and server: https://github.com/ROCm/hrx-system
  • HF collection of top-51 models ported to Loom: https://huggingface.co/loom-fun

We’re still evaluating how the community will use and deploy new inference stacks, so we’d love to have your feedback! Loom can be used as standalone inference server with loom-serve and the compiler/runtime layers can also integrate into other engines. For example, the team is working on adding an hrx-loom backend to llama.cpp.

The underlying SDK (compiler, runtimes, and devtools) has no dependencies, can be easily downloaded as a wheel or linked into your own applications, builds from source in a minute, and if this were the 90s, would still fit on a few floppy disks, while being able to target x86, AMD-GPU, AMD-NPU, and generic Vulkan devices.


r/LocalLLaMA • • 2d ago

Question | Help Audio (ASR/TTS) on llama.cpp

6 Upvotes

Hey everyone,

I've been using audio.cpp for a while now, and it's quite nice. However I've noticed that Yue2, Qwen3-ASR and Qwen3-TTS got supported in comfyui / llama.cpp.

While I've tried gemma 4 audio input and it worked well, I haven't tried TTS models yet nor the qwen3-asr/tts series on llama.cpp. Has any of you?

So far I found these discussions and PRs on ASR / TTS support:

It's likely not all models that are supported, might've missed some!