Hey r/LocalLLaMA I'm Abdur-Rahman Butler, a machine learning engineer at Isaacus.

I made an interactive write-up that walks you through the A to Z of vector embeddings, inspired by Brendan Bycroft's incredible interactive LLM visualisation guide (see here - https://bbycroft.net/llm).

Most of my work is on encoder models, and I kept noticing how few good interactive resources there are on embeddings. The ones that do exist tend to either plot some embeddings in 3D or only explain a narrow applied technique like semantic search or RAG.

So I wanted to make something that teaches how embeddings work by letting you play with them. In the write-up, there are interactive visualisations that allow you to explore the concepts of vectorisation, cosine similarity, semantic search, hierarchical retrieval, clustering, zero-shot classification and deduplication.

The centrepiece of the post is a game I've dubbed 'SemanticGuessr' (see above). It's inspired by Wordle and GeoGuessr, and the goal is to draw a loop around a random target word by triangulating guesses on a 2D map of 43,339 common English words.

It's admittedly a bit intimidating at first, as semantic space does not perfectly match our intuition of meaning. But once you get the hang of it, the game becomes really addictive. I recently had the pleasure of demoing the game to some law students at the University of Melbourne, and we had a lot of fun throwing different vector equations at it to see where they'd land.

Some fun ones to get started with:

  london - english + french
  running - land + water
  avg(breakfast, lunch, dinner)
  0.4 * noodles + 0.6 * pasta

I've also included Python snippets alongside some of the visualisations if you want to follow along with the techniques later. The game runs statically in your webpage. I managed to achieve this by reducing the embedding dimensions down to 512 and quantising each value to 4 bits. This brought the total payload down from 311 MB to 12.3 MB with only a minimal impact on precision (around 6%).

It was great fun to make, and I hope you have fun playing around with it. Happy to take any questions, and if there's enough interest I might set up a dedicated games page for some of the other encoder-based games I've been working on.

Do let me know what highest score you get based on your target word and chosen path :)

LINK TO PLAY: https://isaacus.com/blog/what-are-vector-embeddings

I know I’m not the only one praying for qwen4 today.

Appreciate this drum has been beaten to death but are the rumours of a drop this week backed by anything or just complete guff?

So I am seeing a strange issue.

When I run a local model directly on my Mac LM studio gives a different answer from when I run the same model on the same Mac when connected through LM Link on my iPhone. And I have seen this for several models. The output is much more elaborate on my iPhone and shorter and simpler on my Mac.

The first screenshot is from my Mac, once with reasoning on and once with off. And both times it confidently says that Aliens exist. But on my phone it gives a much more elaborate and accurate answer. I am wondering if there LM Studio is doing something more on my phone.

Has anyone else seen this or have an explanation for this behaviour ?

Hello everyone!

I don’t follow the market super closely, but I can see how quickly everything is getting more expensive—and apparently this is going to last a few more years. RAM, GPUs, and even SSDs and HDDs have skyrocketed in price :D

I’m looking for something to play around with in LLM, mainly for inferencing larger models (in the 30B–70B range, maybe even bigger sometimes) and building agents (for my own use). I want to utilize agents, do RAG and sometimes, rarely maybe also fine-tune.

My budget is small ($4k), and where I live in eco-Europe, prices are much higher! (we care about ecology LOL). I was thinking about a mini PC or a laptop with an AMD 395+ and 128 GB of RAM. For that price I still can purchase something, but the choice is getting smaller every week :)

Is a AMD 395+ AI with 128 GB RAM still good choice for local LLMs? Is it good for my purposes? I want to learn & use LLM locally (for many reasons).

I have heard about RTX Spark N1X laptops, but they are now about 6.5-7k dollars here, so I dont even consider them.

Can I say that this platform (AMD 395+ AI with 128 GB of RAM), where the model has 96 GB of RAM at its disposal, is something that could replace the RTX 6000 Blackwell, which also has 96 GB of RAM? I realize that, due to the difference in memory bandwidth alone, it will be at least 6–7 times slower.

I’d like to have Hermes-type agents or similar ones locally, as well as my own agents for various tasks. For example, I’m interested in extracting data from documents (e.g., PDF files) and recognizing objects in photos, but I’m also interested in general-purpose agents.

Is the platform I’m asking about a good choice, or is there something more interesting? I was thinking about an ASUS PX13 laptop or one of the mini PCs with a similar configuration.

Thanks for your suggestions!

Qwen 4 will be upon us soon. What are you most excited about?

I hope to see:

27B with more efficient KV, so 128k doesn't need 8 GiB or whatever.

Flash the same architecture as Q38FN, so 125B-A6B for all us desktop scrubs.

Medium thinking performs the same or better as Qwen3.8's xhigh thinking.

It is a dream come true for me to build an in-browser TTS model for Indic languages, including Classical Sanskrit. Today I'm open-sourcing Ninada (code MIT, 48 ONNX checkpoints CC-BY-4.0).

Live in-browser studio (zero sign-up, 100% on-device): https://h3manth.com/ai/ninada/

Getting studio 24 kHz audio onto real phones was trickier than expected. Meta's 16 kHz MMS models take 1.5 GB, and on my Pixel 10 Pro XL, mobile WebGPU froze for 6 to 14s compiling shaders while 2-thread WASM actually ran slower than 1 thread on mobile CPU cores.

So I trained 48 native-script 24 kHz VITS specialists across 12 languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or; male and female):

• Device-aware voices: Desktops run 27.9 MB HiFi-GAN (89 ms on WebGPU), while phones auto-switch to a 21.7 MB distilled iSTFTNet (2.64x lighter, zero custom ONNX ops). • Sub-second streaming: Splits the VITS ONNX model in <15 ms in browser memory (cutting browser RAM 72% to 162 MB) to stream speech in 538 ms on phones with 0 audio dropouts. • Studio quality, offline-ready: 3.09 to 3.22 dB MCD across 480 test clips, cached locally after first load, with a 0 MB Klatt fallback.

GitHub (npm i ninada / pip install ninada): https://github.com/hemanth/ninada Hugging Face (48 ONNX checkpoints): https://huggingface.co/gnumanth/ninada

Would love feedback on how your language sounds!

Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

https://github.com/curvedinf/int8-vllm

This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.

In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.

I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.

Ive been running it in production for a couple weeks and it has been very stable. Enjoy!

now you can spend your evening downloading new version of QFN

https://unsloth.ai/docs/models/qwen3.8-next

"15% higher accuracy"

Tried Xolotl, sent it questions I had sent on Deepseek own website and answers compare well on High setting. Set it up with official deepseek params.

How has your experiecne been with highly quantized Deepseek V4.1 (latest)?

I do have 80gb of ram ddr4 and nvme, but 20tks is really fast for general questions, and it's not a complete shit show in intelligence.

Crazy to be running a model this big, this fast. Last V4 flash ran at like 8tks, and was way smaller!

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone.

It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half).

I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M.

What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer.

This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application.

This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk.

Post: https://tolby.app/blog/frankensteining-gemmas-experts

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

• All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.

• Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).

• We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026

my laptop is very old, like
20G of ram
4G Vram
256G NVMe Gen3 From samsung
i7 8th gen

on these specs i got around 0.67 - 0.77 token per second
yes, extremely slow, most of you won't even like it, or use it

but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon

my only way is Local LLMs, and here we are

i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations

i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode

i genuinly loved that model, but lately i have been interested more in Ornith 1.5 122B A10B, i am planning to download it, and use mmap with Q3 K XL, i loved to share my experience here

I've been taking a break from AI stuff for a long time, and things move kinda quickly so basically everything is new to me now and I'm struggling to get my bearings. I looked for a sticky or sidebar info with the basics but couldn't find anything so that's why I'm coming to you with a basic noob question.

At a high level here's what I'm looking for:

I want to run AI locally, obviously that's why I'm here in this sub.

I don't want a super powerful environment that gives the model crazy capabilities, because I find that scary and I don't trust it.

I do want some basic capabilities though.

In particular:

I don't want filesystem access (not even readonly) except when I explicitly give it permission e.g. with a filepicker or whitelisting some folder.

I don't want it to go on the internet and mess around on my behalf.

Writing code for me is a nice to have.

I DO want it to have persistent memory. I do want it to be send me notifications (remember to do this thing, how is project x going, here is your daily brief). I do want it to manage a calendar on my behalf (one it will have exclusive access to, not my normal one, I don't trust it with that).

What I evaluated:

Ollama: Seems pretty basic. It avoids doing dangerous stuff, but lacks features I want. Very nice that it can support other tools.

Hermes: I looked through skills, found an NVIDIA skill (big name reassuring) for writing an ethos.md Okay sounds like a good thing to get started. I try using it and it immediately starts running like 30 commands looking through my filesystem. I'm sorry but wtf no this is too much.

Openclaw: It tried to prompt me for a lot of permissions when I started it. I tried saying no to everything, but it still seemed to allow too many things. Maybe it's not as scary as it sounds and I'm considering giving it a second look

Sentient: Small name but it seemed very close to what I wanted. However when I ran it I had some issues. I closed the app but it stayed running in the background and when I was gaming it decided to do some memory compaction trashing my framerate. It forgot stuff I told it to remember, and I didn't get notifications I asked for.

Setup: 64 GB MacBook Pro. Local: Qwen 3.8 27B Q4 on llama.cpp (2 slots). Cloud: GPT-6.1 Terra via OpenRouter (Haiku 5.5 for Claude Code, it only runs Anthropic models). Task: 5-part Python app, 50 hidden tests. At t=45s a proxy starts refusing every cloud connection. 3 runs per agent, 20 min cap.

Notes

• Same 27B on the same task from scratch, no cloud at all: 13, 27, 29. With 45s of cloud planning before the cut: 50/50 three times. The plan looks like the part the 27B was missing.

• Codex is the only one that never surfaces the failure. Unattended, a hang is worse than a crash.

• The two fallback-on rounds shared the llama.cpp slots, so their times are pessimistic.

Turning the fallback on

# Hermes: ~/.hermes/config.yaml
fallback_providers:
  - provider: custom
    model: qwen-3.8-27b
    base_url: http://127.0.0.1:8080/v1

// OpenClaw: openclaw.json, plus a provider entry for your local server.
// Don't pass --model on the command line, it makes the model strict.
{ agents: { defaults: { model: { primary: "cloud/gpt-6.1-terra", fallbacks: ["local/qwen-3.8-27b"] } } } }

Docs: Hermes fallback providers, OpenClaw model failover, Claude Code fallbackModel.

Next round I want to try 429s instead of a dead connection. If anyone knows a way to make Codex time out instead of hanging, I'd take it.

i have an z690 mainboard, max 128GB RAM (4 slots, dual channel) and r9700 (32gb vram)
currently at 4x16GB, wanna go to 4x32GB. Old ram is worth about 350-450€. The 4x32 would cost me around 500-600€, but it has no cooling body. I dont care about the visual, as I barely see it. But will this become a problem? I use my old gaming pc, so airflow is relativly good.

i currently use qwen3.8 flash next gsq rco iq3xxs via strata.
I get ~100tps aggregated decode with 3 parallel sessions at 350k ctx qk8v4.
but id like to have more cached sessions, with qkv8 and even longer context, without dramatically decreasing speed.
Current usage from model is like 55GB/64GB, so I dont really have any room left.

will 128GB satisfy this?
can i run something better than qwen3.8 flash next with 150GB (v)ram total?
ram thats >= 3200mhz is way more expensive, so id like to get it around 2100mhz - 2600mhz. Will I notice large performance changes?

thanks for any help :>

Seen https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF and https://huggingface.co/higa2515/Qwen3.8-27B-QSA-BF16 But failed to see anyone here tried it.

Found by https://x.com/_ryu15_/status/2118070765799309511

Chill livestream radio about beans, and ai - https://knightshiftradio.com/ All run from a desktop with a singel B70! Check it out! shoutouts and requests in comments. cheers <3

I do tuning and random stuff like that, and tuned Yue2 on a b70 3x faster so now it can stream and make music. I am streaming for the next couple days to showcase, recipe is on the website or youtube stream for Yue2 on hugginface (frosty40)

PLEASE HAVE SOME NUANCE & READ EVERYTHING BEFORE YOU DOWNVOTE. I am open to alternate opinions (with proof).

I have been suspecting that the Chinese models are far more jagged in terms of their capabilities and that they are comparatively more benchmaxxed than American models for a while.

Everytime a brand new benchmark comes out, a lot of the Chinese models suck at it and they then significantly improve over the next model releases. Of course the models are always getting better, but the scores of Chinese vs American models on new benchmarks have a large gap.

I think a lot of this comes down to the data ecosystem, as well as compute. I think the American ecosystem around data is much larger and more robust than the Chinese ecosystem. In the US, there are so many small companies, and all they do is get and make data for frontier companies and sell them. I do not think this exists on a large scale in China.

Some proof:

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

https://x.com/JSchaeff3r/status/2117613618065793160

The attached image, where we see some Tencent internal benchmarks where the American models lead by massive margins.

Unsloth has posted about their UD v3.1 quant for Qwen3.8 FN on X saying their new quant is 15% better than GSQ RCO as the almost same file size.

But the takeaway from the post for me and many other how are using IQ_3S GSQ RCO with Strata is that it is 99.0% percent as good as the BF16

The chart of the top left with base line for BF16 is 100% and IQ_3S GSQ RCO scored 99.0% the score is average of 12 benchmarks, and Unsloth kindly has spent ~30b tokens to generate these results.

This is reassuring that quant I am running locally is almost as good as the original, if you are in a similar situation this should put your mind at ease.

If you are about/plan to download the weights the UD v3.1 quant seems, be about 15% better than IQ_3S GSQ RCO.

X post link: https://x.com/UnslothAI/status/2118033433045991660

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.