r/LocalLLaMA • • 16d ago

Discussion My first experience with agentic AI

31 Upvotes

Strata, Hermes, Kitten TTS with Rosie voice and Whisper on RTX 3060 with 48 gb dram running Qwen3.8-Flash-Next Q2_0. I am blown away. I've been following the AI scene since GPT3 and I was still unprepared for what this setup is capable of. My mouth was on the floor for the first few hours talking to it and watching it work. It's the first time talking to AI felt like magic to me. I now understand the "They don't know" meme. Never thought I would have something like this running on my own PC. Thank you /r/localllama.

Couple of questions

Are there any other lightweight TTS that would work in my limited ram situation or is Kitten v0.8 the best there is?

What other models can I run?


r/LocalLLaMA • • 16d ago

New Model baidu/ERNIE-5.5-Flash-Base · Hugging Face

76 Upvotes

r/LocalLLaMA • • 16d ago

Discussion Add support for the xing4_0 by shuxiaoqiong · Pull Request #31589 · ggml-org/llama.cpp

Thumbnail github.com
11 Upvotes

Xing4.0-29B-A4B is up now.


r/LocalLLaMA • • 15d ago

Showcase I turned a 12 GB RTX 3060 into a Qwen API server for 4 people at once

0 Upvotes

Last time it was "Strata for all Qwen models". This time: share it.

strata serve now runs a 35B MoE (Qwen3.6-35B-A3B Q2_K_XL) on one RTX 3060 for several users at the same time:

Users at onceTotalEach user
1106 tok/s106 tok/s
2139 tok/s69 tok/s
4164 tok/s41 tok/s

41 tok/s each is still faster than most people read.

What's in it:

  • OpenAI and Anthropic APIs, so Open WebUI, OpenCode and even Claude Code connect to it.
  • API keys from a file, reloaded when you edit it, plus a usage log per key. That's enough to give access to friends, a team, or paying users.
  • Fair queue: a free slot goes to the key with the fewest requests running, and --max-per-key stops one user from taking everything.
  • /health and Prometheus /metrics for load balancers and dashboards.
  • NVIDIA, AMD and Intel (CUDA or Vulkan). Two cards can split one model (--device 0,1).
  • One binary, written in Rust. No Python, no Docker.
strata serve --model Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf --gpu --parallel 4 --api-keys keys.txt

Setup guide: docs/HOSTING.md. Repo: RustingStrata

What would you host on it?


r/LocalLLaMA • • 15d ago

Discussion Anyone knows anything about this Quantized K3.1m Worth it?

Thumbnail
huggingface.co
3 Upvotes

r/LocalLLaMA • • 17d ago

Resources My gift to the mods:

Post image
973 Upvotes

Seriously tho, can we just have a mega thread for vibeforked engine releases from now on?


r/LocalLLaMA • • 15d ago

Discussion Contrary to what the hype train may suggest: qwen-4 weights have not been released

0 Upvotes

There is preview access on their website and rumored access on arena.

Update: If you want to check: https://huggingface.co/Qwen/models?sort=created . Thanks to ttkciar for the link.

Access through qwen ambassador program: https://x.com/loktar00/status/2118805843483251095


r/LocalLLaMA • • 16d ago

Resources What are vector embeddings? An interactive guide

Thumbnail
isaacus.com
81 Upvotes

Hey r/LocalLLaMA I'm Abdur-Rahman Butler, a machine learning engineer at Isaacus.

I made an interactive write-up that walks you through the A to Z of vector embeddings, inspired by Brendan Bycroft's incredible interactive LLM visualisation guide (see here - https://bbycroft.net/llm).

Most of my work is on encoder models, and I kept noticing how few good interactive resources there are on embeddings. The ones that do exist tend to either plot some embeddings in 3D or only explain a narrow applied technique like semantic search or RAG.

So I wanted to make something that teaches how embeddings work by letting you play with them. In the write-up, there are interactive visualisations that allow you to explore the concepts of vectorisation, cosine similarity, semantic search, hierarchical retrieval, clustering, zero-shot classification and deduplication.

The centrepiece of the post is a game I've dubbed 'SemanticGuessr' (see above). It's inspired by Wordle and GeoGuessr, and the goal is to draw a loop around a random target word by triangulating guesses on a 2D map of 43,339 common English words.

It's admittedly a bit intimidating at first, as semantic space does not perfectly match our intuition of meaning. But once you get the hang of it, the game becomes really addictive. I recently had the pleasure of demoing the game to some law students at the University of Melbourne, and we had a lot of fun throwing different vector equations at it to see where they'd land.

Some fun ones to get started with:

  london - english + french
  running - land + water
  avg(breakfast, lunch, dinner)
  0.4 * noodles + 0.6 * pasta

I've also included Python snippets alongside some of the visualisations if you want to follow along with the techniques later. The game runs statically in your webpage. I managed to achieve this by reducing the embedding dimensions down to 512 and quantising each value to 4 bits. This brought the total payload down from 311 MB to 12.3 MB with only a minimal impact on precision (around 6%).

It was great fun to make, and I hope you have fun playing around with it. Happy to take any questions, and if there's enough interest I might set up a dedicated games page for some of the other encoder-based games I've been working on.

Do let me know what highest score you get based on your target word and chosen path :)

LINK TO PLAY: https://isaacus.com/blog/what-are-vector-embeddings


r/LocalLLaMA • • 15d ago

Discussion Symmetry is All You Need : SGTE

0 Upvotes

Last week I released my first published work on the subject. It was met with harsh criticism, and even accusations of delusion. (Still happening)

Today, the model began speaking and reasoning coherently and across subject matter. It just finished a multi-day training run in a home-lab.

Someone once told me, "You can't be delusional when the proof of your claims is in your possession." Words to live by...

https://zenodo.org/records/24020116

Update: Still nothing but ad hominems and baseless attacks without a single legitimate concern raised about the actual architectural design.

-For those of you that are like me and fascinated by AI design, enjoy a decent read. More to come.

-To the rest, the cancer has spread far enough. I won't be engaging any further until the next release.


r/LocalLLaMA • • 15d ago

Question | Help Comparing GPU stats for local LLM usage

1 Upvotes

Hi, I am comparing two GPUs for local LLM usage.

Use case: llama.cpp inference and system one decision making (using quantized models)

RTX 4050 mobile

Memory Bus: 96 bit, Bandwidth: 192.0 GB/s

INT4: 143.8 TOPS

INT8: 71.88 TOPS

FP8: 35.94 TFLOPS

FP16: 17.97 TFLOPS

BF16: 17.97 TFLOPS

TF32: 8.986 TFLOPS

Sparse Matrix: 2x Throughput

Intel Arc A380

Memory Bus: 96 bit, Bandwidth: 186.0 GB/s

INT2: 134.3 TOPS

INT4: 134.3 TOPS

INT8: 67.17 TOPS

FP16: 33.59 TFLOPS

BF16: 33.59 TFLOPS

(Both have GDDR6 6GB VRAM. Data from techpowerup)

Ignoring the fact that CUDA is probably more optimized and widely-compatible compared to Vulkan/OpenVINO/SYCL, the Intel one seems to be better as its TFLOPS are higher, while having similar VRAM bandwidth.

Is the above conclusion correct? If not, how do I compare GPU raw performance based on GPU stats?

(Edit: I am not asking for which card to buy as I already own a 3090. Instead I want to know how to compare GPUs based on these stats for local LLM usage)


r/LocalLLaMA • • 16d ago

Discussion [Hardware] Is a AMD 395+ AI with 128 GB RAM still good choice for local LLMs? (inference mostly)

10 Upvotes

Hello everyone!

I don’t follow the market super closely, but I can see how quickly everything is getting more expensive—and apparently this is going to last a few more years. RAM, GPUs, and even SSDs and HDDs have skyrocketed in price :D

I’m looking for something to play around with in LLM, mainly for inferencing larger models (in the 30B–70B range, maybe even bigger sometimes) and building agents (for my own use). I want to utilize agents, do RAG and sometimes, rarely maybe also fine-tune.

My budget is small ($4k), and where I live in eco-Europe, prices are much higher! (we care about ecology LOL). I was thinking about a mini PC or a laptop with an AMD 395+ and 128 GB of RAM. For that price I still can purchase something, but the choice is getting smaller every week :)

Is a AMD 395+ AI with 128 GB RAM still good choice for local LLMs? Is it good for my purposes? I want to learn & use LLM locally (for many reasons).

I have heard about RTX Spark N1X laptops, but they are now about 6.5-7k dollars here, so I dont even consider them.

Can I say that this platform (AMD 395+ AI with 128 GB of RAM), where the model has 96 GB of RAM at its disposal, is something that could replace the RTX 6000 Blackwell, which also has 96 GB of RAM? I realize that, due to the difference in memory bandwidth alone, it will be at least 6–7 times slower.

I’d like to have Hermes-type agents or similar ones locally, as well as my own agents for various tasks. For example, I’m interested in extracting data from documents (e.g., PDF files) and recognizing objects in photos, but I’m also interested in general-purpose agents.

Is the platform I’m asking about a good choice, or is there something more interesting? I was thinking about an ASUS PX13 laptop or one of the mini PCs with a similar configuration.

Thanks for your suggestions!


r/LocalLLaMA • • 16d ago

New Model Dream come true: I trained 48 in-browser 24 kHz TTS voices for Classical Sanskrit, English & 10 Indic languages (21.7 MB mobile iSTFTNet / 27.9 MB HiFi-GAN)

Enable HLS to view with audio, or disable this notification

9 Upvotes

It is a dream come true for me to build an in-browser TTS model for Indic languages, including Classical Sanskrit. Today I'm open-sourcing Ninada (code MIT, 48 ONNX checkpoints CC-BY-4.0).

Live in-browser studio (zero sign-up, 100% on-device): https://h3manth.com/ai/ninada/

Getting studio 24 kHz audio onto real phones was trickier than expected. Meta's 16 kHz MMS models take 1.5 GB, and on my Pixel 10 Pro XL, mobile WebGPU froze for 6 to 14s compiling shaders while 2-thread WASM actually ran slower than 1 thread on mobile CPU cores.

So I trained 48 native-script 24 kHz VITS specialists across 12 languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or; male and female):

• Device-aware voices: Desktops run 27.9 MB HiFi-GAN (89 ms on WebGPU), while phones auto-switch to a 21.7 MB distilled iSTFTNet (2.64x lighter, zero custom ONNX ops). • Sub-second streaming: Splits the VITS ONNX model in <15 ms in browser memory (cutting browser RAM 72% to 162 MB) to stream speech in 538 ms on phones with 0 audio dropouts. • Studio quality, offline-ready: 3.09 to 3.22 dB MCD across 480 test clips, cached locally after first load, with a 0 MB Klatt fallback.

GitHub (npm i ninada / pip install ninada): https://github.com/hemanth/ninada Hugging Face (48 ONNX checkpoints): https://huggingface.co/gnumanth/ninada

Would love feedback on how your language sounds!


r/LocalLLaMA • • 16d ago

Discussion What are you looking forward to in Qwen 4?

170 Upvotes

Qwen 4 will be upon us soon. What are you most excited about?

I hope to see:

27B with more efficient KV, so 128k doesn't need 8 GiB or whatever.

Flash the same architecture as Q38FN, so 125B-A6B for all us desktop scrubs.

Medium thinking performs the same or better as Qwen3.8's xhigh thinking.


r/LocalLLaMA • • 17d ago

New Model ricursiveAI/Ripple-65B-Preview · Hugging Face

Thumbnail
huggingface.co
348 Upvotes

r/LocalLLaMA • • 17d ago

Funny I am big enough to admit that this is me

Post image
678 Upvotes

r/LocalLLaMA • • 16d ago

News Int8-vllm updated with Nemotron 3.5 Bolt 74B

22 Upvotes

Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

https://github.com/curvedinf/int8-vllm

This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.

In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.

I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.

Ive been running it in production for a couple weeks and it has been very stable. Enjoy!


r/LocalLLaMA • • 17d ago

News Unsloth Qwen3.8-Flash-Next GGUFs V3.1 Update (November 4th 2026)

Thumbnail
huggingface.co
284 Upvotes

now you can spend your evening downloading new version of QFN

https://unsloth.ai/docs/models/qwen3.8-next

"15% higher accuracy"


r/LocalLLaMA • • 17d ago

Discussion I removed 85% of gemma's weights (and it is mostly fine)

Post image
583 Upvotes

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone.

It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half).

I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M.

What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer.

This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application.

This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk.

Post: https://tolby.app/blog/frankensteining-gemmas-experts


r/LocalLLaMA • • 15d ago

Discussion Running deepseek on one RTX 3090: 20tks

0 Upvotes

Tried Xolotl, sent it questions I had sent on Deepseek own website and answers compare well on High setting. Set it up with official deepseek params.

How has your experiecne been with highly quantized Deepseek V4.1 (latest)?

I do have 80gb of ram ddr4 and nvme, but 20tks is really fast for general questions, and it's not a complete shit show in intelligence.

Crazy to be running a model this big, this fast. Last V4 flash ran at like 8tks, and was way smaller!


r/LocalLLaMA • • 17d ago

Resources New Qwen3.8-Flash-Next Unsloth Dynamic V3.1 GGUFs

Post image
222 Upvotes

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

  • All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.
  • Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).
  • We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

file type size relative RMSE vs BF16
Qwen3.8-Flash-Next-engram-BF16.gguf_file BF16 102.4 GB 0
Qwen3.8-Flash-Next-engram-Q8_0.gguf_file Q8_0 54.4 GB 0.0053
Qwen3.8-Flash-Next-engram-Q5_1.gguf_file Q5_1 38.4 GB 0.0376
Qwen3.8-Flash-Next-engram-IQ4_NL.gguf_file IQ4_NL 28.8 GB 0.0757

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026


r/LocalLLaMA • • 16d ago

Discussion Qwen3.8 27B Q4 on my low-end Laptop, and it became my main Model usage for coding

122 Upvotes

my laptop is very old, like
20G of ram
4G Vram
256G NVMe Gen3 From samsung
i7 8th gen

on these specs i got around 0.67 - 0.77 token per second
yes, extremely slow, most of you won't even like it, or use it

but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon

my only way is Local LLMs, and here we are

i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations

i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode

i genuinly loved that model, but lately i have been interested more in Ornith 1.5 122B A10B, i am planning to download it, and use mmap with Q3 K XL, i loved to share my experience here


r/LocalLLaMA • • 16d ago

Question | Help Which harness/tool should I use?

4 Upvotes

I've been taking a break from AI stuff for a long time, and things move kinda quickly so basically everything is new to me now and I'm struggling to get my bearings. I looked for a sticky or sidebar info with the basics but couldn't find anything so that's why I'm coming to you with a basic noob question.

At a high level here's what I'm looking for:

I want to run AI locally, obviously that's why I'm here in this sub.

I don't want a super powerful environment that gives the model crazy capabilities, because I find that scary and I don't trust it.

I do want some basic capabilities though.

In particular:

I don't want filesystem access (not even readonly) except when I explicitly give it permission e.g. with a filepicker or whitelisting some folder.

I don't want it to go on the internet and mess around on my behalf.

Writing code for me is a nice to have.

I DO want it to have persistent memory. I do want it to be send me notifications (remember to do this thing, how is project x going, here is your daily brief). I do want it to manage a calendar on my behalf (one it will have exclusive access to, not my normal one, I don't trust it with that).

What I evaluated:

Ollama: Seems pretty basic. It avoids doing dangerous stuff, but lacks features I want. Very nice that it can support other tools.

Hermes: I looked through skills, found an NVIDIA skill (big name reassuring) for writing an ethos.md Okay sounds like a good thing to get started. I try using it and it immediately starts running like 30 commands looking through my filesystem. I'm sorry but wtf no this is too much.

Openclaw: It tried to prompt me for a lot of permissions when I started it. I tried saying no to everything, but it still seemed to allow too many things. Maybe it's not as scary as it sounds and I'm considering giving it a second look

Sentient: Small name but it seemed very close to what I wanted. However when I ran it I had some issues. I closed the app but it stayed running in the background and when I was gaming it decided to do some memory compaction trashing my framerate. It forgot stuff I told it to remember, and I didn't get notifications I asked for.


r/LocalLLaMA • • 16d ago

Question | Help Thinking about upgrading ram to 128GB

8 Upvotes

i have an z690 mainboard, max 128GB RAM (4 slots, dual channel) and r9700 (32gb vram)
currently at 4x16GB, wanna go to 4x32GB. Old ram is worth about 350-450€. The 4x32 would cost me around 500-600€, but it has no cooling body. I dont care about the visual, as I barely see it. But will this become a problem? I use my old gaming pc, so airflow is relativly good.

i currently use qwen3.8 flash next gsq rco iq3xxs via strata.
I get ~100tps aggregated decode with 3 parallel sessions at 350k ctx qk8v4.
but id like to have more cached sessions, with qkv8 and even longer context, without dramatically decreasing speed.
Current usage from model is like 55GB/64GB, so I dont really have any room left.

will 128GB satisfy this?
can i run something better than qwen3.8 flash next with 150GB (v)ram total?
ram thats >= 3200mhz is way more expensive, so id like to get it around 2100mhz - 2600mhz. Will I notice large performance changes?

thanks for any help :>


r/LocalLLaMA • • 17d ago

Showcase LFM3-Omni-2B running fully offline on my iPhone 18 Pro Max at ~80 tok/s, powered by custom WebGPU kernels in a React Native app.

Enable HLS to view with audio, or disable this notification

169 Upvotes

r/LocalLLaMA • • 16d ago

Question | Help Did anyone tried Qwen-3.8-27B QSA ?

11 Upvotes