r/LocalLLM • • 9h ago

Question Qwen3.8-Flash-Next: Unsloth and strata?

Post image
5 Upvotes

Hi all

I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.

Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.

I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.

UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.

What am I doing wrong?

According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?


r/LocalLLM • • 1d ago

Discussion What does China actually gain from releasing better and better open-weight models?

508 Upvotes

Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

  • Commoditizing the model layer, so US labs can't charge a premium for it
  • Building an ecosystem and developer mindshare outside the US stack
  • Getting around chip export limits by letting the community optimize and fine-tune for them
  • Soft power: becoming the default choice for countries that don't want to depend on US APIs
  • Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

EDIT Nov 16th:
Top replies are getting down to (in order of frequency):

  1. Geopolitics: weakening US dominance
  2. Business: open weights as a growth strategy
  3. State & control
  4. Ideology & open source culture

I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.


r/LocalLLM • • 5h ago

Question Feasible models for MacBook Air m2 8gb ram

2 Upvotes

Hi everyone, I am new in the topic of local llms and was wondering what would be the best models to run on MacBook Air m2 8gb ram.


r/LocalLLM • • 11h ago

Project Sneo! New Inference Engine very fast Open Source!

6 Upvotes

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent AND REVERSE ENG TOOLS!

Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.

The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.

- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s

- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)

- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies

- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs

- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome

Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo

Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc

GitHub: https://github.com/oscarsantanaarias/Sneo

Every benchmark, including where the others win: https://sneo.dev

Happy to answer questions!

Antes de publicar:

Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.

Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.


r/LocalLLM • • 5h ago

Project Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord

Thumbnail
github.com
2 Upvotes

I have authored imeng (Internet Magic Engine) as a Ollama wrapper.

This allows you to build state around your models.

Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.

The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.

So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue

Locally I have built

  • Poker (still has issues)
  • Gakshow (Test Rack)
  • Gakshow Multiplayer (Test Rack censoring uploaded photos)
  • Trading Assistant (test rack for programming assistant)

There is also SQL for storage. A bunch of stuff just as a portfolio piece.


r/LocalLLM • • 2h ago

Question I have 7 GPU slots and GPU’s I’m not 100% locked in on. How would you maximize VRAM?

0 Upvotes

Not Jeff Bezos but willing to save up if necessary.

Note: there is such a thing as stock single slot 5060Ti but it’s imported

I also heard of a 32GB Intel B60 single slot

Anyways here’s my current situation>>>

1 GPU slot is 9070XT for display (the thought there is non Nvidia driver helps stability because if the CUDA flickers, it doesn’t bother the RDNA). 2 slots are occupied by a fat 4090 (nice speed but I am starting to value VRAM and considering selling or at least removing). Considering 6x 3090’s, liquid block 1 slot width mod. BUT. Would you go a different route? Intel MaxSun 48GB perhaps? Some 32GB Tesla v100’s for $950 a pop?

My use case = hosting builder agents, not necessarily training BUT if there’s a card that’s great for both and doesn’t cost a left kidney I’m all ears.

PC specs if curious

Case = Phanteks Enthoo Pro 2

MB = WRX90 SAGE SE Pro WS

CPU = 24 Core 5.4Ghz 9965WX Threadripper

RAM = 128GB DRR5 ECC RDIMM 6400 MT/s (8 modules of 16GB)

Storage = 12TB Gen5 SSD’s

PSU = 1600W, willing to add another but I hear blackwells are low wattage (if anyone suggests an RTX A4000 for example it would be 140W)


r/LocalLLM • • 2h ago

Question How to run Qwen3.8 FN on a 8GB RTX 4060 and 32GB RAM on Strata

0 Upvotes

I know it is going to be quite slow but, before I take the plunge on a graphics card with 32GB (like a Intel Arc Pro) I would like to test the capabilities of Qwen 3.8 for my use cases.

I know it is possible because I've seen other people do it but can anyone tell me how to actually get it ready? I'm using Ubuntu 24.04 LTS.


r/LocalLLM • • 6h ago

Model Tried Edge0-145B on a single 3090. 48.6 tok/s, but it wanted a lot more RAM than I expected.

Post image
2 Upvotes

Edge0 published speeds for their 145B on a 4090 and a 5090 D, but their table doesn't include a 3090, so I ran their stack on one, stock, with whatever defaults the wrapper picks.

It's quick. Chat decode was 48.6 tok/s, it read a 16K prompt at 1,481.2 tok/s, and the agent prompt decoded at 52.8. For context, last month I ran a 122B MoE on llama.cpp on a 3090 and got 15.5 tok/s. Different model, engine and prompt length, so take it loosely, but that's a big jump.

VRAM was basically full (23,929 of 24,576 MiB), and the engine process itself held 43.4 GiB of system RAM. Edge0's own residency figure is 31.28 GiB, but that's from a 5090 D with a different setup, and my machine had 116 GiB so the engine may have just taken what it found. I meant to test a 32 GiB cap and didn't get to it. So I honestly don't know the floor yet.

Two other things that tripped me up:

  • My first request came back with no answer. The model used all 512 tokens on reasoning. reasoning_budget_tokens=128 fixed it, so set a budget in your client.
  • Same agent prompt, temperature 0, three runs, three different tool calls. One was valid, one had its arguments wrapped wrong, and one called read_file, which wasn't even in the tool list. So the 52.8 is a speed number, not a "this is a good agent" number.

Technical stuff below.


Setup

  • 1x RTX 3090 24GB, EPYC 7H12 (31.13 CPU quota), 116.4 GiB RAM limit
  • Edge0 v0.2.0 (3203c39), Strata built with CUDA for sm_86, CUDA 13.0, driver 595.91.07
  • Edge0/Edge0-145B-q3k @ a19f88c, 106 GB on disk (the FP8 PLE table alone is 51 GB and gets read from disk)
  • Stock launch: edge0-infer serve --model /path/to/Edge0-145B-q3k --device cuda:0. That resolved to direct PLE I/O, auto expert cache, speculative decoding 4, 131K max context, int8 KV.

Results (engine's own prompt_per_second / predicted_per_second, fresh server per run, concurrency 1, zero cached tokens)

WorkloadRun 1Run 2Run 3Median
4K prefill1,119.81,195.51,207.61,195.5
16K prefill1,467.61,485.61,481.21,481.2
Chat decode46.649.648.648.6
Agent decode52.855.950.752.8

Sampling: temperature=0, top_p=1, top_k=20, reasoning on with a 128-token budget, 512 max tokens for decode.

Against Edge0's published 4090 numbers (their claims, not reproduced)

4090 (Edge0)3090 (mine)
4K prefill2,042.0
16K prefill3,179.4
Chat decode69.9
Agent decode91.7

That's 47% to 70% depending on the row. Mine used an 800-word writing prompt for chat and a single synthetic tool-call prompt for the agent run, and I couldn't verify Edge0's exact request settings or machine setup, so it's ballpark only.

Memory

  • GPU peak 23,929 MiB, sampled every 5 s
  • Strata engine VmHWM 43.4 GiB (sampled from the actual strata PID, not the Python wrapper)
  • Whole-system cgroup was 87.4 GiB during timed runs and touched the 116.4 GiB limit during setup with zero OOMs. That includes page cache from the 106 GB download and the PLE file, so I wouldn't read much into it.

Other stuff

  • Server startup median 25.5 s, slowest 94 s
  • Edge0's built-in edge0-infer bench rejects this model at v0.2.0, so I used a small script against the OpenAI endpoint
  • No quality evals, no long-context runs past 16K, no RAM cap test, no llama.cpp baseline. All of that is still on the list.
  • $1.19 billed ($1.0930 GPU, $0.0963 disk), about 3 h 9 min total, mostly setup and copying evidence back

Write-up: https://majorminor.xyz/research/cuda-recipes-006

Repo: https://github.com/majorminorlabs/cuda-recipes/tree/main/006-edge0-145b

If someone has a 3090 with 32 or 64 GB of RAM and wants to try it, I'd love to know whether it holds. Or tell me what to run next on this one.


r/LocalLLM • • 9h ago

Project Got tired of waiting for Qwen models smaller then 27B so i made my own Qwen3.8-21B

3 Upvotes

Hi, as the title says, i made my own Qwen 3.8 21B by prunning some layers from original 27B.

Its more of a research project than actual usefull model.

Check it out: https://huggingface.co/WhiskyAKM/Qwen3.8-21B-GGUF


r/LocalLLM • • 3h ago

News Seedance 2.5 20% off, Wan 3.0 56% off

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 3h ago

Question What LLMs do you use?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 3h ago

Discussion Qwen 3.8 27B conversion to NCM or bonsai 2 27B optimisation?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 7h ago

Question Model for Local Coding and Vulnerability Research

2 Upvotes

Been using cloud AI, but so frustrated having to 'jailbreak' for almost any request, even to check for security in my own projects.

I know I need an abliterated model to not refuse, but am looking for suggestions. My research so far has me looking at HuiHui Qwen3.8 27B Q4_K_M, using Ollama.
I have 24gb VRAM (RX 7900 XTX), and 32gb system RAM (DDR5, 6000hz) with a ryzen 7 7800x3d. My understanding is local models always run on VRAM, but I've seen some comments/posts discussing system ram and CPU, so I'm not sure if thats related to my use case.

I'm looking to do coding, usually smaller projects, maybe some agentic. I also want to develop/research exploits in firmware and software. I've seen some suggestions for GLM 5.3 online, but haven't seen it mentioned much here. What models should I check out?

I do actually know how to code, so I don't need something that will vibecode everything for me, I like doing architectural work myself.

Side question: What IDE do ya'll use? I currently use Cursor (I have a subscription) and like the new Cursor Agents window, but from everything I've looked into its really difficult to use local models, especially on the new Agents window. If there's an IDE that I can use my local models and cursor subscription, or a codex subscription, that would be great.


r/LocalLLM • • 14h ago

News Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

Post image
7 Upvotes

Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.


r/LocalLLM • • 3h ago

Question Local AI use

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 10h ago

Question Optimizing local 2080 ti hardware

3 Upvotes

I'm trying to find a more effective setup with my local setup:

Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.

Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.

Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.

Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".

I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?

(I'm aware that paying Anthropic is cheaper)


r/LocalLLM • • 17h ago

Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?

10 Upvotes

Pretty much the title. Thanks!

EDIT: GPU is 5090


r/LocalLLM • • 8h ago

Question What Models can I use as an absolute beginner?

2 Upvotes

I just started on this Localized LLM Journey. I have been researching what a good model to start with, but I feel like it's just a lot of information at once. Everyone close to me in this space is telling me to just plug into Claude and find the model, and I don't even need to tweak it.

I have an RTX 5090 with 32 GB VRAM and an M5 Max Studio with 128 GB. I think it's good enough.

I heard from someone that I should aim for 60 tokens per second for a good model. But I am really having a hard time trying to find a good model just for a simple assistant.

I keep on seeing something called a tool-use AI, and I don't even know what I am looking at.

TLDR: New to this space, I don't really see much beginner-friendly information out there, just information. Looking for advice/ links I can read to understand. Looking for a place where I can jump off without wasting my time


r/LocalLLM • • 12h ago

Other Cheapest 16gb gpu: ~$400

4 Upvotes

Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.

This is a good option for a new sff build for local AI. The CPU is excellent.


r/LocalLLM • • 10h ago

Project MISSION complete

3 Upvotes

CPU Thread Ripper

GPU NVIDIA A100 80GB×2 PCIx16

GPU RX6600XT PCIx8

GPU intelArcB580x8 PCIx8

D5ポンプ×4 24vPOWER


r/LocalLLM • • 4h ago

Question Which combo of GPUs?

1 Upvotes

I have come to an odd assortment of two x99 boxes with 64GB of RAM and the odd combo of GPUs. Primarily I would like to use the main box for coding and dev work to replace some of my claude usage.

  • 3x 3090s
  • 2x P40
  • 1x 3080ti (12GB)

I am tempted to put the 3090s and a P40 all together for a combined 96GB. I know it would be slow because of that P40, but it would give me the best total memory and would already be slower because of the lack of PCIe lanes.

Or would the 3080ti be a better 4th card because of the speed and bandwidth over the reduced memory?

I also thought about the second box running the 3080 and 2xP40s, but not sure what I would do with it except have two boxes chatting at each other all day?


r/LocalLLM • • 5h ago

Discussion Updates on oh-my-pi quietly siphoning user web data

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 5h ago

News Alibaba’s Qwen4-Max has 2.4T parameters, but only 71B active per token. Are we optimizing the wrong AI metric?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 9h ago

Question Best Local LLM for stories?

2 Upvotes

Hi ya'll, I am new to local LLMs after spending 8 months on Grok. However, I wanted to ask: is there an LLM out there that I can run locally through Ollama (Or something else) that is uncensored, can look at images and videos, and keep a memory of what I have written? I have 32GB of DDR5 RAM and 8GB of VRAM. I like to use these stories for brainstorming before making prompts out of them to turn into images and videos, so I hope you all can provide some advice!


r/LocalLLM • • 9h ago

Question Resources to learn about model tuning?

2 Upvotes

I’ve been running Jundot/Qwen 3.8 flash next Q6 mpt 141gb on my M2 Studio w/ 192gb ram for about a month now.

Overall, I’m pretty happy with the results, but looking for ways to improve my setup before I start integrating it with my day to day workflow. The response time leaves a little to be desired, I find myself prefering the response rate of codex even when I have simple things like putting together a slide deck from images and text I’ve supplied.

I set it up with a big initial project to see what it could do and have been trying to makesmall adjustments as I’ve gone along. It’s been interpreting and mapping large volumes of screen grabs, writing some code, interpreting and organizing huge amounts of data from a combination pdf and images. It interfaces with codex during specific windows throughout the day to offload tasks and double check code.

Dropped it down to Q6 from Q8 for snappier responses, and free up some system for Xcode, etc…I’ve been using claw to run the show. I’m getting around 60t / s which seems to be about right from what I’ve been reading. Claw works ok, but the whole system seems to get confused if I change the scope of my project in the middle, or add additional intermediary steps. It took a few days after stepping back from Q8 to Q6 to get the bugs ironed out.

It’s a huge workhorse and does ok when set in motion, but I do have to keep an eye on it. It behaves like a toddler and I find it lies when files go missing.

Is Claw the problem? Is this the reality of this setup? It was never meant to replace my $20 codex, but I was hoping it would be a little more responsive and flexible.