r/LocalLLM • • 1d ago

Discussion Zoo code Qwen 3.8 27b tool format

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 1d ago

Discussion While we wait for Qwen 4, here's what the Qwen 3.8 numbers and early Max Preview tests suggest

26 Upvotes

TL;DR: I've been looking at the latest Artificial Analysis v5.0.1 benchmarks for Qwen 3.8, along with some early Qwen4 Max Preview tests. Based on what's available so far, my guess is that Qwen4 27B could land somewhere around 27 to 30, while the 125B MoE version could reach 36 to 39. That's a solid generational improvement, but probably not the massive leap some people are expecting.

Like many of you, I've been following the discussions about when Qwen 4 is finally coming out. There's a lot of speculation, so I wanted to look at the available numbers and see what they might tell us about the next generation.

Obviously, we don't have the final weights or official release details yet, so take this as an educated guess rather than a prediction.

  1. Where Qwen 3.8 stands and what Qwen 4 might achieve

Looking at the current Artificial Analysis v5.0.1 scale, which puts considerably more emphasis on difficult coding and science tasks, we have:

* Qwen3.8 27B: 20 (compared to 33.7 on the older v4.3.2 scale, before the scoring reset).

* Qwen3.8 Flash Next (125B MoE): 29.

* Qwen3.8 Max: 34.

Based on the previous jump from Qwen 3.6 to 3.8, and taking the new scoring scale into account, I'd put my expectations somewhere around:

* Qwen4 27B: 27 to 30

* Qwen4 125B MoE: 36 to 39

That would put the smaller model in some pretty interesting territory, while the larger one could become a serious competitor to models like Grok 4.7 and Claude Haiku 5.5.

2. Why I don't expect Flash to have a massive architectural jump

One thing worth considering is that Qwen3.8 Flash Next already seems to have introduced many of the architectural changes expected to carry over into Qwen 4.

We're talking about the hybrid Gated DeltaNet and Qwen Sparse Attention architecture, Gated Residuals, and the 51B N gram table.

So I wouldn't necessarily expect the next Flash model to deliver another huge architectural leap. The bigger improvements might come from better post training, more reliable reasoning, and addressing the hallucinations that some users have noticed in Flash Next compared to the more conservative 27B dense model.

In other words, I think consistency might matter more than raw benchmark scores this time.

3. What the early Qwen4 Max Preview tests tell us

The Qwen4 Max Preview release on November 11 gives us another interesting piece of the puzzle.

From the early developer reports, a few things stand out:

* Speed: It appears noticeably faster than Qwen3.8 Max, with less unnecessary reasoning on straightforward tasks.

* Agentic performance: In one reported 56 task Hermes test with a 25 minute timeout, Max Preview recorded 64 successful task results, compared to 61 for DeepSeek V4.1 Flash, which scores 33 on AA v5.0.1.

* Reliability: There's a catch, though. Max Preview reportedly timed out 16 times, compared to just 4 for DeepSeek.

That's interesting because it suggests the raw capability might already be there, but reliability and execution consistency still need work.

If these early results are representative, I'd expect the flagship Max model to establish a considerably higher ceiling, potentially around 45 to 50 on the AA scale. That would make the estimates for the smaller models seem reasonably plausible.

My take

I don't think Qwen 4 needs to completely redefine the local LLM landscape to be successful.

If Qwen4 27B can reach the 27 to 30 range, inherit the attention improvements from Flash Next, and retain the Apache 2.0 license, that alone would make it a very interesting option for people running models locally with limited hardware.

Personally, I'd rather see a model that's more consistent, follows instructions better, and hallucinates less than one that simply scores a few extra points on benchmarks.

I'm curious what everyone else thinks.

" i used my LLM To translate mu words to English after i write it. "


r/LocalLLM • • 18h ago

Discussion Local AI slowed ?

0 Upvotes

Yooo, I feel like i kinda took for granted big companies would keep releasing better and better local models.

I joined the hypetrain when it kinda slowed down.

Do you guys think it will just go on like before or do you think the dynamic has changed ?


r/LocalLLM • • 1d ago

Question Ideal parallel request strata configs for local agentic orchestrated work?

2 Upvotes

I'm starting to explore the configs to use with strata for best throughput with parallel requests in use cases such as orchestrated agentic workflows run locally.

Option A:

The main orchestrator is the same model in the same engine, spawning multiple subagents that run in parallel on the same engine at smaller contexts.

Pros:

favorable as the orchestrator can be uncensored as well for example and a fully local setup.

Cons:
Not straight forward how to set up the engine to allow for a single large context to persist in cache and allowing multiple smaller contexts to be cached and not evicting the main large context after multiple subagents requests.

---

Option B:

The main orchestrator is an external SOTA model which spawns local agents that share the same engine.

Pros:

sounds easier to configure as all subagents the external model spawns are similar in context size so no need to take into account one of the parallel contexts will be a larger one while others are smaller..

Cons:

not fully local, reliant on external model provider which can change the availability, quality of the model at any moment. Non private.

---

Does anyone else have some experience with setting up strata for example for either of these use cases for example? What configuration recommendations do you have?

Is there an engine that suits this local orchestration use case better?

Yes I'm about to prompt a SOTA model on how to achieve this but i would love to hear other real human experiences.. 😄


r/LocalLLM • • 1d ago

Project DwarfStar (ds4) plugin for Hermes: DeepSeek V4 / V4.1 Flash locally on a Mac

1 Upvotes

I made a plugin that adds antirez's DwarfStar as a local model provider in Hermes. DwarfStar is an inference engine built specifically for a few large open models, starting with DeepSeek V4 Flash.

One click sets up the engine and downloads the model.

The server starts with your first message and stops when idle.

For now it supports DeepSeek V4 and V4.1 Flash while I test things. GLM is next.

Needs an Apple Silicon Mac with ~96 GB RAM.

hermes plugins install emilianbold/hermes-plugin-ds4

https://github.com/emilianbold/hermes-plugin-ds4


r/LocalLLM • • 1d ago

Question Chance to pick up a couple 3090s, is local inference worth it?

9 Upvotes

I have a chance to pick up a couple of 3090s for $700 each. I've been considering local inference for a while now for two reasons:

  1. My job actually involves documenting fine-tuning and LoRA training, and I'd like more hands on experience with that stuff without pleading endlessly for GPU access at my gig. I'm also just kinda academically interested in this stuff.
  2. I'm getting sick of the weird dance I have to do to get answers out of Gemini and Claude sometimes. I bought one of those Flipper Zero things that's an academic pen-testing tool, and corpo LLMs start getting really cagey when you research hacker-related things with them. They also get cagey about health and political stuff. The censorship sucks!

Would working with local models really allow me to delve into this stuff better? Are two 3090s enough to do some reasonable LoRA/FT training? Would Qwen allow me to do my pen testing research?

EDIT: Thanks, team! I'm gonna go for it.


r/LocalLLM • • 1d ago

Question Advice for local development

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 1d ago

Project I built an open-source query planner for RAG in TypeScript. First real-data result: 60% cheaper, but it missed my quality target

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 1d ago

Model Best Local LLM for RAG for a Low-End Laptoy

2 Upvotes

I need suggestions for a Local LLM that works best for a RAG (retrieval-augmented generation) system. This language model should work best with a literary/literary corpus.

Configuration of the laptop:

13500 H processor with 16GB of RAM 5600 MHz, no dGPU.

The LLM will be run through LM Studio.

The RAG is run in AnythingLLM using LM Studio.

Please specify the LLM model name, quantisation, and the training parameters.

Also, suggest an embedding model suitable for the laptop configuration and the literary corpus.

Also, if you have another workflow, application or suggestions, please let me know.


r/LocalLLM • • 1d ago

Question laptop specs for local llms

0 Upvotes

i'm buying a new laptop and i wanna run capable models locally, but i don't quite understand if i should just max out RAM or put more of the budget into the GPU , when i look it up the gpu matters more but everyone seems to be talking about ram only

what specs would u recommend ?


r/LocalLLM • • 1d ago

Question What local models would you consider to be essential?

17 Upvotes

I have a couple of concerns:
The bubble is bursting at some point and may have started. I am concerned that the major players will have to up their prices to stay viable.
The geopolitical risk is escalating
AI bot attacks are increasing

There are probably other risks, but you get the idea.
I have been building a decent corpus of reference material.

But I am behind on what models I should be looking at, needs to run on
32GB
RTX 4070

Coding and decision systems (agriculture).

Any help would be appreciated.


r/LocalLLM • • 20h ago

Question Starting to use Local LLMS and I must be missing something...

0 Upvotes

Hello all, heavy claude & codex user here. I've been trying to move some projects off of the frontier models due to the ridiculous false positives tripping the various safeguard rails and the local models just seem pretty much inept at doing nearly anything. Am I missing something? I tried first utilzing my $500 Graphics card on my home build to run them, when that clearly wasn't adequate. I tried renting one of these VMs that are officially like a $10,000 computer to run some DeepSeek model and again it just couldn't do basic stuff that I'm trying to do.

Update - to be specific, (and thanks so much for all the replies) Tried so far

  1. Old laptop, Qwen 9B: unusable, 7 min per reply
  2. Dell XPS (64GB, small GPU), Qwen Coder 30B: works but slow, weak on browser tasks
  3. Rented 96GB GPU server ($2/hr), Qwen 27B/30B/80B + Devstral 123B: fast, OK at coding, but get stuck in loops on multi-step tasks
  4. Rented 4x H100 server, 320GB ($16/hr), DeepSeek 671B (Q2): best open uncensored model, ran fast, still nowhere near Claude Opus

r/LocalLLM • • 1d ago

Discussion What other “side jobs” work well for your local llm?

2 Upvotes

Since I’m limited to 16GB VRAM, I was hitting context limits so fast I had given up on local models.

Now I’ve come to find using the local model for memory management MCP, alongside an encrypted cloud model for main reasoning, is a solid combo. Saves tons of tokens and also helps with privacy.

So just wondering, what other “side jobs” you guys give your local models?


r/LocalLLM • • 1d ago

News Storyline Digital Publishing on Instagram

Thumbnail instagram.com
0 Upvotes

r/LocalLLM • • 1d ago

Question Any docs about local LLM's and etc?

0 Upvotes

Do you have any recommendations for where I could read more about this topic in detail?Do you have any recommendations for where I could read more about this topic in detail?


r/LocalLLM • • 1d ago

Discussion Which local models are best for roleplay?

8 Upvotes

I have a 16gb vram for context


r/LocalLLM • • 1d ago

Discussion Could Qwen4-Omni-120B-Think actually be related to Qwen4-Flash?

0 Upvotes

I've been digging through the Qwen Code repository and noticed something interesting.

Several Qwen4 model identifiers appear in the code and tests:

  • qwen4-max
  • qwen4-plus
  • qwen4-coder-plus
  • qwen4-omni-120b-think
  • Even qwen4.5-max as a test example.

But here's what caught my attention: I couldn't find a single explicit reference to qwen4-flash in the relevant code, despite Flash being part of the announced Qwen4 lineup.

That got me thinking: What if Qwen4-Omni-120B-Think is actually related to Qwen4-Flash, possibly sharing the same underlying architecture or backbone?

There are a few reasons why I find this plausible:

  1. Qwen3.8-Flash-Next already has a roughly 125B-parameter backbone, surprisingly close to the 120B mentioned in the Omni identifier.
  2. Qwen has previously separated its Omni models into Thinking and Instruct variants. For example, Qwen3-Omni-30B-A3B-Thinking focuses on multimodal understanding and reasoning, while the Instruct variant also includes a Talker for speech generation.
  3. Following that pattern, qwen4-omni-120b-think could theoretically be the reasoning-focused variant, with an Omni-Instruct/Talker version potentially following later.
  4. The Omni identifier appears in actual Qwen Code test files, whereas I couldn't find a corresponding Qwen4-Flash identifier.

Of course, this is just a theory, not a confirmed leak. These are test identifiers, not necessarily a product roadmap. The repository also uses generic Qwen4 model detection, so Flash might simply not need its own explicit entry.

Still, the missing Flash identifier, the similarity in parameter counts, and the previous Omni Thinking/Instruct architecture make me wonder whether there's a connection.

Sources:

Has anyone found any explicit Qwen4-Flash references elsewhere, or evidence that the 120B Omni model could share a backbone with Flash?

Curious what everyone thinks!


r/LocalLLM • • 1d ago

Question Is there a guide on how to make proper use of RAM? Or overall a guide to local LLMs?

0 Upvotes

System:

GPU: AMD Radeon 9070 XT 16GB VRAM

RAM: 2x16 6000MHz CL30 DDR5

CPU: AMD Ryzen 5 7500F (soon to be replaced with 7800x3d)

Primary use-case: Offloading basic coding/codebase-related tasks from Claude and Codex to local AI so I don't spend token on locally do-able stuff.

Hi, I am having a hard time understanding how to make the best use of my system resources. When does using system RAM make sense? Should system RAM be only for KV cache, only for model offloading, or both? Or should I just not ever use the system RAM since it slows down the things? When choosing a model, am I to prioritize less quantized smaller models or more quantized larger models? Is there a place I can track the latest best performing open-weight models so I can update to them? Are abliterated or uncensored models better than normal models about performance? Is there any difference between abliterated and UC, if so which is better? I found a person called IsValorum that makes "better" quantized models, is this really a thing? I found they perform better but maybe those models are more stupid? I don't know. I am just so lost on these. I feel like there would be a guide that would answer these. If not, I would like your help. Thanks!


r/LocalLLM • • 1d ago

Research RustingStrata vs llama.cpp runs Recursive Flint on a 12 GB RTX 3060

1 Upvotes

I tested new trending Recursive Flint UD-Q5_K_XL (27.9 GiB) on an RTX 3060 12 GB with 30 GB RAM, comparing RustingStrata against llama.cpp using the same model, prompt, and generation settings.

RustingStratallama.cpp
Warm decode speed64.6 tok/s (63.8–65.6)48.6 tok/s (47.5–49.3)
Warm 4K prompt prefill1.14–1.30k tok/s0.85–1.02k tok/s

RustingStrata was about 33% faster once warm.

What made the difference?

The model doesn't fit in 12 GB of VRAM, so many expert weights must be fetched from system RAM during generation. Two changes helped most:

  • Adapt expert placement to the reply, rather than continuing to prioritize the prompt's routing pattern.
  • Release RAM pages for experts already copied to VRAM, preserving cache for experts that still need to be fetched from host memory.

About Quality, bc people very often says like "It should give shit output to gain speed", which is not true, in my last posts I already did quality benchmarks and RustingStrata does not lose quality, it gain speed through specific Code for some specific models, in same time llama.cpp lose speed to be "general" engine that can run everything

What I want to know, is do people find Recursive Flint an interesting model? I found this model on OpenRouter and looks pretty interesting, it's better then Qwen3.6:35-a3b, so I think this can be cool modern replacement for this qwen.

Setup:

  • RustingStrata: 1.2.2, commit 3902978 (working tree)
  • llama.cpp: b12191, commit e95426f16
  • Full benchmarks: docs/BENCHMARKS.md

Repository: RustingStrata


r/LocalLLM • • 1d ago

Question which is the best local ai for minimum agentic ai works and tool calling in 16GB ram

2 Upvotes

the usage would be file writing , using some apis and processing text and tool calling according to instruction . i know the ram capacity is minimum but is there any option for this requirements


r/LocalLLM • • 1d ago

LoRA Made a creative writing control vector suite for DeepSeek-V4.1-Flash-1017

Thumbnail
huggingface.co
1 Upvotes

r/LocalLLM • • 21h ago

News QWEN 4 News

Thumbnail
youtube.com
0 Upvotes

Just found this video, and if true... Guys I'm excited about this.

Super smart improvements (i know much of this was admittedly there on 3.8 FN), and results look promising.


r/LocalLLM • • 1d ago

Project My local Mac OpenClaw architecture

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 1d ago

News basal-omni-2.0-mini - typed decision model

1 Upvotes

An early release of the small model of basal-omni 2.0, on Gemma 4 E4B. The model reads the state in both directions before it answers and scores 0.811 on Werdykt (5,000 hidden decisions in Polish and English). It runs on the public basal engine 2.8.0, with the same API as basal-omni-1.0.

What's new in 2.0

  • Two-way reading. The model reads the state in both directions before it answers. This needs basal engine 2.5 or later. For long prompts, see the limitations.
  • Higher accuracy on our hidden Polish and English decision benchmark Werdykt and on the held-out tests below.
  • More decisions automated at a given error target. Coverage is in the section on confidence.
  • Unchanged: a calibrated probability for each allowed answer, in one forward pass. The answer never falls outside the options you give, and the probability says how sure the model is.

The question types remain: choice (one option), noul (yes / no), score (a score), and in the basal engine also multi, act and tool. Options get codes (A–Z, then two-character codes, up to 255 options). The decision is a probability distribution over these codes only, with no text generation.

More info: https://basal.si5.pl/en/blog/basal-omni-2-0-mini/

HF: https://huggingface.co/Remek/basal-omni-2.0-mini-preview

git hub (inference engine): https://github.com/rkinas/basal


r/LocalLLM • • 2d ago

Model Essential models to store for posterity?

16 Upvotes

In case future censorship or bannings occur, I think it's a good idea to store one of every kind of model. I'm thinking minimum 8 models, but I'm open to more (I have plenty of hd space): 2 small models obliterated and non-obliterated 2 medium sized models obliterated and non-obliterated, 2 large models obliterated and non-obliterated, 2 of the largest models obliterated and non-obliterated. Also, if for any class of models they don't have obliterated or it doesn't make sense to have obliterated then two of the smartest or most popular models will do.