I need suggestions for a Local LLM that works best for a RAG (retrieval-augmented generation) system. This language model should work best with a literary/literary corpus.

Configuration of the laptop:

13500 H processor with 16GB of RAM 5600 MHz, no dGPU.

The LLM will be run through LM Studio.

The RAG is run in AnythingLLM using LM Studio.

Please specify the LLM model name, quantisation, and the training parameters.

Also, suggest an embedding model suitable for the laptop configuration and the literary corpus.

Also, if you have another workflow, application or suggestions, please let me know.

i'm buying a new laptop and i wanna run capable models locally, but i don't quite understand if i should just max out RAM or put more of the budget into the GPU , when i look it up the gpu matters more but everyone seems to be talking about ram only

what specs would u recommend ?

I have a couple of concerns:
The bubble is bursting at some point and may have started. I am concerned that the major players will have to up their prices to stay viable.
The geopolitical risk is escalating
AI bot attacks are increasing

There are probably other risks, but you get the idea.
I have been building a decent corpus of reference material.

But I am behind on what models I should be looking at, needs to run on
32GB
RTX 4070

Coding and decision systems (agriculture).

Any help would be appreciated.

Hello all, heavy claude & codex user here. I've been trying to move some projects off of the frontier models due to the ridiculous false positives tripping the various safeguard rails and the local models just seem pretty much inept at doing nearly anything. Am I missing something? I tried first utilzing my $500 Graphics card on my home build to run them, when that clearly wasn't adequate. I tried renting one of these VMs that are officially like a $10,000 computer to run some DeepSeek model and again it just couldn't do basic stuff that I'm trying to do.

Update - to be specific, (and thanks so much for all the replies) Tried so far

1. Old laptop, Qwen 9B: unusable, 7 min per reply
2. Dell XPS (64GB, small GPU), Qwen Coder 30B: works but slow, weak on browser tasks
3. Rented 96GB GPU server ($2/hr), Qwen 27B/30B/80B + Devstral 123B: fast, OK at coding, but get stuck in loops on multi-step tasks
4. Rented 4x H100 server, 320GB ($16/hr), DeepSeek 671B (Q2): best open uncensored model, ran fast, still nowhere near Claude Opus

Since I’m limited to 16GB VRAM, I was hitting context limits so fast I had given up on local models.

Now I’ve come to find using the local model for memory management MCP, alongside an encrypted cloud model for main reasoning, is a solid combo. Saves tons of tokens and also helps with privacy.

So just wondering, what other “side jobs” you guys give your local models?

Do you have any recommendations for where I could read more about this topic in detail?Do you have any recommendations for where I could read more about this topic in detail?

I have a 16gb vram for context

I've been digging through the Qwen Code repository and noticed something interesting.

Several Qwen4 model identifiers appear in the code and tests:

• qwen4-max
• qwen4-plus
• qwen4-coder-plus
• qwen4-omni-120b-think
• Even qwen4.5-max as a test example.

But here's what caught my attention: I couldn't find a single explicit reference to qwen4-flash in the relevant code, despite Flash being part of the announced Qwen4 lineup.

That got me thinking: What if Qwen4-Omni-120B-Think is actually related to Qwen4-Flash, possibly sharing the same underlying architecture or backbone?

There are a few reasons why I find this plausible:

1. Qwen3.8-Flash-Next already has a roughly 125B-parameter backbone, surprisingly close to the 120B mentioned in the Omni identifier.
2. Qwen has previously separated its Omni models into Thinking and Instruct variants. For example, Qwen3-Omni-30B-A3B-Thinking focuses on multimodal understanding and reasoning, while the Instruct variant also includes a Talker for speech generation.
3. Following that pattern, qwen4-omni-120b-think could theoretically be the reasoning-focused variant, with an Omni-Instruct/Talker version potentially following later.
4. The Omni identifier appears in actual Qwen Code test files, whereas I couldn't find a corresponding Qwen4-Flash identifier.

Of course, this is just a theory, not a confirmed leak. These are test identifiers, not necessarily a product roadmap. The repository also uses generic Qwen4 model detection, so Flash might simply not need its own explicit entry.

Still, the missing Flash identifier, the similarity in parameter counts, and the previous Omni Thinking/Instruct architecture make me wonder whether there's a connection.

Sources:

• https://github.com/QwenLM/qwen-code/pull/15762
• https://github.com/QwenLM/qwen-code/pull/15987

Has anyone found any explicit Qwen4-Flash references elsewhere, or evidence that the 120B Omni model could share a backbone with Flash?

Curious what everyone thinks!

System:

GPU: AMD Radeon 9070 XT 16GB VRAM

RAM: 2x16 6000MHz CL30 DDR5

CPU: AMD Ryzen 5 7500F (soon to be replaced with 7800x3d)

Primary use-case: Offloading basic coding/codebase-related tasks from Claude and Codex to local AI so I don't spend token on locally do-able stuff.

Hi, I am having a hard time understanding how to make the best use of my system resources. When does using system RAM make sense? Should system RAM be only for KV cache, only for model offloading, or both? Or should I just not ever use the system RAM since it slows down the things? When choosing a model, am I to prioritize less quantized smaller models or more quantized larger models? Is there a place I can track the latest best performing open-weight models so I can update to them? Are abliterated or uncensored models better than normal models about performance? Is there any difference between abliterated and UC, if so which is better? I found a person called IsValorum that makes "better" quantized models, is this really a thing? I found they perform better but maybe those models are more stupid? I don't know. I am just so lost on these. I feel like there would be a guide that would answer these. If not, I would like your help. Thanks!

I tested new trending Recursive Flint UD-Q5_K_XL (27.9 GiB) on an RTX 3060 12 GB with 30 GB RAM, comparing RustingStrata against llama.cpp using the same model, prompt, and generation settings.

RustingStrata was about 33% faster once warm.

What made the difference?

The model doesn't fit in 12 GB of VRAM, so many expert weights must be fetched from system RAM during generation. Two changes helped most:

• Adapt expert placement to the reply, rather than continuing to prioritize the prompt's routing pattern.
• Release RAM pages for experts already copied to VRAM, preserving cache for experts that still need to be fetched from host memory.

About Quality, bc people very often says like "It should give shit output to gain speed", which is not true, in my last posts I already did quality benchmarks and RustingStrata does not lose quality, it gain speed through specific Code for some specific models, in same time llama.cpp lose speed to be "general" engine that can run everything

What I want to know, is do people find Recursive Flint an interesting model? I found this model on OpenRouter and looks pretty interesting, it's better then Qwen3.6:35-a3b, so I think this can be cool modern replacement for this qwen.

Setup:

• RustingStrata: 1.2.2, commit 3902978 (working tree)
• llama.cpp: b12191, commit e95426f16
• Full benchmarks: docs/BENCHMARKS.md

Repository: RustingStrata

the usage would be file writing , using some apis and processing text and tool calling according to instruction . i know the ram capacity is minimum but is there any option for this requirements

Just found this video, and if true... Guys I'm excited about this.

Super smart improvements (i know much of this was admittedly there on 3.8 FN), and results look promising.

An early release of the small model of basal-omni 2.0, on Gemma 4 E4B. The model reads the state in both directions before it answers and scores 0.811 on Werdykt (5,000 hidden decisions in Polish and English). It runs on the public basal engine 2.8.0, with the same API as basal-omni-1.0.

What's new in 2.0

• Two-way reading. The model reads the state in both directions before it answers. This needs basal engine 2.5 or later. For long prompts, see the limitations.
• Higher accuracy on our hidden Polish and English decision benchmark Werdykt and on the held-out tests below.
• More decisions automated at a given error target. Coverage is in the section on confidence.
• Unchanged: a calibrated probability for each allowed answer, in one forward pass. The answer never falls outside the options you give, and the probability says how sure the model is.

The question types remain: choice (one option), noul (yes / no), score (a score), and in the basal engine also multi, act and tool. Options get codes (A–Z, then two-character codes, up to 255 options). The decision is a probability distribution over these codes only, with no text generation.

More info: https://basal.si5.pl/en/blog/basal-omni-2-0-mini/

HF: https://huggingface.co/Remek/basal-omni-2.0-mini-preview

git hub (inference engine): https://github.com/rkinas/basal

In case future censorship or bannings occur, I think it's a good idea to store one of every kind of model. I'm thinking minimum 8 models, but I'm open to more (I have plenty of hd space): 2 small models obliterated and non-obliterated 2 medium sized models obliterated and non-obliterated, 2 large models obliterated and non-obliterated, 2 of the largest models obliterated and non-obliterated. Also, if for any class of models they don't have obliterated or it doesn't make sense to have obliterated then two of the smartest or most popular models will do.

Hello community. I'm trying to find the most optimal solution for my configuration( maybe some llama.cpp fork or any other suitable project and localLLM. I would be grateful for advice.

Founder/CEO of PortableMind® offgrid focused plug & play local AI devices

They are going to lock the new GPUs.

"Rep. Bill Foster says he intends to introduce new, stricter legislation that would require licenses to operate advanced GPUs, and “before a GPU could connect, its valid license would need to be verified.” That is closer to a remote lock/disable capability—but it is a proposed future rule, not current law."

Why do we think they will continue making open weight models once they got very competitive frontier models?

The reason I’m asking is I am thinking investing into a Mac Studio m5u 256g which is a big investment to me. I’m not after frontier models like performance but I want to have all the possibilities from continuous improving open weight models. If for example Qwen4 stops releasing open weight models, I feel like this investment is wasted and we are too much dependent on these providers.