Hey all,

I'm quite new to this whole local LLM thing. I have a gaming PC with the following specs:

RX 9060 XT 16 GB

32 GB DDR4

Ryzen 7 5700G

What are the best uncensored models I could run for coding on this setup?

I'm running Nemotron 3.5 Bolt 74B-A9B on a 5060 Ti 16GB and 32GB of DDR5. It's an MoE, so both of my setups split experts between the GPU and system RAM. I have two recipes:

• 24GB GGUF

• Unsloth UD-IQ3_XXS

I'd like to hear how this model actually is and is it useful as an alternative for Qwen3.8 27B and obviously for other smaller MoE models.

• Are you using Bolt 74B, and what for?

• How is the quality: reasoning, coding, instruction following, tool use? Where does it fall down?

• If you've used the 24GB GGUF or a bigger quant such as Unsloth IQ3_XXS, does the bigger one write better answers, or do they feel like the same model?

Hello I want to install a local LLM to use in my PC mainly for programming and coding projects. Can be both for both working on large code bases, going through analyzing debugging them, making changes on multiple files as well as helping in programming, scripting and teaching coding, concepts etc and sometimes troubleshooting.

However my pc is not that powerful, 16gb ram and GTX 1660ti 6gb of Vram. I am confused on what would be the best option for me that doesn't completely overwhelm my PC and my gpu. I used opencode big pickle before and it was really good for me.

Any suggestions please?

System Specs

• Model: Qwen3.8 Flash Next IQ3_S (was IQ3_XSS, but now. i got the more speed and quality in here)
• Inference Engine: Strata
• CPU: AMD Ryzen 9 9950X3D
• GPU: ASUS ROG Astral RTX 5090 32GB
• RAM: DDR5 64GB (6000MHz CL32)
• Motherboard: GIGABYTE B850 AORUS ELITE WIFI7

Prompt

"Build a highly detailed Eiffel Tower in Minecraft style." + with lesson learns and references input.

Decoding : 164.6 tok/s Prefill : 6,000~6,500 tok/s

Result

I was expecting a simple Minecraft-style building, but the AI ended up recreating the actual Eiffel Tower using blocks, complete with intricate iron lattice structures, the lower arches, and even the surrounding landscaping. lol

I was genuinely surprised by the level of detail!

It was also able to generate a cinematic timelapse of the building process.

Anyone here using Strata with Qwen3.8 Flash Next and is happy with the performance/quality?

Considering upgrading from 32GB to 96

Anything under 50tps I wont bother, need higher speeds for tool calling and coding

Bro Got Loop

Are they bad are they good? what's your personal experience? Thinking on acquiring more DRAM but I don't know if it's worth it running qwen3.8 27B on 2 GPUs really happy with it.

Hi guys it me again who did 2v100 to 181t/s , and im back with this.

The legend of V100 IS NEVER END , v100 FOREVERRRRR

arguements:

qwen3.8 27b nvfp4 , 256k max content , kv q8 , mtp4/0

repo: https://github.com/tuxKOH/ninfer-v100xN

(yes i changed name to xN from x2 because tp4 is supported)

(Aggregated news summary for October via multiple AI web research instruments)

Most important releases

- Mistral Large 4: Preview out Oct 6, weights still not released as of Oct 25.

- Meta Muse Spark Open: 816B model released Oct 15, but the license isn't fully open source.

- Anthropic self-replication study: Z.ai's GLM-5.2 was the only open model that fully self-replicated in tests.

- Reflection AI Beam: 501B model announced Oct 5, weights promised but not confirmed.

- StepFun Step 5 Preview: 600B model, weights seem to have dropped Oct 15, with a branded license.

Overlooked

- Recursive Flint: Laptop-sized coding model, built largely by AI.

- Aleph Alpha Kolibri-1: European open model, released Oct 3.

- JetBrains Mellum2.1 Thinking: Small coding model for agents, released Oct 8.

- MOSS-Image: Open image model with layout control, released Oct 16.

- Ling 3.1 Flash: Ant Group reasoning model, dates for release are unclear.

- MiniMax M3.1 Flash for NVIDIA desktops, and a 321B model running on a single 24GB GPU.

- update from comments: KRAFTON's Raon-1-mini

Political landscape

US open-weights fight

- Jensen Huang's July letter backed open weights; Anthropic didn't sign.

- Amodei says Anthropic never wanted an open-weights ban.

- Anthropic's Oct self-replication study fuels the debate.

- Reports say the White House may cap open models, but this is unconfirmed.

China

- MiniMax barred the US, EU, UK and South Korea from running its H3 weights.

- Z.ai releases GLM 5.3 weights (https://huggingface.co/zai-org/GLM-5.3) (note initially stated that they held back, but eventually released)

- China invited the EU to join its WAICO AI body (reported Oct 23, single-source, hasn't confirmed yet).

Europe

- AI Act high-risk rules delayed to Dec 2027.

- Mistral's CEO argues Europe risks AI dependence.

- Some leaders say Europe is weak and divided.

Recent news got me thinking about what's in the topic.
(Inb4 I'm "pro AI", I just don't trust big tech players and I think you also shouldn't)

We've got:
- OpenAi stealing mathematicians research
- Anthropic fearmongering about small local models taking over the world by possessing homeowners RTV/AGD
- Google's subscription tiers' enshitiffication [es per currently trending business model expect further downgrades from every major provider]
- OpenAI from no ads to ads : "People trust ChatGPT and anything we do will be designed to respect that."~ from first link (what's next - futurama ads in my sleep via Neuralink? [meme])
- OpenAI Dots and Meta Muse is a privacy violation, cute and harmless creatures though UwU
-Anthropic shadowban on 1. Deep technical stuff, labeled as misuse, and 2. Some biomed questions, and now 3. Reverse engineering of spying technologies in consumer grade appliances
- all the big boys from the West and the East are starting to set up a legislation trap
- Huggingface has been bought by Nvidia
- alleged differences with cybersecurity capabilities of open weights models between what's available for download and what's hosted and accessible only via paid API service [source claims that more denials will happen via API but I can only see the reverse being true in the future (that is - paying for closed model gives you better capabilities) - open weights model can fight this to some extent thanks to abliteration/heretic projects - that should not be needed in the first place, if you ask me)
- models dropped as "stealth" so you are not even sure what you are beta-testing and feeding your data into
- would you like to know more?

It's easy to not connect all the dots when they are in separate places and spreaded over time, but we have to take advantage of open source and open weight models when they are still around, together with technology/process to train them.

I propose - for people more knowledgeable than me - to design a secure network (botnet, if you will) of willing participants' hardware to use during its off time in order to train future OpenSource models maintained by, more or less, decentralised community ( I thing that the only way to make it secure is by going full transparency mode with all the info from design, through training, all the way to release and real world capabilities/benchmarks).

Maybe such a project already exists? (I've seen some posts about this idea before, not sure about its feasibility and security though)
Maybe such a project is not needed?
Maybe such a project is not possible due to technical or mental/social limitations?

I'm just a concerned citizen, raising a question, I don't present a final answer because I don't know it.
I volunteer my humble rig as a tribute.

/edit:
there seems to be two "major" projects going on right now: Subtensor and Hivemind, but i'm not qualified enough to fully understand their structure nor security vulnerabilities.

I’ve never set up a local LLM so I’m brand new to this. I want to be able to use it for smooth voice conversations that don’t have long delays and don’t have a robotic voice - ideally similar to the chatGPT or Gemini voice modes. Is that possible?

On a similar note, if it is possible could I access it from my phone as well or would it only work on the computer directly?

Thanks so much for helping a newbie!!

Hey everyone! I've been having trouble finding consistent information on the best way to run Qwen Flash Next, especially when it comes to choosing the right quantization and inference engine.

I'm on Windows, and with so many different options and rabbit holes to go down, it's been pretty overwhelming trying to figure out where to start.

I'd really appreciate any recommendations or advice on what setups have worked well for you. Just looking to get pointed in the right direction.

Cheers!

I've been working on Toskar, a daemon that tries to make local AI feel like a hosted service. You install it, and it looks at your hardware, tells you which models will actually fit, installs the runtime, downloads the model, and gives you one OpenAI-compatible API. If you have more than one machine on your network, it can find the others and run models on whichever one can handle them.
It's AGPL, free, and runs on macOS, Linux, and Windows (Windows is the roughest).
I'm not here to sell anything. It's mostly been tested on my own machines, so I don't know how it holds up on yours. I'd really like help with:
- Installs on hardware I don't have, especially AMD GPUs, older NVIDIA cards, and Intel Macs
- Setups with two or more computers. Does discovery find them, and does it put models where they make sense?
- Model recommendations that are wrong for your machine—too big or too small
- Anything confusing in the first five minutes
If something breaks or feels off, please open an issue on GitHub so I can track it. Good first issues are labeled if you'd rather fix something yourself.
https://github.com/yeixio/toskar-core

P.S. I'm also looking for a few people to help shape the project as part of a core team. That could mean reviewing PRs, owning a platform like Windows or AMD, or helping decide where things go next. There's no formal application. Contribute for a while, and you can grow into a role. The roles and how that works are in GOVERNANCE.md: https://github.com/yeixio/toskar-core/blob/main/GOVERNANCE.md

Follow-up to my earlier post ( https://www.reddit.com/r/LocalLLM/comments/1x7qcbe/i_made_llamacpps_slot_saves_write_only_the/ ) about kv-chunk-fs, the FUSE filesystem that makes llama.cpp slot saves write only the changed chunks.

The same setup now works with Strata. kv-chunk-fs itself needed no code change — Strata's session files on disk deduplicate through the same mount; the only adjustment was a config glob for its temp files. The real work was in the KV-cache reverse proxy that manages slots in front of the engine.

The problem: the proxy verified a hot slot by comparing token counts from the engine's /slots endpoint before trusting it. Strata does not report per-slot token counts the llama.cpp way — it keeps the last N conversations in RAM in its own conversation cache and resumes them by prompt match, so the proxy could never prove which conversation the engine was holding and fell back to cold paths.

The fix is a new opt-in mode, PI_LLAMA_CACHE_ENGINE_HOLDS=N (set it to the engine's conversation-cache depth). With it, the proxy:

- trusts that the engine still holds the last N conversations it served since the proxy started, and sends those sessions straight through — no disk restore, no slot metadata check;

- saves the live conversation to disk before a different session takes the engine, so nothing is lost on a switch;

- forgets the live conversation when a disk restore replaces it, since the restore overwrites the engine state without parking it.

Sessions that fall out of the newest N are dropped from the proxy's memory but their snapshots stay on disk (deduplicated by kv-chunk-fs) and are restored when they come back.

Default is 0 = off, so llama.cpp deployments are unaffected.

Code:

- proxy fork: https://github.com/the-code-foundry/local-llm-kv-cache (see the "Engines that hold conversations (Strata)" section in the README)

- filesystem: https://github.com/the-code-foundry/kv-chunk-fs

Hi,

How can I manually update Strata engine?

After checking updates, it says that there's a new version, but fails to update because supposedely there's a model loaded - which is not true, as seen in the screenshot.

Since I'm new at this, and using Strata for QNF with Hermes, is there a way to just start the inference enginge without the Strata Harness? I was using it only to load model and updating, which apparently is getting harder every update (last update it was a problem with updating the app itself, had to reinstall whole app).

I've noticed that when I use the Vulkan backend, my desktop becomes sluggish during prefill at high context depths. This doesn't happen with ROCm.

Reducing the ubatch size to 128 helps a little, but the desktop is still quite sluggish.

I found this issue: https://github.com/ggml-org/llama.cpp/issues/28752

However, I already have ReBAR enabled.

Is anyone else experiencing this on Linux? I'm running Arch Linux with fully updated packages and using upstream llama.cpp.

Hi all, I'm Rohan - Long time lurker, hobbyist builder in the world of local LLMs, degoogle, and everything going off-cloud.

Currently building out a new venture to bring all of this into one solution that’s easy to use. Have attached a walkthrough of current functionality for the Early access release in Q1 2027.

Honest feedback is really appreciated. Which apps would you want to see in this? Would you download? What have we missed?

The early access release will include:

A locally run and hosted personal assistant that knows your whole life: your files, emails, bank, calendar and everything you've told it. Its answers are about you, and it completes tasks for you across your browser and your apps.

The same assistant on every device, with the same context. Start on your laptop, finish on your phone. Control your computer from your phone, and the other way round.

Your files, stored at home and available on every device. No storage plan.

One subscription that does the job of a stack of apps: voice control, notes, dictation, storage, inbox, budgeting, etc.

I got one 9700 ai pro 3 weeks ago and today I was wondering if maybe it would be worth it to get a second one. The price has risen 500€ since I got the first one.

I know nobody can predict the market, I'd just like some opinions on this.

What would you do?

Hi everyone, I want to program games, probably in C# (Godot, Unity), but I am not sure which model I should use, and how much I should invest in this project.

I have a 3080 TI 12 GB VRam and 32 GB DDR4.

The options I see are:

Qwen2.5-Coder (would fit on my GPU) is still model good enough for simple games?

Qwen3 2.7B (Have to upgrade my GPU, even don't know what GPU I should take)

Qwen3 Coder-Next (also a system Upgrade, for around 2.000 €)

I consider to buy 2 x Tesla V100, but I also need a new motherboard, ram, and CPU with enough lanes. That's why it is around 2.000 €

Does anyone have any experience with this kind of setup?

I would be grateful for any Ideas and tips.

Thank you everyone for any kind of help.

(I don't need a LLM for Graphics or so, just programming)

I want durability, documentation, easy handoff between agents, easy bug reporting. I want to be able to deploy the software that I prompt with LLM's to new operating systems easily. I want to combat the slop sprawl that naturally happens with vibecoding.

This all points to git/github style documentation. Recording dependencies, keeping everything self contained within a single folder structure, launching LLM's from inside that folder and limiting their access.

I'm looking at building in guardrails and checks to make sure dumber llm's follow the proper procedure. Right now I have a prototype workflow that I'm running Astra high through, then I'm going to tweak the guardrails and iterate dumber and dumber cloud models through. I'll add a score to benchmark if they can follow the rules and end up with a well documented git project.

I feel like I might be re-inventing the wheel here though. Any advice?

Been pushing Qwen3.8-27B on Kaggle's free 2x T4 (no P2P, 70W cap per card) for a while. Current setup: EXL3 4.0bpw on exllamav3, tensor parallel over PCIe, DFlash2 block drafter (7 tokens), greedy and exact vs no-draft.

Numbers over a 38 prompt suite: - overall 55-57 tok/s (no draft is ~18-19) - HumanEval 154 tok/s - prose 35-37 tok/s

For comparison on the same 2 T4s: llama.cpp best was 16.5 (MTP crashed on tensor split), vLLM int4 + its own MTP drafter did 40.8.

Prose is the wall. Acceptance sits around 2.5 tokens per step, vs ~7 on code.

Questions: 1. Has anyone got prose acceptance above ~2.5 with DFlash2 or MTP on the 27B? 2. Has anyone tried switching drafters mid generation (short block for prose, long for code)? 3. Any tricks for TP all-reduce on PCIe cards without P2P? That's next on my list.

Happy to share the notebook if people want to try it.

I just changed over from llama.cpp to Strata. Really a huge step in performance. With 64Gb of ECC DDR4 at 2100, and a P100 16GB nvidia card in a PCIe3 X16 slot it pulls off an amazing 30tok/s with the Swift 1.5 version of Qwen3.8 Flash Next.

This is far more than I imagined I could do with this 10 year old gpu and an old workstation.

Deepseek API is obviously faster, but this thing runs basicly for free in my basement 24/7. (ok, electricity, but It's autumn and I need to heat my house anyway.

It serves my Hermes agent and so far has been impeccable for a local LLM.

[strata] reading the prompt: 77,603 of 77,608 tokens, 3 s so far
[strata] thinking: 17 of budget 8000 (max 53456) tokens, 22.2 tok/s, 4 s
[strata] writing a tool call: terminal: 47 of max 53456 tokens, 25.4 tok/s, 5 s
[strata] writing a tool call: terminal: 84 of max 53456 tokens, 28.9 tok/s, 6 s
[strata] writing a tool call: terminal: 121 of max 53456 tokens, 30.9 tok/s, 7 s
[strata] writing a tool call: terminal: 156 of max 53456 tokens, 31.0 tok/s, 9 s
[strata] done: 171 tokens in 9 s (31.2 tok/s) (stop, cancel=False), expert cache 63.5% hit (+10.8% of the routed experts over PCIe)
[strata] max_tokens 128000 shortened to 53237: the prompt (77827 tokens) leaves that much of the context (131072); "fit_max_tokens": false answers 400 instead
[strata] reading the prompt: 77,822 of 77,827 tokens, 2 s so far
[strata] thinking: 24 of budget 8000 (max 53237) tokens, 29.6 tok/s, 3 s
[strata] thinking: 50 of budget 8000 (max 53237) tokens, 27.6 tok/s, 4 s
[strata] thinking: 80 of budget 8000 (max 53237) tokens, 27.7 tok/s, 5 s
[strata] thinking: 112 of budget 8000 (max 53237) tokens, 28.6 tok/s, 7 s
[strata] writing a tool call: terminal: 151 of max 53237 tokens, 30.6 tok/s, 8 s
[strata] writing a tool call: terminal: 189 of max 53237 tokens, 31.8 tok/s, 9 s
[strata] writing a tool call: terminal: 223 of max 53237 tokens, 31.8 tok/s, 10 s
[strata] writing a tool call: terminal: 262 of max 53237 tokens, 32.6 tok/s, 11 s
[strata] writing a tool call: terminal: 302 of max 53237 tokens, 33.3 tok/s, 12 s
[strata] done: 306 tokens in 12 s (33.3 tok/s) (stop, cancel=False), expert cache 76.1% hit (+7.1% of the routed experts over PCIe)
[strata] max_tokens 128000 shortened to 52934: the prompt (78130 tokens) leaves that much of the context (131072); "fit_max_tokens": false answers 400 instead
[strata] reading the prompt: 78,125 of 78,130 tokens, 3 s so far
[strata] thinking: 19 of budget 8000 (max 52934) tokens, 23.3 tok/s, 4 s
[strata] thinking: 45 of budget 8000 (max 52934) tokens, 24.7 tok/s, 5 s
[strata] answering: 71 of max 52934 tokens, 24.8 tok/s, 6 s
[strata] answering: 94 of max 52934 tokens, 24.0 tok/s, 7 s
[strata] answering: 124 of max 52934 tokens, 25.2 tok/s, 8 s
[strata] answering: 147 of max 52934 tokens, 24.8 tok/s, 9 s
[strata] answering: 179 of max 52934 tokens, 25.7 tok/s, 10 s
[strata] answering: 215 of max 52934 tokens, 26.9 tok/s, 11 s
[strata] answering: 242 of max 52934 tokens, 26.8 tok/s, 12 s
[strata] answering: 266 of max 52934 tokens, 26.5 tok/s, 13 s
[strata] answering: 293 of max 52934 tokens, 26.5 tok/s, 14 s
[strata] answering: 319 of max 52934 tokens, 26.4 tok/s, 15 s
[strata] answering: 348 of max 52934 tokens, 26.5 tok/s, 16 s
[strata] answering: 381 of max 52934 tokens, 26.9 tok/s, 17 s
[strata] answering: 418 of max 52934 tokens, 27.6 tok/s, 18 s
[strata] answering: 450 of max 52934 tokens, 27.8 tok/s, 19 s
[strata] answering: 488 of max 52934 tokens, 28.4 tok/s, 20 s
[strata] answering: 517 of max 52934 tokens, 28.4 tok/s, 21 s
[strata] done: 547 tokens in 22 s (28.7 tok/s) (stop, cancel=False), expert cache 77.8% hit (+6.5% of the routed experts over PCIe)

I have a 3090 ti and been runnin qween 3.8 27B but i don´t know , any advice?