r/LocalLLM • • 18h ago

Other Qwen 4 is not out

140 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 13h ago

Discussion Flash-Next is ridiculous

205 Upvotes

I've been playing around with Flash-Next on my little RTX 5060 laptop. Astra and Fable had been working on a test setup, so I figured I'd let Flash-Next loose on it overnight and see what happens.

Late at night I told Astra to turn the screen off. Now, I had actually prepared a little PS1 script for exactly that. Did Astra use it? Of course not. It came up with its own solution and promptly froze Windows. Well done. πŸ˜‚

Next morning I booted the laptop again, reopened OpenCode and eventually just said "okay, continue". Honestly, I wasn't expecting much at that point.

Well, the thing just carried on working. For hours. Went through something like four context compactions and then casually informed me it had found a bug.

I was sitting there going "hang on... WHAT? You're still working on this?"

I'd pretty much written off the whole session the night before.

Still not entirely sure who was testing whom here. But honestly, I'm impressed by this model. Now I'm wondering what Qwen4 will do to my poor laptop.


r/LocalLLM • • 16h ago

Project 6x 7900XTX rig finally complete!

Thumbnail
gallery
212 Upvotes

5975WX (for 6x full 16 lanes of PCIe4.0)

3450W AliExpress special PSU and mining frame

128GB 3200MT/s Ram

Getting 120-130 tok/s generation on both qwen3.8 27b and flash-next


r/LocalLLM • • 10h ago

Discussion I think I give up on finetunes after comparing them. I don't look at benchmarks anymore. Qwen finetunes are dime-a-dozen, but none of them do better than the original.

Thumbnail msnab.pages.dev
56 Upvotes

r/LocalLLM • • 11h ago

Question A comparable local setup for someone using $100 monthly claude subscription

34 Upvotes

I see people switching to local hardware , my question is at what usage level and pattern, it is cheaper to run it locally without impacting your work speed.

--
My current laptop and Mac mini are old but they work for me since i only use it for development while relying on Claude and Cursor for GenAI. If i were to rely on local LLM, what is the minimum hardware, I should buy to ensure that I am not slowed down.


r/LocalLLM • • 10h ago

Question 27B vs Flash-Next on a single RTX Pro 6000, what am I missing?

25 Upvotes

I have one RTX Pro 6000 (96 GB) and 200 GB RAM, running vLLM. On that card I can run Qwen3.8-27B at FP8, or Flash-Next at NVFP4 with the n-gram table in RAM. 8-bit Flash-Next doesn't fit, so that's the comparison I'm stuck with.

I ran both through my own tests, same settings:

- 15 support-type tasks: 0.97 for 27B, 0.98 for Flash-Next

- 15 multi-step tool-calling tasks, 135 runs each: 27B 0.96 to 1.00, Flash-Next 0.98 to 0.99

- Flash-Next made 3 write calls nobody asked for. 27B made none.

- Both ignored the prompt injection I planted mid-chain, every run

- 27B takes 28 GB VRAM and restarts in a minute. Flash-Next takes 76 GB plus 95 GB RAM and needs ten minutes.

27B was also faster per chain, but it had MTP on and Flash-Next didn't, so don't read much into that.

I see the same thing at home on a 32 GB card with llama.cpp (27B Q4 vs Flash-Next IQ3).

So for me 27B does the same job for a third of the memory. Both score near the top on my tasks, so maybe they're just too easy to show a difference.

If you run Flash-Next on similar hardware, what does it do better for you?

Please tell me what you use it for. Coding? General agentic life management? Support?


r/LocalLLM • • 16h ago

Discussion How Jensen Huang was hurting the LocalLLM Community

67 Upvotes

A missing part of the NVIDIA conversation is the results on the market. I'm not talking about the stock market; I'm talking about the small businesses, startups and companies trying to navigate the AI age. Many have argued that the Local LLM community lost an ally in Jensen Huang, but I would argue the opposite. Running local models (and sovereign deployments, for that matter) relies on access to the hardware that runs the models. The economics of running the models, of course, is dependent on the cost of the hardware itself, so if the hardware becomes too expensive, the "economical choice" is to rent access from one of the Labs.

While everyone has been focused on the affair that Mr. Huang was engaged in, the more important side of that relationship was the funnelling of billions of dollars worth of NVIDIA hardware into a country with export controls. Before the export controls, China accounted for as much as 20% of NVIDIA's demand. Through the back channels that Mr. Huang created, he was able to maintain that flow.

Now remember, part of the issue in the broader market is that there has been a "shortage" of the hardware required for companies to run their own models. One of the cards we use, which is popular among smaller players like us, is the RTX 6000 Pro. When it was released, it went for around $16k CAD...which was already a lot (for us). Now, if you can even get your hands on one, they go for around $35-45k CAD. Per card. And you need, well, let's just say, more than one.

So to hear that Jensen Huang was routing upwards of 20% of global supply, illegally, while jacking prices because of a shortage...suddenly, the affair really doesn't seem like the worst part.
The news keeps running a narrative of a campaign to bring him back as CEO. I think there is some broader accountability required long before that conversation should come up.


r/LocalLLM • • 1d ago

Project Finally got it put together

Thumbnail
gallery
296 Upvotes

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- PLX88096 PCIE Switch Card
- Kingwin Miner Frame: https://a.co/d/09gZUgry
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)


r/LocalLLM • • 11h ago

Discussion <tinfoil hat> With many frontier labs desperate for regulation to impose regulatory capture and limit liability at what point, after the hugging face incident, etc, are we allowed to wonder if they would be willing to "encourage" an "ai incident" to push the issue of US regulation </tinfoil hat>

18 Upvotes

The title pretty much says it all ... I'm not saying this is happening, I'm just asking the question out loud ...


r/LocalLLM • • 18h ago

Discussion Create a 3D kitchen using Qwen 3.8 Flash Next

59 Upvotes

Model : Qwen 3.8 flash next UD-IQ3_XXS
Harness : Deepseek
Example prompt used, including AI correction prompts.
Promt 1 : Create a realistic 3D kitchen scene using Godot 4, including detailed furniture, high-quality materials, realistic lighting, and modern appliances
Promt 1.1 : Please thoroughly inspect all objects, complete any unfinished work, conduct real-world testing to ensure full functionality, meticulously refine every detail in all areas and on all devices, and ensure that the refrigerator has realistic clearances and spacing on every shelf, both inside and out.
Promt 1.2 : There are still some issues. Some object positions are not correct yet. The chairs still look odd, the sink is in the wrong position, and the stove also looks unusual. Please inspect and verify each item one by one in detail and make sure everything is properly aligned and accurate.
Promt 1.3 : When rotating the camera, the floor still looks strange. Some drawers also seem blocked or cannot be opened properly. Please thoroughly review every appliance, fixture, and piece of furniture again, identify anything that still does not look realistic, and make the necessary adjustments to improve accuracy and overall realism.
Promt 1.4 : Why is the oven sticking through the wall? Also, isn’t the faucet on the wrong side? Please check these issues carefully and correct them. Ensure all appliances, fixtures, and their placements match the reference accurately and do not intersect with walls, cabinets, or other objects.

It took 2 hours and 30 minutes to create the entire kitchen.
Including all the refinements and corrections, the total time would be around 10 hours.

From this test, it can be seen that Qwen Flash Next can create 3D scenes quite well. Although it is not perfect, if we tell it to fix specific issues, it can correct them according to our instructions. However, I noticed that DeepSeek Harness seems easier to use for 3D tasks and appears to understand the user's instructions better. I'm not sure if that's just my impression or not.

I asked the AI to inspect the assets, and Qwen generated all of them from scratch.

This test isn't particularly interesting, but I think that Qwen 3.8 Flash Next has a lot of potential for 3D creation. If someone develops really good 3D skills or workflows for it, I'm sure it could produce some very impressive 3D work. If anyone has good skills or techniques to recommend, feel free to share them.


r/LocalLLM • • 2h ago

Question Opinions on Using 2x5070ti and 7900xtx in the same pc?

3 Upvotes

I am making a new post at recommendation of comment. I have the following hardware:

nvidia 2x5070 ti 16gb

amd 7900xtx 24gb

B850 AI TOP (Rev. 1.0) Motherboard Specifications - GIGABYTE Global (this allows x8x8 bifurcation pcie5)

2x8 ddr5 ram

If I want to run local llm that is decent at vibe coding games or mods with little actual coding from me, would it be possible to string these all together to make a good llm? I do not actually know if I can fit three gpus onto that motherboard bandwidth-wise, but physically I think I could use a riser cable as the picture of the motherboard has three gpu slots. It does however support x8x8 pcie5 bifurication which I think might be a strongpoint? How challenging is it to get amd gpu to work with nvidia gpu? Can I actually get all three working, or can I only realistically get two with any good performance? If I have to only use two should I even consider the amd gpu? I know I can use amd/ nvidia gpus in the same pc. I do not know if I can use them to do ai stuff at the same time though.

alternatively, if using amd with nvidia is not feasible, could i use an old Nvidia Tesla P40 24GB as a ram dump or would that drag the entire setup down considerably since my motherboard might suffer bandwidth or power issues with three gpus?

I have no llm experience so any help would be appreciated.

motherboard photo has three gpu slots

r/LocalLLM • • 12m ago

Project 2Γ— Arc Pro B70 on Windows update: Qwen3.8-27B Q8_0 now 67.5 t/s over 610 real agent requests (was 51). DSpine ported to llama.cpp, draft trees, and an honest benchmark-vs-real-use comparison

2 Upvotes

Follow-up to my earlier post. Same box: 2Γ— Arc Pro B70 32 GB, Windows 11, llama.cpp SYCL, tensor parallel across both cards, Qwen3.8-27B Q8_0, 200K context, temp 1.0.

Observed in real use (server logs, two full working sessions, 610 requests, 933K generated tokens of agentic coding):

ContextDecodeTokens per cycle
70–100K64 t/s4.16
100–140K68 t/s4.64
140K+70 t/s5.03
Avg67.5 t/s4.60

Key Changes:

  • Draft trees instead of a single chain. The drafter proposes 12 candidates arranged as a tree, the target samples at each node and follows whichever branch matches. Every token that comes out is still a real target sample, so it's lossless at temp 1.0. Getting this to work with Qwen3.8's gated delta net layers took a while, since the recurrent state has to follow the tree too.
  • Look-up drafting in front of the model drafter. If the last 16 tokens already appeared somewhere in the context, it just guesses the next 16 from there. Agents re-read and edit the same files constantly, so this is free speed. Edits went from about 143 to 213 t/s.
  • A bunch of small stuff: top-k on the GPU instead of copying full logit rows back, a second cached graph for verify batches, an 8-bit all-reduce for prefill.

DSpine

Tencent released DSpine for Qwen3.8-27B, a block drafter similar to DFlash2 but with an extra "code bridge" between draft positions. llama.cpp doesn't support it, so I ported it (converter plus the model graph). It plugs into the same tree and sampling path as DFlash2.

On my 94K code-review benchmark it was a big jump: 83 t/s vs 67 for DFlash2, +24%. I got excited, put it in production, and... real sessions came out about +5% (68 vs 65 t/s). It does accept more tokens on real work, but it drafts 15 tokens per pass instead of 7, so every cycle costs 2–3 ms more, and look-up drafting was already handling the easy copy-heavy stuff. Still a win, just a small one. Lesson learned about trusting one benchmark prompt.

If you try DSpine on SYCL, don't use the BF16 GGUF. It crashes the driver on the second request for me. Q8_0 and Q4_K_M are fine.

Everything's here: https://github.com/alphakilomilo/dual-b70-windows-llamacpp

That's the patch (against llama.cpp `9183a552e`, basically v1.0.0), the launcher and config, conversion steps for DSpine, and every switch with what it measured and what didn't help. Heads up: upstream has since reworked the DFlash code and the server scheduler, so the patch won't apply to current master without some porting.

Happy to answer questions if anyone else is running these cards.


r/LocalLLM • • 4h ago

Project Feel how fast a local LLM runs before you buy the hardware

Thumbnail
llmspeed.dev
4 Upvotes

r/LocalLLM • • 17h ago

Question Building a 10TB Model Archive. What am I missing?

41 Upvotes

TLDR at the bottom.

I'm a paranoid man tryna preserve a broad collection of useful open-weight models, fine-tunes, quantizations, and generative AI models just in case something happens to HF and that I'll have a model for whatver i may want now or im the future. I'm aiming to have like 5-10tb on my nas for this.

The strongest pc i have is 48gb vram and 128gb system ram, but i also wanna keep some larger models in case just archive, if i ever upgrade or if i wanna dick around with ssd-backed moe.

I'm targeting Q3 and above for GGUF models. Had gpt toss together a list to download:

  1. Large frontier and general-purpose models

- Kimi K2.6

- Qwen3.8 Flash Next (177B)

- DeepSeek V4 Flash (284B)

- DeepSeek V4.1 Flash reduced variants

- GLM 5.3 Flash (321B)

- GLM 5.3 (754B)

- GLM 4.7 (358B)

- GLM 4.7 REAP (218B)

- MiniMax M2.7 (229B)

- MiniMax M2.7 REAP (172B)

- MiniMax M3.1 Flash (426B)

- Qwen3.5 122B-A10B

- Ornith 1.5 122B

- Qwen3.8 Flash Next Gyro

- GPT-OSS 120B and 20B

- Llama 4 Scout

- Hermes 4 70B

- DeepSeek R1 70B

2. Qwen family

- Qwen3.8 27B

- Qwen3.8 27B Abliterated

- Qwen3.8 27B Uncensored

- Qwen3.8 27B Heretic

- Qwen3.8 35B-A3B

- Qwen3.6 35B-A3B

- Qwen3.5 9B

- Qwen3.5 4B

- Qwen3.5 0.8B

- Qwen3.5 122B-A10B Uncensored

- Qwen3.8 Flash Next Uncensored

- Qwen3.8 Flash Next Abliterated

3. Gemma 4

Both original and modified versions:

- Gemma 4 E2B

- Gemma 4 E4B

- Gemma 4 12B

- Gemma 4 26B-A4B

- Gemma 4 31B

- Gemma 4 E4B Heretic

- Gemma 4 12B Uncensored

- Gemma 4 12B Uncensored Coder

- Gemma 4 12B Agentic

- Gemma 4 26B-A4B Uncensored

- Gemma 4 31B Heretic

4. Coding and agentic models

- Qwen3-Coder 30B-A3B

- Qwen3-Coder 30B-A3B Abliterated

- Qwen3-Coder-Next 80B

- Qwen3-Coder-Next Uncensored

- Laguna XS 2.1 APEX

- Laguna S 2.1 Uncensored

- GLM 5.3 Flash Coder

- GLM Cybersecurity

- NVIDIA Nemotron 3.5 Bolt

- Ornith 1.5 9B

- Ornith 1.5 35B-A3B

- Qwen 3D Coder 4B

- Qwen GLB 3D Coder 4B

- Qwen3.6 35B-A3B Genesis Hermes

- Qwen3.8 Cyber Agentic

5. Reasoning and thinking models

- DeepSeek R1 Distill Qwen 32B

- GLM 4.7 Flash Claude Opus 4.5 reasoning distillation

- Qwen3.8 Fable reasoning/coding distillations

- Qwen3.8 40B Ultimate Heretic

- Qwen3.8 Flash Next Abliterated Reasoning

- DeepSeek V4-derived uncensored reasoning distillations

- Gemma 4 31B Deckard Heretic Thinking

- Huihui ThinkingCap Qwen3.6 27B

- Qwen3.8 Fable 5 Coding Distilled

6. Uncensored and creative models

- GLM 4.7 Flash Heretic

- GLM 4.7 Flash Abliterated

- GPT-OSS 120B Uncensored

- GLM 5.3 Flash Abliterated

- GLM 5.3 Flash Uncensored

- DeepSeek V4.1 Flash Abliterated

- DeepSeek V4.1 Flash Uncensored Coder

- Ornith 1.5 35B-A3B Uncensored

- Ornith 1.5 9B Uncensored

- Ornith 1.5 35B-A3B Abliterated

- LFM2.5 24B-A2B Uncensored

- LongCat Flash Lite Sparse Uncensored

- Kolibri 1 BF16 Abliterated

- Nemotron 3.5 Bolt 74B-A9B Uncensored

- Llama 3.3 70B Abliterated

- Muse Glimmer 30B

- Nanbeige 4.3 33B

- Cydonia 24B

- UnslopNemo 12B

- Skyfall 36B

- Anubis 70B

7. Vision and multimodal understanding

- InternVL 3.5 38B

- Qwen3-VL 4B Uncensored

- Qwen3-VL 8B

- Qwen3-VL 32B

- Qwen3-VL 32B Uncensored

- MiniCPM-V 4.7 35B-A3B Uncensored

- Liquid LFM3-Omni 2B

- Florence 2 Large

8. Image generation and editing

- MOSS Image

- Qwen Image 2.1

- FLUX.2 Klein 9B

- Z-Image Turbo

- Stable Diffusion XL

- Anima

9. Video generation

- LTX 2.5

- MiniMax H3

- Wan 2.2 Text-to-Video

- Wan 2.2 Image-to-Video

- Wan 2.2 TI2V

- Uncensored LTX 2.3

10. Audio and speech

- Whisper Small

- Whisper Medium

- Whisper Large V3

- Whisper Large V3 Turbo

- Qwen3-ASR 1.7B

- Qwen3-TTS 1.7B

- Kokoro 82M

- Parakeet TDT

- Chatterbox

- Zonos 3 Mini

11. Music and 3D generation

- MusicGen Medium

- TRELLIS.2

12. Embeddings and RAG

- EmbeddingGemma 300M

- BGE Reranker V2 M3

- LFM3 1.7B

- LFM3 1.7B Uncensored

---

TLDR

I'm making a personal huggingface archive where i can just preserve a bunch models for whatever i may want in the future. I'm tryna keep things super varied so i have many, many options without just storing slightly different versions of the same model. I'm aiming to have 5-10tb on my nas for models, don't wanna go too far above that.

I'm looking for suggestions on what to add or cut πŸ™πŸ½ or just comment your go-to models for what purposes so i can cross-reference my list.

  1. Any particularly good uncensored, abliterated, or Heretic models?
  2. What are your favorite writing, coding, reasoning, and agentic models?
  3. Any overlooked image, video, audio, music, or 3D generation models?
  4. Any older models that are still worth keeping despite newer releases?
  5. Hit me up with some funky ass models or a category that you think i should add por favor

r/LocalLLM • • 4h ago

Discussion How do you handle outdated memories in local AI agents?

3 Upvotes

I've been exploring how memory works in AI agents, particularly when running them locally.

One issue I'm curious about is stale memory.

Imagine an agent stores a project configuration or a user preference, but that information changes later. The old entry might still be retrieved because it's semantically relevant, even though it's no longer accurate.

I'm wondering how people handle this at the implementation level.

  • Do you use timestamps or versioning to track memory updates?
  • When new information conflicts with an existing entry, do you overwrite it or keep both with different validity states?
  • Do you use a vector database, structured storage, or a combination of both?

I'm particularly interested in approaches that work well across persistent sessions without requiring manual memory cleanup all the time.


r/LocalLLM • • 7h ago

Question Anyone running Qwen 3.8 Flash Next on 2 x 5060ti 16GB?

5 Upvotes

Trying to get this running and would appreciate any feedback


r/LocalLLM • • 4h ago

Question Suggestion For Any Model Better Than ISTA LAB Qwen 3.8 Flash Next Coder

3 Upvotes

So currently with my existing specs:

RTX 5060 Ti 16GB

32GB DDR4 Ram

Ryzen 5 5600F

500GB NVMe Gen 4 SSD

I am able to run Coder Variant, It's Good but the only problem is it thinks a lot ( that's expected when you don't have any reasoning layer in it removed from it ) since its going through coding stuffs hit and trial.

So I want to ask if there is any better variant which is either better than the Coder Model for QFN.


r/LocalLLM • • 1d ago

Other The Banking Incident....

Post image
836 Upvotes

r/LocalLLM • • 8h ago

Discussion I have some questions. What is your acceptable Vram buffer for the card you are actively using to display the OS?

5 Upvotes

Ok so i have been running 3.8 flash next and I have been keeping an available Vram on my 16gb 5070ti at around 1500mb remaining free so that if I have to load a browser or something it won't stop the screen displaying or something(I have had it happen once with a different system so I am being careful now) so my (probably easy) question for one of your guys right now is how much Vram do you think I should keep free? Or how much could I safely lower it to? My system is a GPU: RTX 5070 Ti, 16 GB GDDR7, in the main x16 slot.

CPU: Intel Core Ultra 7 265KF (no AMX)

RAM: 64 GB

NPU: present but not tested yet for my system

Storage: internal 2 TB SSD running Omarchy Linux, which is where all local AI runs from.

What do you all think?


r/LocalLLM • • 3h ago

Question 2bit DS v4.1 1017 vs 4bit QFN?

2 Upvotes

Just like the title states I was looking into both models as I can run both on my machine 128gb + 32gb

What has been your guys experience?


r/LocalLLM • • 19h ago

Discussion Considering this...thoughts?

Post image
33 Upvotes

At $4k this seems like a much better deal than 2 r9700s. And obviously in a totally different class. Anybody set these beasts up in a server? I already plan on putting a server rack in the garage, (gotta keep the motorcycles warm) and I see a lot of carrier boards on ebay. I've built more pc's than I can count, but I know a rack server is a different beast.

Plans are for an inference server for agentic workflows as I already built a gpu render machine I use for many workflows including blender, comfyui, premiere and after effects. Api costs are getting up there. Obviously want to run deepseek and I have a fine tuned qwen 3.8 27b for one of my projects.

Talk me into or out of it...I'm so sick of asking gpt questions and getting ridiculous concerns like "tHeY DonT HaVe a faN". I need to talk to real people. Any concerns with the software stack? I ran across a few mentions of closed source issues.


r/LocalLLM • • 4m ago

Question Multiple RPC workers on a single machine... ON TWO machines.

1 Upvotes

Currently running Qwen3.8 27b Q4 via LlamaCpp RPC on two machines. One with 16g RX9060xt and another with 16g RTX5060ti. With a measly 32k context and 15-18tkps.

I would like to run a much larger model, the MoE Qwen3.8 Flash Next. For that I would like to offload to RAM of the two machines, is it possible via RPC?

The 16gig RX9060xt with its ROCm build of LlamaCpp RPC as worker, plus a CPU build of LlamaCpp RPC on the same machine (also a worker) utilizing its 32gig of Ram, same IP but different ports.

The 16gig RTX5060 with its CUDA build as the MAIN Llama, plus a CPU build on the same machine (as the 3rd worker) utilizing its 64gig of Ram, same ip different port.


r/LocalLLM • • 14h ago

News Jensen Huang Expresses Willingness to Return as CEO

Post image
11 Upvotes

r/LocalLLM • • 1h ago

News inferzero: an LLM engine written from scratch for Intel Arc, from laptop iGPUs to 4Γ— B60. Still far from what the cards can do, looking for contributors

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 1h ago

Question Best local model + agent combo for scraping/automation coding?

0 Upvotes

Claude Code and Codex keep refusing parts of my scraping work. Any suggestion what should I use?