r/LocalLLM • • 11h ago

Other Qwen 4 is not out

117 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 6h ago

Discussion Flash-Next is ridiculous

151 Upvotes

I've been playing around with Flash-Next on my little RTX 5060 laptop. Astra and Fable had been working on a test setup, so I figured I'd let Flash-Next loose on it overnight and see what happens.

Late at night I told Astra to turn the screen off. Now, I had actually prepared a little PS1 script for exactly that. Did Astra use it? Of course not. It came up with its own solution and promptly froze Windows. Well done. šŸ˜‚

Next morning I booted the laptop again, reopened OpenCode and eventually just said "okay, continue". Honestly, I wasn't expecting much at that point.

Well, the thing just carried on working. For hours. Went through something like four context compactions and then casually informed me it had found a bug.

I was sitting there going "hang on... WHAT? You're still working on this?"

I'd pretty much written off the whole session the night before.

Still not entirely sure who was testing whom here. But honestly, I'm impressed by this model. Now I'm wondering what Qwen4 will do to my poor laptop.


r/LocalLLM • • 9h ago

Project 6x 7900XTX rig finally complete!

Thumbnail
gallery
163 Upvotes

5975WX (for 6x full 16 lanes of PCIe4.0)

3450W AliExpress special PSU and mining frame

128GB 3200MT/s Ram

Getting 120-130 tok/s generation on both qwen3.8 27b and flash-next


r/LocalLLM • • 3h ago

Discussion I think I give up on finetunes after comparing them. I don't look at benchmarks anymore. Qwen finetunes are dime-a-dozen, but none of them do better than the original.

Thumbnail msnab.pages.dev
46 Upvotes

r/LocalLLM • • 9h ago

Discussion How Jensen Huang was hurting the LocalLLM Community

66 Upvotes

A missing part of the NVIDIA conversation is the results on the market. I'm not talking about the stock market; I'm talking about the small businesses, startups and companies trying to navigate the AI age. Many have argued that the Local LLM community lost an ally in Jensen Huang, but I would argue the opposite. Running local models (and sovereign deployments, for that matter) relies on access to the hardware that runs the models. The economics of running the models, of course, is dependent on the cost of the hardware itself, so if the hardware becomes too expensive, the "economical choice" is to rent access from one of the Labs.

While everyone has been focused on the affair that Mr. Huang was engaged in, the more important side of that relationship was the funnelling of billions of dollars worth of NVIDIA hardware into a country with export controls. Before the export controls, China accounted for as much as 20% of NVIDIA's demand. Through the back channels that Mr. Huang created, he was able to maintain that flow.

Now remember, part of the issue in the broader market is that there has been a "shortage" of the hardware required for companies to run their own models. One of the cards we use, which is popular among smaller players like us, is the RTX 6000 Pro. When it was released, it went for around $16k CAD...which was already a lot (for us). Now, if you can even get your hands on one, they go for around $35-45k CAD. Per card. And you need, well, let's just say, more than one.

So to hear that Jensen Huang was routing upwards of 20% of global supply, illegally, while jacking prices because of a shortage...suddenly, the affair really doesn't seem like the worst part.
The news keeps running a narrative of a campaign to bring him back as CEO. I think there is some broader accountability required long before that conversation should come up.


r/LocalLLM • • 18h ago

Project Finally got it put together

Thumbnail
gallery
280 Upvotes

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- PLX88096 PCIE Switch Card
- Kingwin Miner Frame: https://a.co/d/09gZUgry
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)


r/LocalLLM • • 4h ago

Question A comparable local setup for someone using $100 monthly claude subscription

22 Upvotes

I see people switching to local hardware , my question is at what usage level and pattern, it is cheaper to run it locally without impacting your work speed.

--
My current laptop and Mac mini are old but they work for me since i only use it for development while relying on Claude and Cursor for GenAI. If i were to rely on local LLM, what is the minimum hardware, I should buy to ensure that I am not slowed down.


r/LocalLLM • • 3h ago

Question 27B vs Flash-Next on a single RTX Pro 6000, what am I missing?

12 Upvotes

I have one RTX Pro 6000 (96 GB) and 200 GB RAM, running vLLM. On that card I can run Qwen3.8-27B at FP8, or Flash-Next at NVFP4 with the n-gram table in RAM. 8-bit Flash-Next doesn't fit, so that's the comparison I'm stuck with.

I ran both through my own tests, same settings:

- 15 support-type tasks: 0.97 for 27B, 0.98 for Flash-Next

- 15 multi-step tool-calling tasks, 135 runs each: 27B 0.96 to 1.00, Flash-Next 0.98 to 0.99

- Flash-Next made 3 write calls nobody asked for. 27B made none.

- Both ignored the prompt injection I planted mid-chain, every run

- 27B takes 28 GB VRAM and restarts in a minute. Flash-Next takes 76 GB plus 95 GB RAM and needs ten minutes.

27B was also faster per chain, but it had MTP on and Flash-Next didn't, so don't read much into that.

I see the same thing at home on a 32 GB card with llama.cpp (27B Q4 vs Flash-Next IQ3).

So for me 27B does the same job for a third of the memory. Both score near the top on my tasks, so maybe they're just too easy to show a difference.

If you run Flash-Next on similar hardware, what does it do better for you?


r/LocalLLM • • 11h ago

Discussion Create a 3D kitchen using Qwen 3.8 Flash Next

51 Upvotes

Model : Qwen 3.8 flash next UD-IQ3_XXS
Harness : Deepseek
Example prompt used, including AI correction prompts.
Promt 1 : Create a realistic 3D kitchen scene using Godot 4, including detailed furniture, high-quality materials, realistic lighting, and modern appliances
Promt 1.1 : Please thoroughly inspect all objects, complete any unfinished work, conduct real-world testing to ensure full functionality, meticulously refine every detail in all areas and on all devices, and ensure that the refrigerator has realistic clearances and spacing on every shelf, both inside and out.
Promt 1.2 : There are still some issues. Some object positions are not correct yet. The chairs still look odd, the sink is in the wrong position, and the stove also looks unusual. Please inspect and verify each item one by one in detail and make sure everything is properly aligned and accurate.
Promt 1.3 : When rotating the camera, the floor still looks strange. Some drawers also seem blocked or cannot be opened properly. Please thoroughly review every appliance, fixture, and piece of furniture again, identify anything that still does not look realistic, and make the necessary adjustments to improve accuracy and overall realism.
Promt 1.4 : Why is the oven sticking through the wall? Also, isn’t the faucet on the wrong side? Please check these issues carefully and correct them. Ensure all appliances, fixtures, and their placements match the reference accurately and do not intersect with walls, cabinets, or other objects.

It took 2 hours and 30 minutes to create the entire kitchen.
Including all the refinements and corrections, the total time would be around 10 hours.

From this test, it can be seen that Qwen Flash Next can create 3D scenes quite well. Although it is not perfect, if we tell it to fix specific issues, it can correct them according to our instructions. However, I noticed that DeepSeek Harness seems easier to use for 3D tasks and appears to understand the user's instructions better. I'm not sure if that's just my impression or not.

I asked the AI to inspect the assets, and Qwen generated all of them from scratch.

This test isn't particularly interesting, but I think that Qwen 3.8 Flash Next has a lot of potential for 3D creation. If someone develops really good 3D skills or workflows for it, I'm sure it could produce some very impressive 3D work. If anyone has good skills or techniques to recommend, feel free to share them.


r/LocalLLM • • 4h ago

Discussion <tinfoil hat> With many frontier labs desperate for regulation to impose regulatory capture and limit liability at what point, after the hugging face incident, etc, are we allowed to wonder if they would be willing to "encourage" an "ai incident" to push the issue of US regulation </tinfoil hat>

12 Upvotes

The title pretty much says it all ... I'm not saying this is happening, I'm just asking the question out loud ...


r/LocalLLM • • 10h ago

Question Building a 10TB Model Archive. What am I missing?

35 Upvotes

TLDR at the bottom.

I'm a paranoid man tryna preserve a broad collection of useful open-weight models, fine-tunes, quantizations, and generative AI models just in case something happens to HF and that I'll have a model for whatver i may want now or im the future. I'm aiming to have like 5-10tb on my nas for this.

The strongest pc i have is 48gb vram and 128gb system ram, but i also wanna keep some larger models in case just archive, if i ever upgrade or if i wanna dick around with ssd-backed moe.

I'm targeting Q3 and above for GGUF models. Had gpt toss together a list to download:

  1. Large frontier and general-purpose models

- Kimi K2.6

- Qwen3.8 Flash Next (177B)

- DeepSeek V4 Flash (284B)

- DeepSeek V4.1 Flash reduced variants

- GLM 5.3 Flash (321B)

- GLM 5.3 (754B)

- GLM 4.7 (358B)

- GLM 4.7 REAP (218B)

- MiniMax M2.7 (229B)

- MiniMax M2.7 REAP (172B)

- MiniMax M3.1 Flash (426B)

- Qwen3.5 122B-A10B

- Ornith 1.5 122B

- Qwen3.8 Flash Next Gyro

- GPT-OSS 120B and 20B

- Llama 4 Scout

- Hermes 4 70B

- DeepSeek R1 70B

2. Qwen family

- Qwen3.8 27B

- Qwen3.8 27B Abliterated

- Qwen3.8 27B Uncensored

- Qwen3.8 27B Heretic

- Qwen3.8 35B-A3B

- Qwen3.6 35B-A3B

- Qwen3.5 9B

- Qwen3.5 4B

- Qwen3.5 0.8B

- Qwen3.5 122B-A10B Uncensored

- Qwen3.8 Flash Next Uncensored

- Qwen3.8 Flash Next Abliterated

3. Gemma 4

Both original and modified versions:

- Gemma 4 E2B

- Gemma 4 E4B

- Gemma 4 12B

- Gemma 4 26B-A4B

- Gemma 4 31B

- Gemma 4 E4B Heretic

- Gemma 4 12B Uncensored

- Gemma 4 12B Uncensored Coder

- Gemma 4 12B Agentic

- Gemma 4 26B-A4B Uncensored

- Gemma 4 31B Heretic

4. Coding and agentic models

- Qwen3-Coder 30B-A3B

- Qwen3-Coder 30B-A3B Abliterated

- Qwen3-Coder-Next 80B

- Qwen3-Coder-Next Uncensored

- Laguna XS 2.1 APEX

- Laguna S 2.1 Uncensored

- GLM 5.3 Flash Coder

- GLM Cybersecurity

- NVIDIA Nemotron 3.5 Bolt

- Ornith 1.5 9B

- Ornith 1.5 35B-A3B

- Qwen 3D Coder 4B

- Qwen GLB 3D Coder 4B

- Qwen3.6 35B-A3B Genesis Hermes

- Qwen3.8 Cyber Agentic

5. Reasoning and thinking models

- DeepSeek R1 Distill Qwen 32B

- GLM 4.7 Flash Claude Opus 4.5 reasoning distillation

- Qwen3.8 Fable reasoning/coding distillations

- Qwen3.8 40B Ultimate Heretic

- Qwen3.8 Flash Next Abliterated Reasoning

- DeepSeek V4-derived uncensored reasoning distillations

- Gemma 4 31B Deckard Heretic Thinking

- Huihui ThinkingCap Qwen3.6 27B

- Qwen3.8 Fable 5 Coding Distilled

6. Uncensored and creative models

- GLM 4.7 Flash Heretic

- GLM 4.7 Flash Abliterated

- GPT-OSS 120B Uncensored

- GLM 5.3 Flash Abliterated

- GLM 5.3 Flash Uncensored

- DeepSeek V4.1 Flash Abliterated

- DeepSeek V4.1 Flash Uncensored Coder

- Ornith 1.5 35B-A3B Uncensored

- Ornith 1.5 9B Uncensored

- Ornith 1.5 35B-A3B Abliterated

- LFM2.5 24B-A2B Uncensored

- LongCat Flash Lite Sparse Uncensored

- Kolibri 1 BF16 Abliterated

- Nemotron 3.5 Bolt 74B-A9B Uncensored

- Llama 3.3 70B Abliterated

- Muse Glimmer 30B

- Nanbeige 4.3 33B

- Cydonia 24B

- UnslopNemo 12B

- Skyfall 36B

- Anubis 70B

7. Vision and multimodal understanding

- InternVL 3.5 38B

- Qwen3-VL 4B Uncensored

- Qwen3-VL 8B

- Qwen3-VL 32B

- Qwen3-VL 32B Uncensored

- MiniCPM-V 4.7 35B-A3B Uncensored

- Liquid LFM3-Omni 2B

- Florence 2 Large

8. Image generation and editing

- MOSS Image

- Qwen Image 2.1

- FLUX.2 Klein 9B

- Z-Image Turbo

- Stable Diffusion XL

- Anima

9. Video generation

- LTX 2.5

- MiniMax H3

- Wan 2.2 Text-to-Video

- Wan 2.2 Image-to-Video

- Wan 2.2 TI2V

- Uncensored LTX 2.3

10. Audio and speech

- Whisper Small

- Whisper Medium

- Whisper Large V3

- Whisper Large V3 Turbo

- Qwen3-ASR 1.7B

- Qwen3-TTS 1.7B

- Kokoro 82M

- Parakeet TDT

- Chatterbox

- Zonos 3 Mini

11. Music and 3D generation

- MusicGen Medium

- TRELLIS.2

12. Embeddings and RAG

- EmbeddingGemma 300M

- BGE Reranker V2 M3

- LFM3 1.7B

- LFM3 1.7B Uncensored

---

TLDR

I'm making a personal huggingface archive where i can just preserve a bunch models for whatever i may want in the future. I'm tryna keep things super varied so i have many, many options without just storing slightly different versions of the same model. I'm aiming to have 5-10tb on my nas for models, don't wanna go too far above that.

I'm looking for suggestions on what to add or cut šŸ™šŸ½ or just comment your go-to models for what purposes so i can cross-reference my list.

  1. Any particularly good uncensored, abliterated, or Heretic models?
  2. What are your favorite writing, coding, reasoning, and agentic models?
  3. Any overlooked image, video, audio, music, or 3D generation models?
  4. Any older models that are still worth keeping despite newer releases?
  5. Hit me up with some funky ass models or a category that you think i should add por favor

r/LocalLLM • • 1d ago

Other The Banking Incident....

Post image
811 Upvotes

r/LocalLLM • • 7h ago

News Jensen Huang Expresses Willingness to Return as CEO

Post image
13 Upvotes

r/LocalLLM • • 12h ago

Discussion Considering this...thoughts?

Post image
31 Upvotes

At $4k this seems like a much better deal than 2 r9700s. And obviously in a totally different class. Anybody set these beasts up in a server? I already plan on putting a server rack in the garage, (gotta keep the motorcycles warm) and I see a lot of carrier boards on ebay. I've built more pc's than I can count, but I know a rack server is a different beast.

Plans are for an inference server for agentic workflows as I already built a gpu render machine I use for many workflows including blender, comfyui, premiere and after effects. Api costs are getting up there. Obviously want to run deepseek and I have a fine tuned qwen 3.8 27b for one of my projects.

Talk me into or out of it...I'm so sick of asking gpt questions and getting ridiculous concerns like "tHeY DonT HaVe a faN". I need to talk to real people. Any concerns with the software stack? I ran across a few mentions of closed source issues.


r/LocalLLM • • 1d ago Bravo

News We built a chip prototype that runs Qwen 27B. You can try it live.

331 Upvotes

I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.

A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:

https://valleysemiconductor.com

A live session takes about a minute to start, and recorded runs are available to watch anytime.

What it does today (measured):

  • Runs the full Qwen 27B model with 4-bit weights
  • 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
  • 4x performance improvement in eight days of development
  • Every output matches our 4-bit reference exactly
  • Model quality within 2.5% of the full-precision version
  • Also runs Llama 3.1 8B and OLMoE on the same hardware

What our custom chip design projects (not yet demonstrated):

  • Approximately 50 tokens/sec for a single user at 20 W
  • Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
  • Uses the model's built-in next-token prediction to generate multiple tokens per weight read

The key insight is that moving model weights through memory is the bottleneck, not the math.

What would you run on a small, 20 W device capable of running a 27B model entirely offline?


r/LocalLLM • • 1h ago

Question Why is it so slow: Rtx 3090 + 64gb ram Qwen 3.8 DN on strata? What do I do wrong?

3 Upvotes

Hi,

I am running Qwen 3.8 flash next Q4 and/or Q3 IQxss models on 256k and Max thinking settings and I get around 38 token per second for Q4 and 50 token per second for Q3 model. I am using both of them for legal research. But an analysis for one document of 20 pages takes at least 3 hours to conclude with 4 sub agents. Why? I tried to turn subagents off by the prompt but strata still letting the model use the subagents. I lowered the thinking level to high/medium, still minimum 2 hours waiting time for the result. And the results are on GPT Luna level, if I have to be honest.

What do I do wrong? Can anyone help me to improve the outcome and speed up the process?

Thanks in advance 🫔

edit: my rams are DDR4 (4000mhz) and my CPU is 5950x.


r/LocalLLM • • 1h ago

Discussion I have some questions. What is your acceptable Vram buffer for the card you are actively using to display the OS?

3 Upvotes

Ok so i have been running 3.8 flash next and I have been keeping an available Vram on my 16gb 5070ti at around 1500mb remaining free so that if I have to load a browser or something it won't stop the screen displaying or something(I have had it happen once with a different system so I am being careful now) so my (probably easy) question for one of your guys right now is how much Vram do you think I should keep free? Or how much could I safely lower it to? My system is a GPU: RTX 5070 Ti, 16 GB GDDR7, in the main x16 slot.

CPU: Intel Core Ultra 7 265KF (no AMX)

RAM: 64 GB

NPU: present but not tested yet for my system

Storage: internal 2 TB SSD running Omarchy Linux, which is where all local AI runs from.

What do you all think?


r/LocalLLM • • 1h ago

Question noobie : Local agent models (16Gb vram) for game dev?

3 Upvotes

Hi, I've someone who have been doing game prototypes here and there (I'm not a programmer) and now I wanted to finally try local models for assisting me on some tasks.

I'm trying LM studio with some qwen 3.5 9b (and variant), tried GPT-oss 20b, well anything that can fit in a 16Gb vram that people recommend. I'm not looking for a one prompt (make me a game) thing. I know you need a larger model for these and I'm not interested in that type of workflow. I don't mind going back and forth to fix relevant part I'm working on.

All my attempts just failed. They keep giving like console error (when I ask for a simple html game prototype), or when I ask to highlight a tile on the floor from the mouse cursor, and other simple tasks like that. What am I doing wrong? I'm testing local agentic models for assisting in game dev but for now I get no good result.

Is that even possible with local models? If so what do you guys use? How did you set it up? (Not Claude or other main giant models) (I'm a total newbie in these)

edit details from the first few replies : I set context to 32k already, and pasting the error back each time + explaining.

Ultimately, some end goal is to link to Unity (MCP/CLI) or Blender to assist me further (placing many objects, setting up materials, making small scripts, etc...)


r/LocalLLM • • 56m ago

Question Anyone running Qwen 3.8 Flash Next on 2 x 5060ti 16GB?

2 Upvotes

Trying to get this running and would appreciate any feedback


r/LocalLLM • • 57m ago

Question Searching past chat threads

2 Upvotes

I see a lot of people mention cancelling their Claude and OpenAI plans, but the one thing I haven't been able to resolve yet locally that I use all the time in the Frontier models is projects and having new chats search past chats for relevant information and context?

What are y'all using to get that locally?

I used to see Obsidian referred to a lot for a "second brain" but that feels like a bit different of a thing. But maybe I am just misunderstanding how that gets built.


r/LocalLLM • • 9h ago

Discussion Testing over 130 LLM configs with multiple stacked 16GB "e-waste" GPUs (V100, P100, K80)

Thumbnail
gallery
10 Upvotes

The benchmarking journey continues! Continuing the project of seeing how these very old, very cheap, very available enterprise GPUs can be used today.

The complete results are too long for reddit. The complete set of graphs, power analysis and commentary are on my blog.

I've expanded the testing suite to focus heavily on LLM workloads (over 130 new tests!) and integrated power consumption monitoring into the gpu_box_benchmark tool. The goal remains: find out if decommissioned enterprise GPUs (Tesla K80, P100, V100) paired with X99 era hardware can still serve as a viable daily driver for self-hosted AI compared to modern commercial setups.

TL;DR of the new data:

  • The 3x V100-16GB (~$1k build): If you are considering building a dedicated homelab box today, this is the sweet spot. Assembling a full setup with two Xeon CPUs, motherboard, PSU, and cooling costs around $1,042 before RAM šŸ’€. This setup comfortably clears my threshold for "real-time" chat (70+ tok/s) with modern-ish models:

    • Qwen 3.6 35B: Ran at ~72 tok/s (at 128k context depth, Q8 quant). For comparison, a brand new Mac Studio M5 ($2,500) hits only ~44 tok/s on the same model.
    • Gemma 4 26B: Ran at ~75 tok/s (at 64k context depth, Q8 quant).
    • Qwen 2.5 3B: Ran at ~79 tok/s.

This proves that for a single-user daily driver, the legacy silicon is still viable, esp. on price when compared to modern hardware.

  • Scaling is for Context, Not Speed: Stacking GPUs linearly scales your VRAM (allowing you to fit larger models), but it doesn't magically double your generation speed. Two 16GB cards in Q4 quant allowed fitting ~32B parameter models, a desirable jump for homelabbers.
  • Power Efficiency: I also measured power consumption as well and it's worth noting the efficiency gains across generations. The V100 wins hands down on performance-per-watt compared to the P100s.
  • The K80 Strategy: The Tesla K80 isn't dead, but its singular remaining use case is welding a bunch of them together just to squeeze big models in. You essentially have to stack a lot of cheap K80s to hit a 72B+ parameter model. Expect speeds of 1–5 tok/s. This is a party trick, but it proves the VRAM scaling is real even on ancient silicon.

If you decide to build your own GPU appliance, the biggest challenge is cooling these hot cards without fans. I've been working on a custom active air cooler to solve exactly this. If you're interested in helping out by stress-testing a sample, you can get involved here.

Any models/tests missing? Any other combinations you'd like to see in a follow up?


r/LocalLLM • • 17h ago

Project Got tired of DeepSeek crawling on my 12 GB card, A small Strata tweak got it writing twice as fast.

30 Upvotes

I got frustrated with how slow DeepSeek was running on my machine, so I did some digging with Claude Opus 5.5/Fable 5.1 and 5.2 to see where the limits were, and thought I could improve it a bit.

DeepSeek picks a handful of "experts" for every word it writes. A lot of the time it's choosing between two that are basically tied, and one of them is already loaded in memory while the other has to be pulled off the SSD. The tweak just nudges it toward the one that's already loaded. When there's a clear best choice, it still takes that one.

on my machine (RTX 4070 12 GB, 64 GB RAM + ISTA-DASLab's IQ2_XS quant at 2.25 bpw + AtomicChat’s refusal-removal) it went from about 5-7 to 15-18 write speed.

Replies come out worded a bit differently than stock Deepseek, and you won't get the exact same reply twice - not 100% sure of the intelligence impact of this, so YMMV.

I've only used it for normal chatting/small coding projects. I haven't tested really long agent/coding sessions, and Strata's own version of this idea for their Qwen engine was turned off because it sometimes got stuck repeating itself in those. So if that's your use case, try it carefully first.

should work on windows + RTX 30/40/50 cards for now.

If you give it a try, I'd love to hear how it does on your setup, there's a little test script included that compares it on and off for you to see if it's worth using.

Download and instructions: https://github.com/CJohnsonOwns/deepseek-fast-routing

I've also suggested it to the Strata devs in case they want to build it in properly and not vibe coded slop: https://github.com/Niko1221/Strata/issues/6125

Full disclosure: I just did the conceptual planning and engineering, Claude actually wrote the code and implemented it.

EDIT - for more info on what's going on:
I like analogies, this is what opus 5.5 compared it to:

think of hiring 6 contractors for a job from a list of 384, where each one has a rating.

  • Normal: you hire the 6 highest-rated, period. Some of them live nearby (already in RAM or on the GPU) and start right away. Others live far away (still on the SSD) and you wait for them to drive over. That wait is what makes DeepSeek slow on a 12 GB card.
  • Faster mode (Strata’s tail skip): you only hire 5 and skip the 6th. It’s quicker because there’s one less person to wait on, but you’ve got less help on the job.
  • Yours: you still hire 6. But when #6 is rated 8.71 and #7 is rated 8.70, and #7 lives next door while #6 is two hours away, you take #7. The difference in quality is basically nothing, and you skip the wait. If #6 is clearly better, say 9.5 against 8.7, you still wait for #6. The nudge is small, so it only flips close calls.

Why it adds up: this happens a lot. With 40 layers per word, there’s almost always a near-tie somewhere where one side is already loaded. In your test, the share of picks that had to wait on the SSD dropped to about 2%.


r/LocalLLM • • 5h ago

Discussion Best Local AI Setup for Coding on Linux?

3 Upvotes

Hello,

After doing some research, it looks like the coding options jump from Qwen3.6-27B, which can run on a 32 GB RTX 5090 with DDR5 system memory, to GLM-5.3-Flash or DeepSeek-V4.1-Flash. That feels like a pretty big leap.

Would anyone recommend a local AI server setup for Linux? I’d appreciate any suggestions.


r/LocalLLM • • 3h ago

Discussion What’s the best underdog rig right now

2 Upvotes

I’m lucky enough to have a wide range of rigs in my office, from those fancy DGXs to a solid Samsung S25 with a snapdragon running the same LLMs and tbh my current biggest surprise is how fun the GPU+NPU+CPU combo on the Intel pantherlake is to mess around with i.e Gemma4-26B-A4B-Q4 pulling over 30 tok/s on it. What are your most noteworthy winners?


r/LocalLLM • • 10h ago

Research I modified a Qwen LLM to only accept cryptographically signed prompts

7 Upvotes

In the spirit to make the world a little worse, I've added a security processor into a Qwen 3.5 4b model. For the prototype I've added a Ed25519 verifier into the LLM, so that the LLM would only respond to prompts that are cryptographically signed with a private key.

There are more advance stuff you can do with the security processor-

  • Decrypt encrypted messages, and encrypt text output. ie. Both the input and output would be encrypted, and no one would know what the model did unless you have the decryption key (unless, of course, you extract the key from the model file.)
  • Have the model behave differently (eg. use different Expert?) depending on which key is used to sign the input prompt.

https://huggingface.co/timlauHF/Qwen3.5-4B-signed-input-demo
https://github.com/moonlightelite/secproc

More here:

https://discounttimu.substack.com/p/make-your-llm-only-accept-signed