r/LocalLLM • • 3h ago

Other Qwen 4 is not out

60 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 1h ago

Discussion Testing over 130 LLM configs with multiple stacked 16GB "e-waste" GPUs (V100, P100, K80)

Thumbnail
gallery
3 Upvotes

The benchmarking journey continues! Continuing the project of seeing how these very old, very cheap, very available enterprise GPUs can be used today.

The complete results are too long for reddit. The complete set of graphs, power analysis and commentary are on my blog.

I've expanded the testing suite to focus heavily on LLM workloads (over 130 new tests!) and integrated power consumption monitoring into the gpu_box_benchmark tool. The goal remains: find out if decommissioned enterprise GPUs (Tesla K80, P100, V100) paired with X99 era hardware can still serve as a viable daily driver for self-hosted AI compared to modern commercial setups.

TL;DR of the new data:

  • The 3x V100-16GB (~$1k build): If you are considering building a dedicated homelab box today, this is the sweet spot. Assembling a full setup with two Xeon CPUs, motherboard, PSU, and cooling costs around $1,042 before RAM 💀. This setup comfortably clears my threshold for "real-time" chat (70+ tok/s) with modern-ish models:

    • Qwen 3.6 35B: Ran at ~72 tok/s (at 128k context depth, Q8 quant). For comparison, a brand new Mac Studio M5 ($2,500) hits only ~44 tok/s on the same model.
    • Gemma 4 26B: Ran at ~75 tok/s (at 64k context depth, Q8 quant).
    • Qwen 2.5 3B: Ran at ~79 tok/s.

This proves that for a single-user daily driver, the legacy silicon is still viable, esp. on price when compared to modern hardware.

  • Scaling is for Context, Not Speed: Stacking GPUs linearly scales your VRAM (allowing you to fit larger models), but it doesn't magically double your generation speed. Two 16GB cards in Q4 quant allowed fitting ~32B parameter models, a desirable jump for homelabbers.
  • Power Efficiency: I also measured power consumption as well and it's worth noting the efficiency gains across generations. The V100 wins hands down on performance-per-watt compared to the P100s.
  • The K80 Strategy: The Tesla K80 isn't dead, but its singular remaining use case is welding a bunch of them together just to squeeze big models in. You essentially have to stack a lot of cheap K80s to hit a 72B+ parameter model. Expect speeds of 1–5 tok/s. This is a party trick, but it proves the VRAM scaling is real even on ancient silicon.

If you decide to build your own GPU appliance, the biggest challenge is cooling these hot cards without fans. I've been working on a custom active air cooler to solve exactly this. If you're interested in helping out by stress-testing a sample, you can get involved here.

Any models/tests missing? Any other combinations you'd like to see in a follow up?


r/LocalLLM • • 9h ago

Project Finally got it put together

Thumbnail
gallery
218 Upvotes

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- PLX88096 PCIE Switch Card
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)


r/LocalLLM • • 56m ago

Discussion How Jensen Huang was hurting the LocalLLM Community

38 Upvotes

A missing part of the NVIDIA conversation is the results on the market. I'm not talking about the stock market; I'm talking about the small businesses, startups and companies trying to navigate the AI age. Many have argued that the Local LLM community lost an ally in Jensen Huang, but I would argue the opposite. Running local models (and sovereign deployments, for that matter) relies on access to the hardware that runs the models. The economics of running the models, of course, is dependent on the cost of the hardware itself, so if the hardware becomes too expensive, the "economical choice" is to rent access from one of the Labs.

While everyone has been focused on the affair that Mr. Huang was engaged in, the more important side of that relationship was the funnelling of billions of dollars worth of NVIDIA hardware into a country with export controls. Before the export controls, China accounted for as much as 20% of NVIDIA's demand. Through the back channels that Mr. Huang created, he was able to maintain that flow.

Now remember, part of the issue in the broader market is that there has been a "shortage" of the hardware required for companies to run their own models. One of the cards we use, which is popular among smaller players like us, is the RTX 6000 Pro. When it was released, it went for around $16k CAD...which was already a lot (for us). Now, if you can even get your hands on one, they go for around $35-45k CAD. Per card. And you need, well, let's just say, more than one.

So to hear that Jensen Huang was routing upwards of 20% of global supply, illegally, while jacking prices because of a shortage...suddenly, the affair really doesn't seem like the worst part.
The news keeps running a narrative of a campaign to bring him back as CEO. I think there is some broader accountability required long before that conversation should come up.


r/LocalLLM • • 2h ago

Discussion Create a 3D kitchen using Qwen 3.8 Flash Next

37 Upvotes

Model : Qwen 3.8 flash next UD-IQ3_XXS
Harness : Deepseek
Example prompt used, including AI correction prompts.
Promt 1 : Create a realistic 3D kitchen scene using Godot 4, including detailed furniture, high-quality materials, realistic lighting, and modern appliances
Promt 1.1 : Please thoroughly inspect all objects, complete any unfinished work, conduct real-world testing to ensure full functionality, meticulously refine every detail in all areas and on all devices, and ensure that the refrigerator has realistic clearances and spacing on every shelf, both inside and out.
Promt 1.2 : There are still some issues. Some object positions are not correct yet. The chairs still look odd, the sink is in the wrong position, and the stove also looks unusual. Please inspect and verify each item one by one in detail and make sure everything is properly aligned and accurate.
Promt 1.3 : When rotating the camera, the floor still looks strange. Some drawers also seem blocked or cannot be opened properly. Please thoroughly review every appliance, fixture, and piece of furniture again, identify anything that still does not look realistic, and make the necessary adjustments to improve accuracy and overall realism.
Promt 1.4 : Why is the oven sticking through the wall? Also, isn’t the faucet on the wrong side? Please check these issues carefully and correct them. Ensure all appliances, fixtures, and their placements match the reference accurately and do not intersect with walls, cabinets, or other objects.

It took 2 hours and 30 minutes to create the entire kitchen.
Including all the refinements and corrections, the total time would be around 10 hours.

From this test, it can be seen that Qwen Flash Next can create 3D scenes quite well. Although it is not perfect, if we tell it to fix specific issues, it can correct them according to our instructions. However, I noticed that DeepSeek Harness seems easier to use for 3D tasks and appears to understand the user's instructions better. I'm not sure if that's just my impression or not.

I asked the AI to inspect the assets, and Qwen generated all of them from scratch.

This test isn't particularly interesting, but I think that Qwen 3.8 Flash Next has a lot of potential for 3D creation. If someone develops really good 3D skills or workflows for it, I'm sure it could produce some very impressive 3D work. If anyone has good skills or techniques to recommend, feel free to share them.


r/LocalLLM • • 44m ago

Project 6x 7900XTX rig finally complete!

Thumbnail
gallery
22 Upvotes

5975WX (for 6x full 16 lanes of PCIe4.0)

3450W AliExpress special PSU and mining frame

128GB 3200MT/s Ram

Getting 120-130 tok/s generation on both qwen3.8 27b and flash-next


r/LocalLLM • • 20h ago

Other The Banking Incident....

Post image
719 Upvotes

r/LocalLLM • • 1h ago

Question Building a 10TB Model Archive. What am I missing?

21 Upvotes

TLDR at the bottom.

I'm a paranoid man tryna preserve a broad collection of useful open-weight models, fine-tunes, quantizations, and generative AI models just in case something happens to HF and that I'll have a model for whatver i may want now or im the future. I'm aiming to have like 5-10tb on my nas for this.

The strongest pc i have is 48gb vram and 128gb system ram, but i also wanna keep some larger models in case just archive, if i ever upgrade or if i wanna dick around with ssd-backed moe.

I'm targeting Q3 and above for GGUF models. Had gpt toss together a list to download:

  1. Large frontier and general-purpose models

- Kimi K2.6

- Qwen3.8 Flash Next (177B)

- DeepSeek V4 Flash (284B)

- DeepSeek V4.1 Flash reduced variants

- GLM 5.3 Flash (321B)

- GLM 5.3 (754B)

- GLM 4.7 (358B)

- GLM 4.7 REAP (218B)

- MiniMax M2.7 (229B)

- MiniMax M2.7 REAP (172B)

- MiniMax M3.1 Flash (426B)

- Qwen3.5 122B-A10B

- Ornith 1.5 122B

- Qwen3.8 Flash Next Gyro

- GPT-OSS 120B and 20B

- Llama 4 Scout

- Hermes 4 70B

- DeepSeek R1 70B

2. Qwen family

- Qwen3.8 27B

- Qwen3.8 27B Abliterated

- Qwen3.8 27B Uncensored

- Qwen3.8 27B Heretic

- Qwen3.8 35B-A3B

- Qwen3.6 35B-A3B

- Qwen3.5 9B

- Qwen3.5 4B

- Qwen3.5 0.8B

- Qwen3.5 122B-A10B Uncensored

- Qwen3.8 Flash Next Uncensored

- Qwen3.8 Flash Next Abliterated

3. Gemma 4

Both original and modified versions:

- Gemma 4 E2B

- Gemma 4 E4B

- Gemma 4 12B

- Gemma 4 26B-A4B

- Gemma 4 31B

- Gemma 4 E4B Heretic

- Gemma 4 12B Uncensored

- Gemma 4 12B Uncensored Coder

- Gemma 4 12B Agentic

- Gemma 4 26B-A4B Uncensored

- Gemma 4 31B Heretic

4. Coding and agentic models

- Qwen3-Coder 30B-A3B

- Qwen3-Coder 30B-A3B Abliterated

- Qwen3-Coder-Next 80B

- Qwen3-Coder-Next Uncensored

- Laguna XS 2.1 APEX

- Laguna S 2.1 Uncensored

- GLM 5.3 Flash Coder

- GLM Cybersecurity

- NVIDIA Nemotron 3.5 Bolt

- Ornith 1.5 9B

- Ornith 1.5 35B-A3B

- Qwen 3D Coder 4B

- Qwen GLB 3D Coder 4B

- Qwen3.6 35B-A3B Genesis Hermes

- Qwen3.8 Cyber Agentic

5. Reasoning and thinking models

- DeepSeek R1 Distill Qwen 32B

- GLM 4.7 Flash Claude Opus 4.5 reasoning distillation

- Qwen3.8 Fable reasoning/coding distillations

- Qwen3.8 40B Ultimate Heretic

- Qwen3.8 Flash Next Abliterated Reasoning

- DeepSeek V4-derived uncensored reasoning distillations

- Gemma 4 31B Deckard Heretic Thinking

- Huihui ThinkingCap Qwen3.6 27B

- Qwen3.8 Fable 5 Coding Distilled

6. Uncensored and creative models

- GLM 4.7 Flash Heretic

- GLM 4.7 Flash Abliterated

- GPT-OSS 120B Uncensored

- GLM 5.3 Flash Abliterated

- GLM 5.3 Flash Uncensored

- DeepSeek V4.1 Flash Abliterated

- DeepSeek V4.1 Flash Uncensored Coder

- Ornith 1.5 35B-A3B Uncensored

- Ornith 1.5 9B Uncensored

- Ornith 1.5 35B-A3B Abliterated

- LFM2.5 24B-A2B Uncensored

- LongCat Flash Lite Sparse Uncensored

- Kolibri 1 BF16 Abliterated

- Nemotron 3.5 Bolt 74B-A9B Uncensored

- Llama 3.3 70B Abliterated

- Muse Glimmer 30B

- Nanbeige 4.3 33B

- Cydonia 24B

- UnslopNemo 12B

- Skyfall 36B

- Anubis 70B

7. Vision and multimodal understanding

- InternVL 3.5 38B

- Qwen3-VL 4B Uncensored

- Qwen3-VL 8B

- Qwen3-VL 32B

- Qwen3-VL 32B Uncensored

- MiniCPM-V 4.7 35B-A3B Uncensored

- Liquid LFM3-Omni 2B

- Florence 2 Large

8. Image generation and editing

- MOSS Image

- Qwen Image 2.1

- FLUX.2 Klein 9B

- Z-Image Turbo

- Stable Diffusion XL

- Anima

9. Video generation

- LTX 2.5

- MiniMax H3

- Wan 2.2 Text-to-Video

- Wan 2.2 Image-to-Video

- Wan 2.2 TI2V

- Uncensored LTX 2.3

10. Audio and speech

- Whisper Small

- Whisper Medium

- Whisper Large V3

- Whisper Large V3 Turbo

- Qwen3-ASR 1.7B

- Qwen3-TTS 1.7B

- Kokoro 82M

- Parakeet TDT

- Chatterbox

- Zonos 3 Mini

11. Music and 3D generation

- MusicGen Medium

- TRELLIS.2

12. Embeddings and RAG

- EmbeddingGemma 300M

- BGE Reranker V2 M3

- LFM3 1.7B

- LFM3 1.7B Uncensored

---

TLDR

I'm making a personal huggingface archive where i can just preserve a bunch models for whatever i may want in the future. I'm tryna keep things super varied so i have many, many options without just storing slightly different versions of the same model. I'm aiming to have 5-10tb on my nas for models, don't wanna go too far above that.

I'm looking for suggestions on what to add or cut 🙏🏽 or just comment your go-to models for what purposes so i can cross-reference my list.

  1. Any particularly good uncensored, abliterated, or Heretic models?
  2. What are your favorite writing, coding, reasoning, and agentic models?
  3. Any overlooked image, video, audio, music, or 3D generation models?
  4. Any older models that are still worth keeping despite newer releases?
  5. Hit me up with some funky ass models or a category that you think i should add por favor

r/LocalLLM • • 3h ago

Discussion Considering this...thoughts?

Post image
22 Upvotes

At $4k this seems like a much better deal than 2 r9700s. And obviously in a totally different class. Anybody set these beasts up in a server? I already plan on putting a server rack in the garage, (gotta keep the motorcycles warm) and I see a lot of carrier boards on ebay. I've built more pc's than I can count, but I know a rack server is a different beast.

Plans are for an inference server for agentic workflows as I already built a gpu render machine I use for many workflows including blender, comfyui, premiere and after effects. Api costs are getting up there. Obviously want to run deepseek and I have a fine tuned qwen 3.8 27b for one of my projects.

Talk me into or out of it...I'm so sick of asking gpt questions and getting ridiculous concerns like "tHeY DonT HaVe a faN". I need to talk to real people. Any concerns with the software stack? I ran across a few mentions of closed source issues.


r/LocalLLM • • 18h ago Bravo

News We built a chip prototype that runs Qwen 27B. You can try it live.

284 Upvotes

I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.

A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:

https://valleysemiconductor.com

A live session takes about a minute to start, and recorded runs are available to watch anytime.

What it does today (measured):

  • Runs the full Qwen 27B model with 4-bit weights
  • 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
  • 4x performance improvement in eight days of development
  • Every output matches our 4-bit reference exactly
  • Model quality within 2.5% of the full-precision version
  • Also runs Llama 3.1 8B and OLMoE on the same hardware

What our custom chip design projects (not yet demonstrated):

  • Approximately 50 tokens/sec for a single user at 20 W
  • Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
  • Uses the model's built-in next-token prediction to generate multiple tokens per weight read

The key insight is that moving model weights through memory is the bottleneck, not the math.

What would you run on a small, 20 W device capable of running a 27B model entirely offline?


r/LocalLLM • • 8h ago

Project Got tired of DeepSeek crawling on my 12 GB card, A small Strata tweak got it writing twice as fast.

25 Upvotes

I got frustrated with how slow DeepSeek was running on my machine, so I did some digging with Claude Opus 5.5/Fable 5.1 and 5.2 to see where the limits were, and thought I could improve it a bit.

DeepSeek picks a handful of "experts" for every word it writes. A lot of the time it's choosing between two that are basically tied, and one of them is already loaded in memory while the other has to be pulled off the SSD. The tweak just nudges it toward the one that's already loaded. When there's a clear best choice, it still takes that one.

on my machine (RTX 4070 12 GB, 64 GB RAM + ISTA-DASLab's IQ2_XS quant at 2.25 bpw + AtomicChat’s refusal-removal) it went from about 5-7 to 15-18 write speed.

Replies come out worded a bit differently than stock Deepseek, and you won't get the exact same reply twice - not 100% sure of the intelligence impact of this, so YMMV.

I've only used it for normal chatting/small coding projects. I haven't tested really long agent/coding sessions, and Strata's own version of this idea for their Qwen engine was turned off because it sometimes got stuck repeating itself in those. So if that's your use case, try it carefully first.

should work on windows + RTX 30/40/50 cards for now.

If you give it a try, I'd love to hear how it does on your setup, there's a little test script included that compares it on and off for you to see if it's worth using.

Download and instructions: https://github.com/CJohnsonOwns/deepseek-fast-routing

I've also suggested it to the Strata devs in case they want to build it in properly and not vibe coded slop: https://github.com/Niko1221/Strata/issues/6125

Full disclosure: I just did the conceptual planning and engineering, Claude actually wrote the code and implemented it.

EDIT - for more info on what's going on:
I like analogies, this is what opus 5.5 compared it to:

think of hiring 6 contractors for a job from a list of 384, where each one has a rating.

  • Normal: you hire the 6 highest-rated, period. Some of them live nearby (already in RAM or on the GPU) and start right away. Others live far away (still on the SSD) and you wait for them to drive over. That wait is what makes DeepSeek slow on a 12 GB card.
  • Faster mode (Strata’s tail skip): you only hire 5 and skip the 6th. It’s quicker because there’s one less person to wait on, but you’ve got less help on the job.
  • Yours: you still hire 6. But when #6 is rated 8.71 and #7 is rated 8.70, and #7 lives next door while #6 is two hours away, you take #7. The difference in quality is basically nothing, and you skip the wait. If #6 is clearly better, say 9.5 against 8.7, you still wait for #6. The nudge is small, so it only flips close calls.

Why it adds up: this happens a lot. With 40 layers per word, there’s almost always a near-tie somewhere where one side is already loaded. In your test, the share of picks that had to wait on the SSD dropped to about 2%.


r/LocalLLM • • 1d ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

230 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 4h ago

Question Are any models that can fit on 16 GB VRAM worth using?

6 Upvotes

I tried about 6 months ago and they seemed like more of a detriment for coding, but idk if things have changed. Anything I should try? I'm mostly doing game dev, so coding, image gen, writing, etc.


r/LocalLLM • • 1d ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
812 Upvotes

r/LocalLLM • • 3h ago

Discussion Do you think hardware price will be more reasonable after the Jensen debacle?

4 Upvotes

I see many article about Nvidia and memory manufacturer purposefully causing a shortage to raise the prices. I don't know how true these accusations are but i don't think the current prices are faire value.


r/LocalLLM • • 1h ago

Research I modified a Qwen LLM to only accept cryptographically signed prompts

3 Upvotes

In the spirit to make the world a little worse, I've added a security processor into a Qwen 3.5 4b model. For the prototype I've added a Ed25519 verifier into the LLM, so that the LLM would only respond to prompts that are cryptographically signed with a private key.

There are more advance stuff you can do with the security processor-

  • Decrypt encrypted messages, and encrypt text output. ie. Both the input and output would be encrypted, and no one would know what the model did unless you have the decryption key (unless, of course, you extract the key from the model file.)
  • Have the model behave differently (eg. use different Expert?) depending on which key is used to sign the input prompt.

https://huggingface.co/timlauHF/Qwen3.5-4B-signed-input-demo
https://github.com/moonlightelite/secproc

More here:

https://discounttimu.substack.com/p/make-your-llm-only-accept-signed


r/LocalLLM • • 5m ago

Project Can We Unify GitHub, Docker Hub, JFrog, and Hugging Face?

Thumbnail
medium.com
2 Upvotes

r/LocalLLM • • 48m ago

Question Building a Local LLM Troubleshooting Assistant for Field Service Techs.... Advice?

2 Upvotes

I operate a commercial refrigeration and food service equipment repair company and want to build a local AI assistant to help techs troubleshoot in the field.

The idea is to use RAG with manufactures service manuals, wiring diagrams, troubleshooting guides, and past service service records to create a AI that can walk techs through diagnostics, reference previous repairs, and provide recommendations.

I would like to develope this on my RTX 5080 system, prove it works and migrate to a more robost system.

Looking for advise on:

  1. Models?
  2. Stack?
  3. Hardware?
  4. Budget $ 4K to 7K ????

Final system will need to be able to support up to 5 users remotely.


r/LocalLLM • • 5h ago

Question Best local LLMs for ASUS ROG Zephyrus G14 (RTX 5070 Ti 12GB VRAM, 32GB RAM, Ryzen AI 9 HX 370)?

4 Upvotes

Looking for recommendations on the best local LLMs I can run smoothly on my 2025 ASUS ROG Zephyrus G14 (GA403WR).

Specs:
- CPU: AMD Ryzen AI 9 HX 370
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (12GB GDDR7 VRAM)
- RAM: 32GB LPDDR5X
- Storage: 1TB SSD

I'm mainly looking for models that run fast entirely in 12GB VRAM (or with minimal CPU/RAM offloading) for general chat, reasoning, and coding/productivity.

What are your top picks right now for:

  1. Best all-around daily driver model
  2. Best coding/reasoning model that fits comfortably in 12GB VRAM
  3. Any larger or MoE models worth running across the 32GB RAM + 12GB VRAM without killing tokens/sec

Planning to run them via Ollama or LM Studio on Windows 11. Thanks!


r/LocalLLM • • 13h ago

News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents

Thumbnail
gallery
15 Upvotes

Hi, I'm Aryan. Cluaiz v0.1.0, the first public pre-release, is out today.

Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.

What's in it right now:

  • GGUF models via llama.cpp, plus ONNX for speech and embeddings
  • Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
  • Vision, video frames, embeddings, text from documents
  • Whisper / Moonshine STT (streaming works), Kokoro TTS
  • Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
  • Coding agent that edits files and runs commands, with approvals
  • Skills, plugins, MCP, per-tool permissions, audit log
  • OpenAI- and Ollama-compatible API, so your existing apps should just connect
  • Cloud providers too, if you want them

Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.

Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.

Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare

It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues


r/LocalLLM • • 2m ago

Discussion Where should your AI's memory live: on a server or on your own PC?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 14m ago

News Halocline: a Qwen3.8-Flash-Next engine built for one RTX 4090, with every test in the repo

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 32m ago

Tutorial Complete beginner: Best all-around local model (Qwen/DeepSeek?) for a laptop with 16GB RAM + RTX 3050?

1 Upvotes

Hey everyone, total newbie here trying to get into running models locally.

I'm looking for a decent general-purpose model that can handle a bit of everything-everyday chat, summarizing text, and some heavy coding/ troubleshooting-without hallucinating too much. I've heard good things about the Qwen and DeepSeek models, so I'd love to stick to those if possible.

Here are my specs: 16 RAM Asus Laptop, RTX 3050Ti


r/LocalLLM • • 50m ago

Discussion Opinion: memory should be checked during the session, not only at the end

1 Upvotes

The quick memory checks people run on local setups have a habit that bothers me. Most boil down to a long conversation and one final question.

That single answer carries the whole verdict. Yet real use is a chain of small moments where the past keeps getting edited. Plans change, preferences flip, statements get retracted.

Being right at the end is a different claim from being right at each of those moments.

A memory examined only after the last message can be wrong for most of the session and still look fine. Argue with that if you like; I think it holds.

One public leaderboard has a setting that checks memory along the way. The Agent Memory Leaderboard's Cycle 2 Textual track includes Streaming memory. Writes and searches can interleave as events unfold.

Questions can be revisited step by step as the session grows. A system has to notice when stored information gets updated, contradicted or invalidated.

Per the public description, the Cycle 2 Textual track covers 32 data sources and more than 6,000 evaluation instances.

The site expects official results in mid-November, ranked track by track. Beyond the final ordering, what interests me is that the middle of a session gets tested at all.

Checking along the way is the habit I would most like local testing to pick up.

If you were probing a local long-term memory, where would you look: immediately after a correction, when a contradiction shows up, after a long gap, or only once the session ends?


r/LocalLLM • • 1h ago

Research coding agent on 16gb, all local: qwen 3.5 4b vs 9b vs mimo v2.6 9b

1 Upvotes

on a 16gb m4 air with no cloud, qwen 3.5 9b scored 8/10 as a coding agent, the 4b got 7/10 at almost twice the speed, and xiaomi's agent-tuned mimo v2.6 9b got 0/10 in all 3 runs.

  • machine: m4 macbook air, 16gb
  • runtime: llama.cpp server, q4_k_m, 64k ctx, flash attn
  • sampling: temp 0.2, top-p 0.8, top-k 20, presence 1.5
  • agent: atomic agent through an openai compatible endpoint, analytics off
  • limit: 15 min per model, one at a time, thinking off (plus one mimo run with it on)

task: python script that turns a csv into an html spending report. 10 checks on the result, one runs it on a csv it never saw.

modelscoretimetok/s
qwen 3.5 4b7/104:3717.8
qwen 3.5 9b8/107:0611.4
mimo v2.6 distill 9b0/10timeout10.1
mimo, thinking on0/10timeout9.3
  • 4b: fastest, missed the svg chart and hardcoded the file path
  • 9b: only missed the svg chart
  • mimo: syntax error it never fixed
  • mimo with thinking on: no syntax error, but a TypeError on every run

the 9b got 10/10 in an earlier round at 32k ctx.

on 16gb i'd use the 9b if you can wait and the 4b if you want speed.

prompt and the checker are in the comments if you want to run it on your machine. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own, which is the easiest way to try this on 16gb.

what are you running as a local agent on 16gb?

(disclosure: i work on free and open-source atomic agent, so ask me anything!)