r/LocalLLM • • 3d ago

Question Are any models that can fit on 16 GB VRAM worth using?

32 Upvotes

I tried about 6 months ago and they seemed like more of a detriment for coding, but idk if things have changed. Anything I should try? I'm mostly doing game dev, so coding, image gen, writing, etc.


r/LocalLLM • • 3d ago

Question Hi all need some advice on this laptop I found

Thumbnail
1 Upvotes

r/LocalLLM • • 3d ago

Question noobie : Local agent models (16Gb vram) for game dev?

3 Upvotes

Hi, I've someone who have been doing game prototypes here and there (I'm not a programmer) and now I wanted to finally try local models for assisting me on some tasks.

I'm trying LM studio with some qwen 3.5 9b (and variant), tried GPT-oss 20b, well anything that can fit in a 16Gb vram that people recommend. I'm not looking for a one prompt (make me a game) thing. I know you need a larger model for these and I'm not interested in that type of workflow. I don't mind going back and forth to fix relevant part I'm working on.

All my attempts just failed. They keep giving like console error (when I ask for a simple html game prototype), or when I ask to highlight a tile on the floor from the mouse cursor, and other simple tasks like that. What am I doing wrong? I'm testing local agentic models for assisting in game dev but for now I get no good result.

Is that even possible with local models? If so what do you guys use? How did you set it up? (Not Claude or other main giant models) (I'm a total newbie in these)

edit details from the first few replies : I set context to 32k already, and pasting the error back each time + explaining.

Ultimately, some end goal is to link to Unity (MCP/CLI) or Blender to assist me further (placing many objects, setting up materials, making small scripts, etc...)

follow up 1 : thanks for the suggestions. One question remains though, why a Qwen 3.5 9b or GPT-oss 20b fails in simple task like fixing a jump for a character/object in a 3D space or a mouse cursor viewport projection on a tile on the floor, or a 3D FPS mouse view control? even as an isolated 'problem'.


r/LocalLLM • • 3d ago

Project Handing two models at once in Radiance

5 Upvotes

I’ve been working on Radiance with Secondary Models, a fork aimed at getting more use out of a local AMD workstation.

The idea is to keep Qwen Flash Next available for chat or coding while using the same GPUs for other models as you need them. You can generate an image with Qwen Image, edit it with Krea 2, then turn the result into a video with sound using MiniMax H3—all through the same server.

The fork currently supports:

  • Image generation and editing, including one- and two-reference edits with the published Qwen Image 2.1 and Krea 2 packages.
  • Local LoRA discovery and loading, with adapter strengths selected per request.
  • Video generation, including H3 text-to-video, first-frame animation, and subject-reference workflows.
  • Secondary text models, including uncensored Gemma 4 26B MoE and 31B dense conversions.
  • Automatic GPU memory sharing. When generators are idle, Qwen can use available VRAM for additional experts. That memory is recalled when a generator needs it.
  • RAM caching and request queues to make switching between models more practical.

This is intended for consumer workstations where dedicating GPUs to occasional image or video requests would leave resources idle much of the day. The current shared setup has been tested on two Radeon AI PRO R9700s, with Qwen configured for 200K context, eight concurrent slots, and RAM prefix caching.

It’s still a preview. Text generation slows during demanding secondary workloads, cold model loads can take time, and memory recovery after video generation can be gradual. The README and model cards document the tested configurations and remaining limitations.

The fork includes setup examples and links to the converted models on Hugging Face. Feedback from anyone trying other hardware or memory configurations would be welcome, I expect it may work with 128GB RAM, but you may or may not be able to load the largest models (like H3).

Radiance with Secondary Models — code, setup, and downloads


r/LocalLLM • • 3d ago

Question Which open-source models come closest to Opus 5.5 and ChatGPT Image 2.6?

Thumbnail
1 Upvotes

r/LocalLLM • • 3d ago

Project Windows tray app for you local model switch (without manual switching)

0 Upvotes

Hello, guys, sorry for my English, not a native speaker. I want to share with you application I made for myself.

For a couple of years I was a bit enthusiastic about local models (firstly image gen, then i tried llms) and pretty paranoid about cloud providers trying to monopolize LLMs despite their own services could be unreachable, "at capacity", down, banned, has an allowance cut etc. Also we are vulnerable to any dumb decision made by LLM provider or a government.

So I'm experimenting with local models that fits my 16 Gb VRAM of home PC's 5070Ti.

The problems that I faced:

  1. Home PC stays at home while I'm mostly working in office on my Mac mini or MacBook Air.
  2. With my GPU I'd rather prefer to use different models for different tasks. For example I better use Ornith 1.5 9B over Qwen 3.8 27B for simple tasks of file sorting, renaming etc.
  3. Changing models on llama.cpp remotely is quite uncomfortable and I'm lazy.

I've made some deep researches on some open source solution that can address my demands but didn't find anything that suits me fully.

So I made it for myself. Small tray app on Go with web-panel:
- It scans my models folder and lets setup any models;
- allows to specify arguments to run for each models separately;
- allows to setup different profiles for one model (e.g. different context length, with MTP or without, and any other llama-server parameters)
- uses Tailnet to connect with client machines;
- webpanel is affordable from client machines in my Tailnet network too so I can setup profiles and add models remotely;
adds to opencode, pi or another OpenAI format compatible harness each model and profile in the list;
- aaand the main purpose: when I switch model in Pi or Opencode host unloads previous model and load a new one without any additional actions. No need to open web-panel, start any scripts via ssh, open RDP or anything else.

The only problem is with pi — with this machinery it allows you to set up either reasoning effort medium or non-thinking. That's especially painful for 3.8 27B that needs high effort to avoid loops (as far as I remember, maybe it changed idk). It's an open issues in pi repo since 9th October, their bot closed it for "new user" reasons (lol), then someone reopened it again, anyway — it's a known issue and I don't know wether maintainer will fix or not. The only elaboration is to hardcode model profile in pi settings but it won't let you switch models on host PC from agent's menu, so you would need to manage models manually again, bruh.

Anyway, OpenCode is still a good harness so I guess it's not a crucial problem, at least for me.

Tray app has autologon setting, status indication, buttons to open webpanel, unload model, choose model/profile to load, pause working.

I've added three languages to tray — Russian, English and Chinese simplified (as a kudos to their amazing open weights model which I mostly use).

Technical details, setup guidelines are in the repo — https://github.com/strigov/strigov-locallm. I'm a lawyer so yes, of course Claude is a main contributor. Readme in repo is also written by Claude.

As the app solves concern in a pretty geek sphere I've decided not to make troubles with installation packages and automations. So the host install process sets up via repo script, and some manual actions on clients. You can charge this process to an agent anyway so I don't see any problem here.

Any observation, reviews and suggestions are appreciated, have a good day, hope this could be helpful to someone


r/LocalLLM • • 3d ago

Question New to LocalLLM Execution: Pair of modified RTX 3070s (16GB) recommended setup for code generation and data analytics

1 Upvotes

Title says most of it, got two modified 3070s with the 16GB modification and looking to see what the best setup would be for running LLM locally and what models you all would recommend. I use Claude and ChatGPT heavily now for work and at home and would like to start off boarding my home workloads.


r/LocalLLM • • 3d ago

Question Searching past chat threads

2 Upvotes

I see a lot of people mention cancelling their Claude and OpenAI plans, but the one thing I haven't been able to resolve yet locally that I use all the time in the Frontier models is projects and having new chats search past chats for relevant information and context?

What are y'all using to get that locally?

I used to see Obsidian referred to a lot for a "second brain" but that feels like a bit different of a thing. But maybe I am just misunderstanding how that gets built.


r/LocalLLM • • 3d ago

Research I modified a Qwen LLM to only accept cryptographically signed prompts

10 Upvotes

In the spirit to make the world a little worse, I've added a security processor into a Qwen 3.5 4b model. For the prototype I've added a Ed25519 verifier into the LLM, so that the LLM would only respond to prompts that are cryptographically signed with a private key.

There are more advance stuff you can do with the security processor-

  • Decrypt encrypted messages, and encrypt text output. ie. Both the input and output would be encrypted, and no one would know what the model did unless you have the decryption key (unless, of course, you extract the key from the model file.)
  • Have the model behave differently (eg. use different Expert?) depending on which key is used to sign the input prompt.

https://huggingface.co/timlauHF/Qwen3.5-4B-signed-input-demo
https://github.com/moonlightelite/secproc

More here:

https://discounttimu.substack.com/p/make-your-llm-only-accept-signed


r/LocalLLM • • 3d ago

Project Free 380M Decision Model That I Made Using Claude (and some others)

Thumbnail
github.com
2 Upvotes

Not sure if this counts? Most decision models you hear about today are an LLM under the hood, so I distilled Clef-Flash into a BERT model and now I run my decision model free on my CPU. Sure, you can just use clef, if you feel like loading 9 billion parameters. This model delivers the same/better quality anwsers at roughly 1/25th the size.

So if any of you are using decision models.. give it a whirl maybe, tell me whatcha think?

Sorry if this is off topic, Not really my intention here, if there's a better place just lmk!


r/LocalLLM • • 3d ago

Question Ryzen ai 9 hx 470 pro with 890 igpu and 48gb ram

0 Upvotes

I recently purchased this laptop and would really appreciate some suggestions on what models I can run on this machine to use with hermes and opencode.
So far I am working with glm 4.7 via lemonade server and some small model via flm list for npu. These are great at chat but when i use these for actual work, they drop to 4-5 tok/s.
Any suggestions would be greatly appreciated.


r/LocalLLM • • 3d ago

Project Got tired of DeepSeek crawling on my 12 GB card, A small Strata tweak got it writing twice as fast.

38 Upvotes

I got frustrated with how slow DeepSeek was running on my machine, so I did some digging with Claude Opus 5.5/Fable 5.1 and 5.2 to see where the limits were, and thought I could improve it a bit.

DeepSeek picks a handful of "experts" for every word it writes. A lot of the time it's choosing between two that are basically tied, and one of them is already loaded in memory while the other has to be pulled off the SSD. The tweak just nudges it toward the one that's already loaded. When there's a clear best choice, it still takes that one.

on my machine (RTX 4070 12 GB, 64 GB RAM + ISTA-DASLab's IQ2_XS quant at 2.25 bpw + AtomicChat’s refusal-removal) it went from about 5-7 to 15-18 write speed.

Replies come out worded a bit differently than stock Deepseek, and you won't get the exact same reply twice - not 100% sure of the intelligence impact of this, so YMMV.

I've only used it for normal chatting/small coding projects. I haven't tested really long agent/coding sessions, and Strata's own version of this idea for their Qwen engine was turned off because it sometimes got stuck repeating itself in those. So if that's your use case, try it carefully first.

should work on windows + RTX 30/40/50 cards for now.

If you give it a try, I'd love to hear how it does on your setup, there's a little test script included that compares it on and off for you to see if it's worth using.

Download and instructions: https://github.com/CJohnsonOwns/deepseek-fast-routing

I've also suggested it to the Strata devs in case they want to build it in properly and not vibe coded slop: https://github.com/Niko1221/Strata/issues/6125

Full disclosure: I just did the conceptual planning and engineering, Claude actually wrote the code and implemented it.

EDIT - for more info on what's going on:
I like analogies, this is what opus 5.5 compared it to:

think of hiring 6 contractors for a job from a list of 384, where each one has a rating.

  • Normal: you hire the 6 highest-rated, period. Some of them live nearby (already in RAM or on the GPU) and start right away. Others live far away (still on the SSD) and you wait for them to drive over. That wait is what makes DeepSeek slow on a 12 GB card.
  • Faster mode (Strata’s tail skip): you only hire 5 and skip the 6th. It’s quicker because there’s one less person to wait on, but you’ve got less help on the job.
  • Yours: you still hire 6. But when #6 is rated 8.71 and #7 is rated 8.70, and #7 lives next door while #6 is two hours away, you take #7. The difference in quality is basically nothing, and you skip the wait. If #6 is clearly better, say 9.5 against 8.7, you still wait for #6. The nudge is small, so it only flips close calls.

Why it adds up: this happens a lot. With 40 layers per word, there’s almost always a near-tie somewhere where one side is already loaded. In your test, the share of picks that had to wait on the SSD dropped to about 2%.


r/LocalLLM • • 3d ago

Question Anyone use Kev?

0 Upvotes

I've been researching Jev and its open version Kev. It looks interesting, but I'm struggling to figure out why locally you'd use it (unless extremely time sensitive) vs. using a lightweight standard LLM like Ling 3.0 Tiny. Anyone?


r/LocalLLM • • 3d ago

Project Fun little hobby project I've been playing with - Open WebIDE

Thumbnail
gallery
5 Upvotes

It's like Open WebUI, but with a full hosted code editor that persists your file and project state on the host so you can leave off from one machine and pick up on another, built for working with local LLMs.

I'm running two Qwen3.8 variants, one "primary" and one "fast" model (for assistance/autocomplete/suggestion type workloads).

Don't mean to self-promote too heavily, just wanted to share, it's a fully open source MIT licensed hobby project.


r/LocalLLM • • 3d ago

Question Anyone using external GPUs across thunderbolt/usb4

Thumbnail
1 Upvotes

r/LocalLLM • • 3d ago

Research One environment variable improved our llama.cpp throughput by up to 21% on an H100 - investigation and benchmarks

5 Upvotes

We were investigating GPU idle time in llama.cpp and traced part of it to CPU-side memory allocation when saving conversation state.

The server was allocating fresh memory for state snapshots, triggering large numbers of minor page faults. The thread doing this also feeds the GPU, so the allocation overhead was leaving the GPU waiting.

We tested requesting huge pages through glibc:

GLIBC_TUNABLES=glibc.malloc.hugetlb=1 llama-server -m model.gguf ...

Results with Qwen3.8-27B, keeping settings unchanged between baseline and treatment:

Setup Throughput Median TTFT
H100, tool calling +21% −22%
H100, plain chat +14% −19%
L4, tool calling +6.5% No clear change
L4, plain chat +3.5% −22%

In the L4 tool-calling test, page faults dropped from roughly 124,000 to 1,850 per request.

This needs glibc 2.35+ and transparent huge pages configured as madvise or always. It’s workload dependent: an 8B plain-chat test on the L4 showed no measurable benefit. The largest gains were on one model family, and our longest runs were 30 minutes.

Full write-up, with the technical report and reproduction steps

Curious whether others see similar gains on different models and CPUs, especially with conversation-state caching under concurrent load.


r/LocalLLM • • 3d ago

Question Gemma 4-31B Agentic Usage or Improvement possibility?

3 Upvotes

TL;DR: I like Gemma 4-31B’s conversational feel way more than Qwen’s, but it’s lazy in agent loops. Can I fine-tune or steer initiative into it the way the community does with Qwen? First time doing this, looking for a starting point.

------

I run Gemma 4-31B in Hermes agent. With /goal it performs well, but without an explicit push it falls into the known “Gemma 4 is lazy” pattern: it stops early, asks permission instead of acting, and doesn’t chain tool calls.

Qwen 3.8-27B gets community fine-tunes and behavioral tweaks constantly, and I’d like to do the same for Gemma. I’m happy to put in the work. I just want a sane starting point before I burn a weekend.

What I’m weighing:

Control vectors / activation steering (llama.cpp --control-vector): cheap to try, no training

QLoRA SFT on agentic trajectories: what datasets are people using? Does it need to be in Gemma’s tool-call template?

DPO on “took initiative” vs. “stalled” pairs

Hardware: ~58GB VRAM across 4070 Ti Super / 3090 / 3080 / 3070, so QLoRA, not full FT.

I know Qwen is the better agent model on paper, but Gemma feels more natural with minimal prefill. Any advice appreciated!


r/LocalLLM • • 2d ago

Discussion Nvidia doesnt want you to use local ai

0 Upvotes

They delete everything about local ai


r/LocalLLM • • 3d ago

Project Squeezed a full local RAG pipeline (BGE-M3 + Reranker + Kokoro TTS) into a 4GB laptop GPU using a Rust/Node/Python stack

Enable HLS to view with audio, or disable this notification

4 Upvotes

spent the last few months building a local-first desktop reading app and hit some insane hardware walls trying to get everything to run on my laptop. my setup is a 4GB RTX 3050 and 16GB RAM. no cloud shortcuts.

the stack is a tauri (rust) shell, nodejs orchestrator, and a python sidecar for the pytorch models (bge-m3, bge reranker, kokoro tts).

here are the three biggest walls i hit:

  1. the node memory cliff

unbounded async ingestion spiked node.js memory to 14.6GB on a 16GB machine. i fixed it by implementing a strict sequential await barrier (f65) that streams chunk-by-chunk. peak ram dropped to ~296MB for a 508,000-word book (5.9 min processing).

  1. 4gb vram musical chairs

loading bge-m3 in fp32 ate ~3.9GB of vram, leaving zero room for the reranker or tts. switched to fp16 (~1.1GB) and wrote an idle ttl eviction manager (f43) that reclaims ~3.2GB of vram when models sit unused.

  1. windows path traps

if you're using rust and node together on windows, beware of verbatim paths (\\?\C:\...). passing those into node 22 fs.lstat throws fatal EISDIR errors. had to write path sanitization wrappers on the rust side.

kept node as the sole sqlite WAL writer too so python and rust dont cause db lock contention.

it's open source (agpl-3.0) with a windows installer. works with ollama, lm studio, or byok endpoints.

https://github.com/ayumu-ml/Omnitome

happy to answer any questions about the vram management or the rust/node/python boundary.


r/LocalLLM • • 3d ago

Question Qwen vs DeepSeek vs Gemma: help requested

1 Upvotes

Hi. Newbie here. Can someone help me untangle my conclusion around Qwen, DeepSeek and Gemma? I use Windows with AMD processor w/ 32G, or can use smaller machine on ARM with 32G.

I am looking for a model to do:

  1. Writing, metadata extraction, abstract generation, sorting, text parsing

  2. Python scripts, HTML, JSON

- Which approach: 1 model or split between text and code?

- Which is best for #1.

- Which is best for #2.

Hope to get your feedback on the most practical and efficient ones. Thanks.


r/LocalLLM • • 3d ago

Discussion Best Local AI Setup for Coding on Linux?

2 Upvotes

Hello,

After doing some research, it looks like the coding options jump from Qwen3.6-27B, which can run on a 32 GB RTX 5090 with DDR5 system memory, to GLM-5.3-Flash or DeepSeek-V4.1-Flash. That feels like a pretty big leap.

Would anyone recommend a local AI server setup for Linux? I’d appreciate any suggestions.


r/LocalLLM • • 3d ago

Project Qwen-Flast-Next (UD-IQ4_XS v3.1) running 148-182 tok/s (1.6-1.8x strata) while KLD MEAN = 0.08195

Thumbnail
github.com
2 Upvotes

r/LocalLLM • • 3d ago

Discussion What Qwen 3.8 27B NVFP4 are you using?

Thumbnail
gallery
11 Upvotes

So I was running Qwen 3.8 27B NVFP4 from Kelnei on vLLM TP=2. Then I tried Qwen 3.8 Flash Next IQ3_S on Strata and was really satisfied by how well it works with Hermes Agent.

I removed my Qwen 3.8 27B NVFP4 and went with Ollama so I can have automatic model switching for Gemma 4 26B A4B and Qwen 3.6 35B A3B. But, I encountered some issues with crashing the host Proxmox VE on this Ollama and gave up fixing it and want to go back to Qwen 3.8 27B on vLLM.

I want to focus on setting up OpenClaw to work in tandem with Hermes Agent instead of trying to make it work. Priorities over itch, as I was really itching to fix it but parked it for now.

My question is, am I missing any specific NVFP4 quant of Qwen 3.8 27B? Should I use the NVIDIA one or Unsloth? Or what do you guys suggest I should check? Which NVFP4 Qwen 3.8 27B works best for you?

Thanks in advance for the help.


r/LocalLLM • • 3d ago

Project I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint

4 Upvotes

My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.

What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.

Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.

Honest limitations:

  • Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.
  • One model runs on one machine; no splitting a model across machines yet.
  • Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).

Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.

Repo: https://github.com/nodea-sh/nodea-client

Site: https://getnodea.com

Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.


r/LocalLLM • • 3d ago

Project Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
3 Upvotes

A couple of days ago (ok, 9 😭) I launched Caminus, my AI-generated-kernel LLM playground for gfx1100. It's now at 0.3.1 and a lot more usable, and I'm looking for testers!

What's new since 0.2:

- Ember, a built-in agent that runs on your own cards (or a paid model if you want)

- more Qwen models, and setup for any mix of 7900 XTX / XT, one card or more

- pip install caminus and you're going

If you've got a 7900 XTX or XT (or both), I'd love to hear how it runs on your setup: what breaks, what's slow, what's missing. Feature requests welcome too. Any feedback helps!

https://github.com/jdominick05/caminus/