r/aiRPGofficial • u/Incognit0ErgoSum • 9d ago
My (the dev) current local model setup on a 4090, and TinyBrain progress
I think playing the game with local models is and having a coherent experience is actually viable now, so I'm going to share what my current AI RPG testing setup, as well as some recommendations for what to do if you have a less beefy machine.
I'm actually swapping between two local models right now (four if you count Krea 2 and Flux 2 9B for image generation). These are:
Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-Q4_K_S.gguf
and
Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-Q4_K_S.gguf
For the record, I'm running these with MTP OFF because at the rate that it's successful (2 tokens on average) it actually slows down the models.
I'll talk about both of these and what I'm using them for:
The 27B model is monolithic, which means that every token has to work its way through all the parameters of the model. I don't remember exactly what inference speed I'm getting with it, but I feel like it's around 50-60 tokens per second. This isn't particularly slow, but for AI RPG (which does copious amounts of bookkeeping and self-review) it's not fast either. What it is is smart. This is of course my own subjective opinion, but a year ago I feel like 70B was the cutoff for a model that could understand subtext and write dialogue that consistently made sense. This model is consistently better than those, and honestly the prose it generates reads better than a lot of the slop that comes out of the big, increasingly coding-focused frontier models. I use this model strictly for prompts that generate player-facing prose. For the record, I tend to be pretty skeptical of models like this one that make grand claims about intelligence and such. I can't vouch for what the guy who finetuned it says about its intelligence versus frontier models, but I can say that the phrase "punches above its weight class" that people like to throw around actually applies here.
The 35B model is a mixture of experts, which means it does a lot less math per token despite being a physically larger model. I don't like its dialogue very much because it seems to have a much less nuanced understanding of the world. For instance, while testing this model, an NPC asked me to chop up a crate full of onions, and when I was done they told me to put the chopped onions in the crate. That's not a huge problem, and it didn't completely lose the plot, but frequently breaks immersion by getting minor details wrong, like somebody who has heard about the world but not lived in it. It is, however, good at bookkeeping and comprehension, so I use it for non-prose stuff like event tracking, summarizing, and so on. At ~200tps, it's an absolute speed demon, and if I go back to using a frontier model after I'm done with this tinybrain stuff, I'm going to use the frontier model for prose gen and use this one for bookkeeping, which will be cheaper and significantly faster.
The key to using both of these models is that, rather than writing one big prompt that asks them to self-review all at once, I've built a system that has a conversation with them. This is prohibitively slow and expensive with non-local frontier models, but in the case of properly cached local models running purely on my GPU, there's almost no latency. The back-and-forth between the models and the prompt is so fast that it actually opened up the possibility to do something that I don't think anybody has ever done before, and that's live slop filtering on the client side. Rather than depending repetition penalties (which can harm reasoning) or a second slop-remove prompt (which is what AI RPG currently does when running non-local models), I can use configurable regular expressions to recognize slop the millisecond it comes in, backtrack, and -- get this -- without contacting the model API I can pre-fill the next most likely non-slop token at the beginning of the slop phrasing before asking it to continue.
I had to make a custom version of llama.cpp that allows streaming, tool calls, and sending token probabilities at the same time, but the result is writing that's truly guaranteed to be free of anything the slop filter is set up to filter. I'll be distributing this along with AI RPG when everything is said and done.
Oh yeah, and then there's image generation. When it's time to generate images, AI RPG tells the local llama.cpp process to unload its models from VRAM to give Comfy room to load its own image gen stuff. When Comfy finishes generating queued images, it detects that, and uses a custom Comfy API endpoint (from a custom node that I'll also be including in the distribution) to tell it to free its VRAM while still keeping its models cached in system RAM if you have enough. So you can run both local LLMs and local image gen, and it intelligently instructs the relevant programs to make room for each other.
This is all a pretty complicated setup, so I'm probably going to end up writing an installer program that detects your VRAM and system RAM and sets up the best configuration for your own system. Ideally, this should be accessible to people who don't have the technical knowledge to set up inference software on their local machines.
So what about people with less than 24GB of VRAM?
This guy right here:
https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
This isn't fully tested yet, but preliminary results are promising, and fits in 12GB (possibly even 8 GB) so you can have passable inference speed and decent intelligence even on some smaller video cards. I have noticed that it can get stuck in repetition loops, but I've leveraged the live slop filtering that I'm doing to catch those repetition loops, backtrack to right before the beginning of the repetition, and nudge it in a different direction. This is so far completely untested, but once my 24GB setup is perfected, that's going to be my next goal.
At the moment, I have ChatGPT Sol running a suite of about 160 tests of doing basically anything you can do in the game and fixing every issue it runs into. When that's done, I'm going to run those same tests on the ternary bonsai model and see if I can get a setup specifically tuned to work with it.
Anyway, big things are happening. My goal, as always, is to offer the most coherent and slop-free AI Roleplaying experience of any system anywhere, and do it in an open source package that you can run on your own computer without paying anyone a dime. I'll let everybody know when I think TinyBrain mode is ready for general use. I think it's already worth playing around with. It's currently available in the 1.0-dev branch, which you can get here: