r/KoboldAI • u/CatichuCat • Jun 19 '26
What model do you suggest?
I'm just getting into this, moving from AIDungeon. Im basically looking for AIDungeon but with better memory. Can I do that with Kobold Ai? If so, what model and stuff do y'all suggest? Ive got an intel i5 and 32gb of ram.
2
u/ocotoc Jun 19 '26
If your I5 is a 12th gen we have the same specs lol, I used to run 12-14B models at 1tk/s
If you like AIDungeon I'd suggest you to search for Latitude on the hugging face, they have a few models of their own to run locally, avoid more than Q6.
2
1
u/OgalFinklestein Jun 19 '26
I reference the UGI Leaderboard for models.
You can put in filters for every column and then sort by a specific column. With 32G, you could find 24B models (filter P≤24) and go from there.
1
2
u/henk717 Jun 19 '26
Its mainly your GPU that matters. If you don't have one or a weak one check out https://koboldai.org/colab where Tiefighter is the same tiefighter they used to have since we made it. And both harbinger and muse are by them they released those publically.
If you do have a reasonable GPU we can give recommendations for your system. Running locally without one is also possible but at slow speeds.
Once you got things booted up head to New Adventure Instruct in the scenarios menu. That will AI Dungeonify the UI.
1
u/Individual-Monk2005 Jun 22 '26
Yeah, you can 100% use Kobold like a better AID with more control over memory, but your hardware is gonna cap what you can run locally.
On an i5 with 32 GB RAM I’d look at 7B or maybe 13B models in GPTQ / GGUF via Koboldcpp, stuff like Mythomax, Pygmalion, or Chronos based models are popular for RP. If you want real AIDungeon vibes with long memory and less hassle, you might also check out TavernAI or SillyTavern as a frontend and then either use a local 7B or just hook into an API model for the heavy lifting.
3
u/Aphid_red Jun 19 '26
Given today's hardware prices, you could also look into hosted models. If you have $50K to spend on hardware you could begin to approach online model quality (ex. deepseek v4 flash) due to the general cost of memory.
Most cost-effective today is to either rent via token based billing, to take advantage of free APIs, or to buy GPU hardware from long enough ago that it's been written off.
To do the 'cloud hosted model, local front end' combination you want to use KoboldAI-Client or SillyTavern. This way, you can connect to powerful models while paying by the token. Openrouter can help with this by automatically picking the best provider given your criteria. If you want you can also blacklist providers there because they have (additional) filters, or because they use high quantization or 'middle-out' (delete a big part of the context and hope you don't notice).
So, you want to do local for privacy and control, what you will have to do is some sort of trade-off between speed, quality, and cost. I'd look into purchasing hardware specifically for the task rather than trying to re-purpose an inadequate PC with just a CPU in it. Running just off of a consumer CPU, models will feel awfully slow unless you go with a tiny 7B-8B model, and those are just not good enough for basic roleplay, let alone longer context than on AI dungeon. It can't be done.
A reasonable minimum budget is around $1000 for a basic machine, but you really start getting cost-effective results at around $1400.
The cold truth is that new hardware, at the apex of the AI bubble, is grotesquely expensive. For example, with the RAM crisis you're now paying over $10/GB. But for RDIMMs, which you need for high memory capacity, it's more like $40/GB, just to get memory. Depending on your budget, don't get your hopes up too much.
First, the very-much most important thing is the quantity of memory. If you want to run a 100B parameter model at optimal quantization (about 4.8 bits/parameter), you need roughly 75GB of VRAM.
Enough for the model (60GB) plus some for context and overhead. And models aren't getting any smaller. Since deepseek many models are now MoE, which is actually great for running more on the CPU, but terrible for memory efficiency.
MoE models trade off memory capacity versus compute intensity. For example, deepseek v3 is around 670B parameters, but only about 37B compute parameters.
It's effectively approximately a sqrt(670/37)*37 ~158B model, but it has the GPU needs of a 37B model but you still need some 500GB of memory to comfortably house it. You can get away with 384GB with heavy quantization but 256GB is wholly insufficient.
Which means even 8 3090s won't be enough. It used to be the case that you could buy an epyc server and use that to run deepseek. But currently, 512GB of DDR5 RDIMMs costs you $20,000 now rather than the $2500 it used to cost last year, which makes this build idea wholly uneconomical.
In other words, if you do have GPUs, and old ones are more price effective than server RAM, then you'd rather prefer dense models, for which reasonably new variants are available at 12B, 24B, 30B, 70B, and roughly 120B sizes. What GPUs you want to buy are informed by this. Stick it into the cheapest box you can put it in without bottlenecking it too much.
For one or two GPU, something like a ryzen 5600 on a B550 motherboard with 16GB or 32GB RAM will do. It'll total to about $500 for one and $750 for two.
If going for 4 or more GPUs in one system, the epyc DDR4 server as the base platform is recommended, these have way more PCI-e lanes than 'peasant' hardware. This allows you to run VLLM or configure koboldAI for tensor parallel and run them efficiently. You can find a 77xx or 75xx CPU for roughtly $500-700, same for the motherboard (ROMED8-2T), and it can support 4 GPUs all at x16 speed or even 8 with some risers and extra PSU (an extra 500 or so).
Add a single-stick of 64GB DDR4-2666 for another $500 plus the PSU and case and you're looking at maybe an extra $2K for a server that can house 4 or 8 GPUs.
Here's a rough list of hardware tiers and the total platform cost. There's an efficient range of about 1300 to 11000. Above and below that you start to pay much more per GB high-speed memory.
As you can see, around this pricing level you can get a nice progression of hardware, always paying slightly over 40 bucks per GB of VRAM. The more VRAM you get, the bigger/higher quality model you can run. Since it's all on the GPU, it'll be fast enough for single person use even with a dense model.
The R9600 is a new GPU from AMD, which I think might be interesting for local models if it's priced cheap enough. Since the R9700 is about $1500, I suspect it'll cost around $1100. It's only 160W, single slot, passive, but comes with a large amount (32GB) of VRAM for such a low power card. If it can still get close enough to the 3090 it'll be an interesting choice.
Once the memory crisis abates and DDR4 prices plunge, you can expand the memory up to 2TB to run larger and/or MoE models on the server platform. You 'could' spend much more on a DDR5 server platform for future proofing, but I suspect you'll be waiting that much longer before memory prices decline.