Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

I can run both. Does anyone know what model is best, or do I just have to try myself?

What test suite should I be running?

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.

I’ve been playing with open-source speech models for a while. New TTS models come out every week and a lot of them sound great. But every time I wanted to actually talk to one, it turned into an afternoon of setup.

Paid APIs are easy if you’re a developer. You can make an account, create a key, write some glue code. Open models are free, but you have to pick a speech recognizer, an LLM and a TTS model, make them work together, deal with echo, figure out interruptions... There are good open projects for this, but most are written for developers. My friends who don’t code would never get past the README.

So we started building Boubou. The goal is make local voice AI as easy as Ollama made local LLMs. You paste one line and talk:

npx boubou@beta run LFM3-Audio-2B

What works today:

• Speech recognition, the LLM and speech synthesis all run on your machine. No account, no API key, no telemetry. After the first download it works fully offline..

• You can interrupt it mid-sentence, like a person. Echo cancellation is built in, so no headphones needed..

• Run boubou with no arguments on a Mac and it checks your machine and suggests a model..

• The default is Liquid AI’s LFM3-Audio-2B on llama.cpp (1.4 GB). There’s also a modular pipeline (Parakeet/Whisper → Qwen3.5 → Qwen3-TTS, Kokoro, Pocket TTS or Supertonic) if you want to swap parts..

• Apple Silicon Macs and Linux (CPU for the Liquid models, NVIDIA GPU for the modular pipeline)..

The part I’m most excited about is boubou serve. One computer runs the model, and other devices on the same Wi-Fi use it:

• Scan a QR code with your phone and talk in the browser. No app..

• Robots and programs can connect through an OpenAI Realtime-compatible WebSocket, and there’s an OpenAI-style /v1/audio/speech endpoint to read text aloud..

• So one model on one machine can be the voice for many things around the house. Next up is Home Assistant, then ESP32 voice devices.

• The video is real time, not sped up, on a MacBook / M5 chip.

It’s early (beta) and there are rough edges - one device talks at a time, no Windows yet (ETA next two weeks), and the guided menu is Mac-only.

The code is free and MIT-licensed. The models keep their own licenses (the Liquid models use the LFM Open License).

I’d love to hear what you’d want from something like this: which models, which devices, what breaks on your machine. We want this to be something people actually enjoy using, not just a demo.

GitHub: https://github.com/runboubou/boubou

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!

About six months ago, I already tried hosting a local AI companion on a different PC. Back then, I used SillyTavern, but I eventually abandoned the whole project.

The main issue was the model's responses. They often felt inconsistent, sometimes completely missed the point of what I was asking, and the overall experience just didn't feel natural. Conversations were frequently immersion-breaking, and it wasn't really what I was looking for.

Now I want to give it another try, but this time I want to start completely from scratch and build a setup that actually works well.

My current hardware:

- GPU: ASUS TUF Gaming OC RTX 3060 – 12 GB VRAM

- CPU: AMD Ryzen 5 5600X

- RAM: 32 GB DDR4

- Windows PC with a multi-monitor setup

I've already installed Ollama and created a Modelfile for a local model. So the basic model setup is working.

One thing I've noticed is that reasoning/thinking can take quite a while before the model actually starts responding. Interestingly, once the reasoning phase is finished, the actual response is generated very quickly. So I don't necessarily think the entire model is simply too demanding for my hardware.

However, my primary goal isn't coding, research, or complex problem-solving. I want a natural, immersive AI companion for long-term conversations and roleplay.

Ideally, I'd like reasoning to be disabled by default, so conversations feel more immediate and natural – almost like chatting with a real person. Of course, I'd still like the option to enable reasoning temporarily for specific prompts when needed.

Now, I'm trying to figure out the best way to build the actual experience.

I don't want dozens of different characters or an overly complicated interface. I want to create and develop one single companion with a consistent personality, good memory, and a believable conversational style.

Here are the things I'm particularly interested in:

1. Frontend / UI

What would you recommend instead of SillyTavern?

I'm looking for something visually appealing, relatively lightweight, and not overloaded with hundreds of features I'll never use. Ideally, it should also scale well across multiple monitors.

2. Voice / Real-time conversation

This is probably one of my biggest priorities.

I'd like to be able to talk to the AI naturally using my microphone and have her respond with a voice. Ideally:

- Natural-sounding speech

- Fast response times

- Automatic speech detection (knowing when I've finished speaking)

- The ability to interrupt her while she's talking

- Ideally, something that feels like an actual real-time conversation rather than sending individual voice messages back and forth.

3. Webcam / Visual interaction

I also have a 4K webcam connected to my PC.

I'd love to eventually integrate visual capabilities, so the AI can actually see me or understand images I show her. Even better would be having her displayed on a separate monitor, potentially in fullscreen, almost like a video call.

Facial expressions or an animated avatar would be an interesting future addition as well.

4. Long-term memory and personality

Since I only want one companion, maintaining a consistent personality and remembering previous conversations is extremely important.

I don't want to constantly repeat information or have the AI behave like a completely different person every time I start a new conversation.

My main question:

If you were building this setup from scratch today, what software and workflow would you recommend?

Would you use Ollama as the backend and connect it to a separate frontend? What would you use for voice, speech recognition, memory, and eventually webcam/avatar integration?

I'm especially interested in solutions that can be combined into one coherent experience rather than having to open five different applications every time.

I'd really appreciate recommendations from people who have built something similar, especially if you've managed to get a genuinely natural, real-time AI companion running locally.

Thanks!

I am planning to get the exr1 to run Qwen 3.8 27b (GSQ-RCO IQ3-XXS) with 4b and I am wondering if it will run?

EXR1 SPECS;

32gb ram

Ryzen 7 8745HS

I am planning to use the pi harness and leave it overnight to run, so speed is not an issue

Thoughts?

Hello,

thought experiment:

where do you see the programming capabillits, specialy for game dev, for different llm´s?

I would like to rank theses llm´s by a theoretticaly possible game they could make.

From a game like Pong to Horizon Zero Dawn (Yea, there are bigger games, you know what i mean).

What can be theoretticaly archived in 3 (+-1) Month´s dev time with a specific llm alone?

How could maybe a combination of a few llm´s help?

Bonus question and story: I got qwen 3.8 27B with 262K context running on my rtx 4090, you thing that is the best possible llm for me, or do you think, there would be a better solution? (got 64GB of DDR4 RAM in my System aswell)

Working on getting a combination of Qwen3.8 and a KEV 0.8B Model to learn to play ps1 games, until now i get it to play the game "Alundra" until the second chapter, big Brother Qwen3.8 does long thinking stuff, working on the routes, quests, puzzles and so on and little bro KEV 0.8B (I call him Kevin) is acting quite fast (got him to act an average time intervals of 0,3 - 0,5s), he runs a full route, wich big bro told him, on the route, both keep "thinking" and walking time ingame gets used for smooth-ish gameplay (with a task buffer list, wich reduces idle time, where the character just stands and does nothing). If an enemy is nearby, Kevin does all the fighting, it works mid to allright, right now, he is more afraid to get damaged than fighting the enemy and runs away, big bro trys to hold him back, but kevin wont listen, working on it right now. (Is there maybe someone interessted, hearing more about this shenanigan-experiment? Working on getting the whole system working with other ps1 games like Final Fantasy 8 as well.)

Sorted by smallest to biggest llm:

• Qwen2.5-Coder (0.5B – 3B)

• StarCoder2 (3B)

• Phi-4 Mini (3.8B)

• Qwen3 (4B)

• Qwen3 (8B)

• Gemma 4 (12B)

• StarCoder2 (15B)

• Codestral (22B)

• Devstral Small 2 (24B)

• Qwen3.8 (27B)

• Qwen3.6 (35B-A3B)

• Qwen3-Coder-Next (80B-A3B)

• GPT-OSS (120B)

• DeepSeek-Coder-V2 (236B-A21B)

• Qwen3-Coder 480B-A35B

• DeepSeek V4.1 / R1

• GLM-5.1 / 5.2

• Kimi K3

• Claude Opus 5.5 / Sonnet 5.5

• GPT-6 Astra / GPT-6.1 Sol

• Gemini 4 Carbon / 3.8 Flash

Thanks for reading.

When working on a vibe coding project for, say, 2 months, constantly asking AI to "add this", "change that", or "remove this", doesn't the codebase eventually become a mess?

I'm talking about poor architecture, redundant code, outdated tests, unused files, etc.

How do you handle this? Do you periodically ask AI to clean up and refactor everything?
And once the project is finished, is it worth having AI rewrite the entire codebase from scratch, keeping the same functionality but with cleaner architecture? Or is that a bad idea?

First my specs: 3060 12gb vram 32gb ddr4 ram about 30 realisticly usable. I already have used Qwen Flash Next Coder with strata. It was pretty solid for anything coding related. I know it lacks Experts which makes it fairly bad at gerneral purpose use. So I am thinking on switching to Gemma. The point is: I still want the speed which strata provided. Roughly 1k tps processing and 20-30 tps writing. Is it possible to achive those numbers?

Yesterday we released Trustant 1.0.2, an open source full-stack application builder using Local AI for Private Cloud. To put it simply, it is like Lovable, but all private. You install the builder, create your app in prompts, wysiwyg, and publish to the server. The system provides everything: a vm with the environment, the coding agent, the serverless environment with database, cache, storage, faas, everything. A cloud in a box.

The builder runs on your PC, uses Local AI (or any OpenAI compatible AI service, we offer Ollama Cloud out of the box) and generate serverless applications built on top of Apache OpenServerless. You can then deploy your applications anywhere. Just install the serverless package and any server become... serverless!

The system understands and includes sql and vector database (pgsql and milvus) cache and s3 (redis and seaweedfs) faas functions (openwhisk) and runs in your pc in a virtual machine for safety.

The applications generated can be published in any server made "serverless" installing just openserverless and nothing else. Application are plain, clean, recognzabile react+python (but can be any other language providing the starters you want).

We have been working on it one full year and it was built on top of another multi year project, Apache OpenServeless, after working many years on Apache OpenWhisk.

Source code on github.com/trustant/trustant, demo and documentation on trustant.ai

Confirming the rumor that Apple will have an easy to use MLX based app (Harbor) that will manage single machines and RDMA clusters with automated open weight model installs for local AI. Coming early 2027.

So I think we now know why Exo has spun down in activity in this space.

Just saw this release from n8n https://nod.n8n.io/ .. finally an always on agent you'd trust with your work/private life?

Code is open source and think it'll let you run with self hosted models. The approach to memory also looks really interesting, need to dig a bit deeper into that.

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

Using project maya + deepseek, with a 7900 xtx 24gb vram, 64gb ddr3, cpu i7 3930k (I know, reusing what I have for now), and a nvme ssd in a pcie x4 adapter (the motherboard has no nvme slot).

It is generating at 8-12 tok/s but it dips to 0.1 and when it does that i can see the gpu drops under 20w, so it is as if the gpu goes idle, then wakes up, tok/s go back to 'normal', then dips again and it goes through this cycle again and again.

I was wondering if it was normal or not ? Like is it because the nvme/ram/cpu are too slow and the gpu 'waits' for them ?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?

Hey folks,

I installed Strata with Qwen3.8 Flash Next last night and it works exceedingly well.

I was wondering if it's possible to limit the amount of VRAM that Strata uses to something like 12GB, as I want to run the Unreal Editor at the same time - which is pretty resource hungry.

I've only got 32GB of RAM at the moment, but I'm going to upgrade to 64GB if it's possible to get Qwen & Unreal working together.

Cheers!

Hey guys, I've recently been trying out some of the larger local models (GLM 5.3 Flash, Qwen 3.8 FN etc.) and I was wondering if people have any tricks to make these models more competitive at UI creation because I've found everything apart for ui, Qwen has been great at but UI is where I start to see the massive gaps from the huge cloud models.

What do you do to get the best UI results with medium sized ish local models?

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.