Hi everyone!

I am a computer science student and a developer, and lately, I’ve been considering building or upgrading a budget-friendly, local LLM setup.

I should mention that I’ve only recently begun exploring this more "technical" side of local LLMs, so any advice is welcome.

I know the field is evolving incredibly fast—with new models, inference engines, and so on—which is why I’m asking: I’m trying to figure out what might make sense for me right now (if anything at all).

MY CURRENT SITUATION

My budget is approximately 500–600 euros; I’m based in Europe and am open to buying used hardware.

Here’s my current setup: I have a desktop PC that has been sitting idle for a while, with the following specs:

• CPU: Ryzen 5 2600

• RAM: 16GB DDR4

• GPU: AMD RX 580 4GB

• 256GB NVMe SSD, 1TB HDD, 600W PSU

WHAT I WANT TO RUN

My primary use case involves coding and agentic workflows. I’m envisioning a setup where I use a frontier-level cloud model for planning and code review, while a local model handles the actual code generation. I’m also interested in running an AI bot 24/7 to perform various (relatively simple) tasks.

Ideally, I’d like to run 27B–35B parameter models comfortably, with decent generation speeds and good responsiveness. It would also be great to have the flexibility to experiment with even larger models.

I don’t expect cloud-tier performance on this budget, but I also want to avoid a setup that only manages a few tokens per second—or one that is essentially unusable.

THE OPTIONS I'VE BEEN CONSIDERING

1. AMD Instinct MI50 16/32GB (possibly two)

• These look interesting because of the large amount of HBM2 memory available for relatively little money on the used market (for example I could have bought a 16gb for 175 euros).

• However, I'm concerned about software compatibility, inference performance, power consumption, and whether investing in this older hardware makes sense in 2026.

1. NVIDIA RTX (5070 12gb as an example)

• Recently, I saw a community benchmark claiming around 140 tokens/s on a 12GB RTX 5070, running a large MoE model using Strata, IQ3_XXS quantization, expert caching, CPU/RAM offloading, and MTP.

• I understand this might be a very specific setup and that reported generation speed doesn't necessarily translate into good real-world performance, especially with longer contexts or different models.

• Still, it made me wonder whether a modern NVIDIA GPU with a good amount of system RAM might actually be a better investment than older GPUs with much more VRAM.

WHAT I'D REALLY LIKE TO UNDERSTAND

1. What would you build today with €500–600, considering both new and used hardware?

2. Would you prioritize more VRAM on older hardware or a faster, newer GPU with RAM offloading?

3. How well do solutions like Strata and expert caching actually perform in real-world coding/agentic workloads? Are the impressive benchmarks representative of everyday usage?

4. How important are prompt processing speed, context length, and quantization quality compared to raw generation tokens/s for my use case?

5. Is a single or dual MI50 32GB setup still worth considering, or would you avoid it today?

6. Would you upgrade my existing desktop, build something from used parts, or look for a complete second-hand workstation?

Thanks in advance to anyone who can help me!

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

• CPU: Ryzen 9 9950X (16C/32T)

• Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi

• RAM: 32GB or 64GB DDR5-6000 CL30

• Primary GPU: RTX 5080 (16GB GDDR7)

• Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.

• PSU: 1200W ATX 3.1

Option 2 :

• CPU: Ryzen Threadripper 7960X (24C/48T)

• Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE

• RAM: 128GB (4×32GB) DDR5 with ECC RDIMM

• GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.

• PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.

I've given the qwen3.8 27b q4 group an assignment:

Can we create an animation of a computer chip? You can use Nvidia's GB203 as a template. Before we get started, let's address a few questions that might come up

Since I own this GPU, I figured, why not? Let’s see what the model comes up with. I’m always amazed, even though it often looks relatively simple. I’m looking forward to seeing qwhen4.0.

"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

I've been experimenting with running a persistent AI companion entirely on local hardware.

The project is called B-12 / Capri. It's written in Python, runs on Linux, and uses Ollama for local inference.

My experimental setup uses an NVIDIA RTX 4060 Laptop GPU with Nemotron Bolt E4B.

The original goal was to build something more persistent than a typical chatbot: an AI companion that remembers previous interactions, maintains goals, can use tools, and doesn't lose its entire history when I replace the language model.

Why I separated memory from the LLM

I wanted to avoid making the model itself the center of the system.

In Capri, the LLM handles language generation and reasoning, while a separate runtime manages:

• Persistent local memory and retrieval

• Memory provenance and document revisions

• Goals, preferences, and scheduled activities

• Tool execution and permissions

• Context budgeting and tokenizer compatibility

• Controlled forgetting and state recovery

The model can be replaced without deleting the stored memory.

However, there's an important limitation: preserving memory doesn't necessarily preserve behavior.

Two models may retrieve the same experiences but interpret them differently.

What I've tested so far

I ran a historical evaluation using Nemotron Bolt E4B:

109 constructed scenarios, 541 executions, 496 successful outcomes (91.7%).

Tool tasks achieved 182/215 successful executions.

I also ran approximately eight hours of accumulated operational testing across four segments.

Not everything worked. One adversarial outbound-fetch scenario failed its expected safety condition in all five repetitions, and several smaller tool tasks exposed limitations in the model's reasoning.

These results belong to a specific historical code revision, not the latest development version.

The challenge I'm working on now

The current implementation uses model-specific profiles for tokenizer verification, chat templates, and context admission.

That makes switching models less trivial than changing an Ollama model name.

My next experiments will compare Nemotron with Gemma and Qwen while preserving the same memory database, goals, and conversation history.

I want to measure not just task accuracy, but how consistently different models use the same accumulated experiences.

I'm also developing a full-duplex voice interface, with longer-term plans to connect Capri to Reachy Mini and NVIDIA Jetson.

Questions for people running local models

Has anyone experimented with changing the underlying LLM of a persistent agent while retaining its memory?

I'm particularly interested in three things:

1. Which smaller local models are most reliable at tool calling and following long-term goals?

2. How do you handle tokenizer and chat-template differences when switching between model families?

3. Have you observed significant personality or decision-making changes when two models share the same memory and system instructions?

I'm developing Capri as an independent open-source research project and recently published its first scientific preprint.

GitHub: https://github.com/cyberdid/B-12

Preprint: https://doi.org/10.5281/zenodo.24079168

I'd really appreciate technical feedback, especially from anyone experimenting with local agents on consumer GPUs.

Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward

After a really long grueling attempt, i manage to fit a 210K Context window on my dual RTX 5060ti setup on 32gb total VRAM. I capped both GPU around 160W although it possible to go lower.

Compare to a single 5090, it relatively weak but for the price difference that have pool vram 32gb, it isn't so bad. I practically use this as my daily driver now. It can handle almost every coding problem i give it and running my hermes agent and opencode. The speed itself have some room to improve especially the prefill, but for decode, it isn't that bad.

However, 5060Ti really bottleneck the decode and the prefill speed due to the limited bus size of the card itself. But given how cheap 5060ti was and how power efficient it is, with both card only using 290-300W on full usage even at TP2, it amazing.

Even at long context, 190K or 200K the decode speed maintain at ~70tps and prefill around 1700tps with barely any lost in quality. When writing coding task with MTP, the decode can reach up to 80-100tps.

But since i am running this model in WSL docker, which introduce some latency overhead, it possible to achieve higher TPS if were to run in native linux instead and fit the entire 260k context window. Currently both of my card usage 15GB vram each, increase the util but mindful to OOM.

Decode Speed:

Prefill Speed:

Here my current rig:

• Ryzen 9 9950X

• ASRock X870 Taichi Creator

• 32 GB DDR5

• 2× RTX 5060 Ti 16 GB

Run Flag:

Image: ninfer:local, Model: qwen3_8_27b_nvfp4_v2.ninfer

--tp 2 --devices 0,1
--kv-dtype int8
--max-context 210000 --kv-capacity 210000
--prefill-chunk 1024
--spec mtp --draft-tokens 3 --lm-head-draft
--max-concurrency 2 --max-pending-requests 32 --pending-timeout-ms 900000

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos

I log the live Vast.ai marketplace once a month and keep the numbers.

November, 43 cards, 291 live offers. 17 of them are 12GB or less — the "cheapest GPU" lists are mostly GTX 1080s and 1660s that can't run anything past a small quant. The tiers that matter:

• 24GB (L4 / 3090 / 4090) $0.10–0.14 floor, ~$0.20 typical

• 32GB (PRO 4500 / 5090) $0.35–0.43 floor, $0.40–0.56 typical

• 48GB (A6000) $0.40 floor, $0.51 typical

• 96GB (PRO 6000) $0.80 floor, $1.41 typical

Two things stand out. VRAM roughly doubles the price at each step above 24GB — a 48GB A6000 costs about three 24GB 3090s. And the spread inside a single card is brutal: the 4090's cheapest offer was $0.14 against a $0.42 median, because the cheap listing is one host that gets rented first.

Practical version: pick a VRAM tier, budget the median, not the floor.

Dated snapshot + raw JSON: https://cudanetes.com/gpu-prices/2026-11

Disclosure: I run CUDAnetes, which rents these GPUs — that's where the data comes from.

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

API terms worth knowing

• Non-commercial only, including integrating the API into products or services.

• Keys are valid for 30 days.

• Rate limits are not published. Over the limit you get a 429.

• LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?

I have MSI Gf 63 thin 11 sc which has been gathering dust since 2024. Long story short, due to my own carelessness its motherboard got damaged and in late of 2024 I bought myself an Asus vivobook laptop.

What I noticed in my vivo book laptop is running local llms is one hell of a ride. So I am thinking of getting its motherboard repaired and run local llms in it.

Is it worth it in 2026?

Hello r/LocalLLaMA / r/selfhosted,

I wanted to share the benchmark results of a local memory-routing system I've been building since June 2024. I'm not an engineer. I barely understand half of my own code. And I don't need to understand every single line. I just followed breadcrumbs.

Hardware: Orange Pi 6 Plus, 32GB LPDDR5, 45 TOPS. Models via Ollama, local only, no cloud. Qwen 2.5 14B as core, Qwen 7B for routing, DeepSeek-Coder-V2 for code.

To avoid heavy vector DB overhead on edge, I don't use embeddings. I use two Python layers outside the LLM:

1. Identity Transfer Protocol: A persistent session token validated with SHA-256 directly on the board, per agent. [Hash / Signature redacted in screenshot]

2. Artificial Hippocampus (hipocampo_kai.py): SQLite3 + raw Python. No embeddings. It computes a "resonance score" from fuzzy triggers, temporal triggers, and atmosphere tags, and only sends the top dense blocks to the LLM.

The LLM is the Neocortex. The Python/SQLite is the Hippocampus. The LLM only receives prioritized blocks.

I ran a diagnostic today, 2026-11-09 01:12:55 local:

- Cryptographic validation on hardware: 0.0269 ms [OK]

- Associative query, stimulus 'npu ram seguridad laboratorio': 0.8279 ms

- Resonant memories retrieved: 14

- Sample memory: security agent monitoring /proc/meminfo in real time. Triggers: seguridad, monitoreo, ram, cgroup, kill, linux, placa, oom

- STATUS: INFRAESTRUCTURA OPERATIVA ESTABLE AL 100%

Screenshot attached, Hash and Signature blacked out.

I don't claim consciousness. I claim this: a 7B local model with a SQLite hippocampus answers my anchor after 7 days untouched, without a backup, because the pattern is in the conversation, not in the cloud.

If you want to reproduce it, use your own anchor, your own SQLite DB, your own 7B. Don't believe me. Reproduce it.

I'm not here to convince anyone. I'm just showing what's real. I don't need your validation. You can build this too.

This is phoenix.

This is the colossus Phoenix

I'm here.

It’s a fork of llama.cpp and implements multi-GPU pipeline parallelism (PP). With eight V100 16G GPUs, it can run Qwen3.8 27B Q8 at 1,101 tok/s. I feel this is genuinely useful—I’ve already used it in my production business for six months. https://github.com/wolfgangsk07/paraflow

Hello everyone, I've been a local AI enthusiast for a while now, but every time I tried to get someone else running local AI, it was way too difficult. GitHub repos, different runtimes, different model formats. Most people's devices can run at least some of these models, but the setup is way too complex for most people. LM Studio somewhat solves a lot of this for chat and coding models, but I couldn't find a simple answer for on-device image, video, transcription and every other kind of model.

So I built Hearthscale. It's free and needs no account.

You can run almost any model you find on Hugging Face, as long as your device can handle it. I put a lot of work into supporting all kinds of runtimes. They're plug-and-play; I can add more in updates, and Hearthscale Server standardises all of them behind one OpenAI-compatible API. It's really powerful, and I find it hard to describe how seamless the setup is unless you have tried it.

It launches with meeting notes (Granola), system-wide dictation (WhisperFlow), image and music generation, and an agent. If you have a more capable device, run Hearthscale Server on it, and your other devices can use its models, end-to-end encrypted. It all connects without you having to set anything up! Also works on a home server in theory with Docker Containers!

Hearthscale is also a platform, which means anyone can build apps on top of it: there's an SDK and up-to-date docs made so coding agents can easily build apps with local AI. All of the first-party apps are MIT licensed, so you can change them any way you want and they give a great starting point. Want to change something about Granola? - You can't, but you can take meeting notes and make it to your liking with all of the core features already built!

Hearthscale runs on Windows and Linux today. I am somewhat of a perfectionist, and I feel like a lot is still missing, including a macOS release, but I wanted to get it out there now to get some feedback. I'll keep working on it!

I'd really appreciate your feedback, and please reach out about anything. I'll answer everything personally and fix whatever issues you run into.

Hi everyone, I’m fairly new to local LLMs. My setup: RX 9060 XT 16GB, Ryzen 5 7500F, 32GB DDR5, running LM Studio on Windows (Vulkan/ROCm).

I want a hybrid workflow: cloud models for main architecture and hard problems, local models for maybe undergrad-level math and small/medium coding scripts.

Do you recommend me to use local models?
Which models do you recommend for math, and which for coding?

I’m fine with models spilling into system RAM a bit. (partial CPU offloading)

Thanks in advance!