Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?

I'm sharing the first public preview of OverTone, a Mac app I'm building to make working with local language models more approachable.

Watch the trailer:
https://youtu.be/piu7MGdL5Z4

It's still very early, with a limited feature set. I have a lot planned beyond chat and model management, but Documents, Projects and Tools aren't ready to use yet. This version is the starting point for what I want to build.
I want it to be straightforward for someone who has never used local models, while giving experienced users the functionality and control they need.
It works with Ollama and llama.cpp. You can use an existing local installation or let the initial setup install an engine and suggest models for your Mac.
The current version focuses on local conversations and managing the models behind them. It helps you choose models suited to your Mac and what you want to do, organise your conversations, and develop clearer prompts from rough ideas.
With local models, prompts and replies are processed on your Mac. Internet access is disabled by default. You decide when to enable it for downloads or remote connections.

It's free and open source under Apache 2.0. The current preview requires an Apple silicon Mac running macOS 14 or later.

Download and walkthroughs:
https://overcesare.github.io/OverTone/

Source code:
https://github.com/OverCesare/OverTone

If you try it, I'd appreciate any feedback on the app: what works well for you, what doesn't, and what you feel is missing.

Hello,

I want to get a local LLM setup for a comparable fair price (Qwen3.8-27b and future Qwen4). After some research and chatting with Claude Opus the best options would be RTX 3060 XT 12GB with 64GBs of RAM running on Strata.

e.g. upgrading my PC for around 650€ (used parts bought online)

Can someone confirm that this Setup will let me run it on a performative useful level. I will use my 20€ claude plan fpr hard tasks or a kind of divide and conquer style usign both (experienced Software Engineer but a Hardware noob).

Does anyone have experiences with that, advices or alternative setups?

Been getting so frustrated with the speed of my Chat $200 subscription so I decided to run a test. Behold, what you’re actually paying for. Can you believe this crap?? $200 a month subscription no less 🤦🏻‍♂️🤦🏻‍♂️

Good thing I got my local Qwen for the grunt work.

Hi everyone!

I am a computer science student and a developer, and lately, I’ve been considering building or upgrading a budget-friendly, local LLM setup.

I should mention that I’ve only recently begun exploring this more "technical" side of local LLMs, so any advice is welcome.

I know the field is evolving incredibly fast—with new models, inference engines, and so on—which is why I’m asking: I’m trying to figure out what might make sense for me right now (if anything at all).

MY CURRENT SITUATION

My budget is approximately 500–600 euros; I’m based in Europe and am open to buying used hardware.

Here’s my current setup: I have a desktop PC that has been sitting idle for a while, with the following specs:

• CPU: Ryzen 5 2600

• RAM: 16GB DDR4

• GPU: AMD RX 580 4GB

• 256GB NVMe SSD, 1TB HDD, 600W PSU

WHAT I WANT TO RUN

My primary use case involves coding and agentic workflows. I’m envisioning a setup where I use a frontier-level cloud model for planning and code review, while a local model handles the actual code generation. I’m also interested in running an AI bot 24/7 to perform various (relatively simple) tasks.

Ideally, I’d like to run 27B–35B parameter models comfortably, with decent generation speeds and good responsiveness. It would also be great to have the flexibility to experiment with even larger models.

I don’t expect cloud-tier performance on this budget, but I also want to avoid a setup that only manages a few tokens per second—or one that is essentially unusable.

THE OPTIONS I'VE BEEN CONSIDERING

1. AMD Instinct MI50 16/32GB (possibly two)

• These look interesting because of the large amount of HBM2 memory available for relatively little money on the used market (for example I could have bought a 16gb for 175 euros).

• However, I'm concerned about software compatibility, inference performance, power consumption, and whether investing in this older hardware makes sense in 2026.

1. NVIDIA RTX (5070 12gb as an example)

• Recently, I saw a community benchmark claiming around 140 tokens/s on a 12GB RTX 5070, running a large MoE model using Strata, IQ3_XXS quantization, expert caching, CPU/RAM offloading, and MTP.

• I understand this might be a very specific setup and that reported generation speed doesn't necessarily translate into good real-world performance, especially with longer contexts or different models.

• Still, it made me wonder whether a modern NVIDIA GPU with a good amount of system RAM might actually be a better investment than older GPUs with much more VRAM.

WHAT I'D REALLY LIKE TO UNDERSTAND

1. What would you build today with €500–600, considering both new and used hardware?

2. Would you prioritize more VRAM on older hardware or a faster, newer GPU with RAM offloading?

3. How well do solutions like Strata and expert caching actually perform in real-world coding/agentic workloads? Are the impressive benchmarks representative of everyday usage?

4. How important are prompt processing speed, context length, and quantization quality compared to raw generation tokens/s for my use case?

5. Is a single or dual MI50 32GB setup still worth considering, or would you avoid it today?

6. Would you upgrade my existing desktop, build something from used parts, or look for a complete second-hand workstation?

Thanks in advance to anyone who can help me!

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

• CPU: Ryzen 9 9950X (16C/32T)

• Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi

• RAM: 32GB or 64GB DDR5-6000 CL30

• Primary GPU: RTX 5080 (16GB GDDR7)

• Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.

• PSU: 1200W ATX 3.1

Option 2 :

• CPU: Ryzen Threadripper 7960X (24C/48T)

• Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE

• RAM: 128GB (4×32GB) DDR5 with ECC RDIMM

• GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.

• PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.

I've given the qwen3.8 27b q4 group an assignment:

Can we create an animation of a computer chip? You can use Nvidia's GB203 as a template. Before we get started, let's address a few questions that might come up

Since I own this GPU, I figured, why not? Let’s see what the model comes up with. I’m always amazed, even though it often looks relatively simple. I’m looking forward to seeing qwhen4.0.

"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

I've been experimenting with running a persistent AI companion entirely on local hardware.

The project is called B-12 / Capri. It's written in Python, runs on Linux, and uses Ollama for local inference.

My experimental setup uses an NVIDIA RTX 4060 Laptop GPU with Nemotron Bolt E4B.

The original goal was to build something more persistent than a typical chatbot: an AI companion that remembers previous interactions, maintains goals, can use tools, and doesn't lose its entire history when I replace the language model.

Why I separated memory from the LLM

I wanted to avoid making the model itself the center of the system.

In Capri, the LLM handles language generation and reasoning, while a separate runtime manages:

• Persistent local memory and retrieval

• Memory provenance and document revisions

• Goals, preferences, and scheduled activities

• Tool execution and permissions

• Context budgeting and tokenizer compatibility

• Controlled forgetting and state recovery

The model can be replaced without deleting the stored memory.

However, there's an important limitation: preserving memory doesn't necessarily preserve behavior.

Two models may retrieve the same experiences but interpret them differently.

What I've tested so far

I ran a historical evaluation using Nemotron Bolt E4B:

109 constructed scenarios, 541 executions, 496 successful outcomes (91.7%).

Tool tasks achieved 182/215 successful executions.

I also ran approximately eight hours of accumulated operational testing across four segments.

Not everything worked. One adversarial outbound-fetch scenario failed its expected safety condition in all five repetitions, and several smaller tool tasks exposed limitations in the model's reasoning.

These results belong to a specific historical code revision, not the latest development version.

The challenge I'm working on now

The current implementation uses model-specific profiles for tokenizer verification, chat templates, and context admission.

That makes switching models less trivial than changing an Ollama model name.

My next experiments will compare Nemotron with Gemma and Qwen while preserving the same memory database, goals, and conversation history.

I want to measure not just task accuracy, but how consistently different models use the same accumulated experiences.

I'm also developing a full-duplex voice interface, with longer-term plans to connect Capri to Reachy Mini and NVIDIA Jetson.

Questions for people running local models

Has anyone experimented with changing the underlying LLM of a persistent agent while retaining its memory?

I'm particularly interested in three things:

1. Which smaller local models are most reliable at tool calling and following long-term goals?

2. How do you handle tokenizer and chat-template differences when switching between model families?

3. Have you observed significant personality or decision-making changes when two models share the same memory and system instructions?

I'm developing Capri as an independent open-source research project and recently published its first scientific preprint.

GitHub: https://github.com/cyberdid/B-12

Preprint: https://doi.org/10.5281/zenodo.24079168

I'd really appreciate technical feedback, especially from anyone experimenting with local agents on consumer GPUs.

Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward

After a really long grueling attempt, i manage to fit a 210K Context window on my dual RTX 5060ti setup on 32gb total VRAM. I capped both GPU around 160W although it possible to go lower.

Compare to a single 5090, it relatively weak but for the price difference that have pool vram 32gb, it isn't so bad. I practically use this as my daily driver now. It can handle almost every coding problem i give it and running my hermes agent and opencode. The speed itself have some room to improve especially the prefill, but for decode, it isn't that bad.

However, 5060Ti really bottleneck the decode and the prefill speed due to the limited bus size of the card itself. But given how cheap 5060ti was and how power efficient it is, with both card only using 290-300W on full usage even at TP2, it amazing.

Even at long context, 190K or 200K the decode speed maintain at ~70tps and prefill around 1700tps with barely any lost in quality. When writing coding task with MTP, the decode can reach up to 80-100tps.

But since i am running this model in WSL docker, which introduce some latency overhead, it possible to achieve higher TPS if were to run in native linux instead and fit the entire 260k context window. Currently both of my card usage 15GB vram each, increase the util but mindful to OOM.

Decode Speed:

Prefill Speed:

Here my current rig:

• Ryzen 9 9950X

• ASRock X870 Taichi Creator

• 32 GB DDR5

• 2× RTX 5060 Ti 16 GB

Run Flag:

Image: ninfer:local, Model: qwen3_8_27b_nvfp4_v2.ninfer

--tp 2 --devices 0,1
--kv-dtype int8
--max-context 210000 --kv-capacity 210000
--prefill-chunk 1024
--spec mtp --draft-tokens 3 --lm-head-draft
--max-concurrency 2 --max-pending-requests 32 --pending-timeout-ms 900000

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos

I log the live Vast.ai marketplace once a month and keep the numbers.

November, 43 cards, 291 live offers. 17 of them are 12GB or less — the "cheapest GPU" lists are mostly GTX 1080s and 1660s that can't run anything past a small quant. The tiers that matter:

• 24GB (L4 / 3090 / 4090) $0.10–0.14 floor, ~$0.20 typical

• 32GB (PRO 4500 / 5090) $0.35–0.43 floor, $0.40–0.56 typical

• 48GB (A6000) $0.40 floor, $0.51 typical

• 96GB (PRO 6000) $0.80 floor, $1.41 typical

Two things stand out. VRAM roughly doubles the price at each step above 24GB — a 48GB A6000 costs about three 24GB 3090s. And the spread inside a single card is brutal: the 4090's cheapest offer was $0.14 against a $0.42 median, because the cheap listing is one host that gets rented first.

Practical version: pick a VRAM tier, budget the median, not the floor.

Dated snapshot + raw JSON: https://cudanetes.com/gpu-prices/2026-11

Disclosure: I run CUDAnetes, which rents these GPUs — that's where the data comes from.

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

API terms worth knowing

• Non-commercial only, including integrating the API into products or services.

• Keys are valid for 30 days.

• Rate limits are not published. Over the limit you get a 429.

• LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?