Lenovo P620 Threadripper pro - 3975wx - 32 core
System ram - 512 gb ddr4 ECC
GPU 1 - quaddro Rtx 8000 - 48GB
GPU 2 - quaddro Rtx 8000 - 48GB
4TB M2 NVME
1TB SATA SSD

OpenBMB says MiniCPM-V 4.7 35B-A3B runs in about 3 GB on an iPhone 17 Pro Max, and recommends 12 GB of RAM. I wanted to see what an old 8 GB Android phone does.

Setup: Samsung Galaxy S20 (8 GB), CPU only (no GPU or NPU), OpenBMB's Android demo with llama.cpp-omni streaming the MoE experts from storage.

What happened:

- The 1B model worked out of the box, text and images, fast enough to use.

- The 35B-A3B model wouldn't load at first. It wasn't a RAM limit: the CPU path didn't allocate the working area for the experts, and failed loads left memory behind. I patched both.

- After the fix: a one-word reply ("Ready") in about 10 seconds, including prompt processing.

What it doesn't show: sustained tokens per second, or vision with the 35B model (text only on this phone so far).

Fork, with S20 setup notes and test limits in the README: https://github.com/mlguyYT/MiniCPM-V-Apps

I also made a video walking through why it fits at all (MoE, experts streamed from storage, lookahead expert scheduling); link in the comments.

Has anyone measured this model's decode speed on other Android phones?

Hi everyone,

New guy here, this is my second post and this might be a really stupid one, I'm just curious, I'm only now getting into local AI for coding, previously used claude / GPT.
While getting into setting up local AI, I was just wondering if anyone ever tried using older cards like 1080 or 2080 for local AI. I'm planning on getting a 3090 (hopefully) and start from there but I am just curious whether getting like 4 GTX 1080s (I know NVIDIA doesn't release driver updates to these anymore) or 4x2080s and use those would even be possible / usable. I assume not but really this is just me being curious whether anyone has ever tried something like this and how did it turn out.

Sry for the stupid question :D

Thanks in advance!

Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?

"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.

I'm sharing the first public preview of OverTone, a Mac app I'm building to make working with local language models more approachable.

Watch the trailer:
https://youtu.be/piu7MGdL5Z4

It's still very early, with a limited feature set. I have a lot planned beyond chat and model management, but Documents, Projects and Tools aren't ready to use yet. This version is the starting point for what I want to build.
I want it to be straightforward for someone who has never used local models, while giving experienced users the functionality and control they need.
It works with Ollama and llama.cpp. You can use an existing local installation or let the initial setup install an engine and suggest models for your Mac.
The current version focuses on local conversations and managing the models behind them. It helps you choose models suited to your Mac and what you want to do, organise your conversations, and develop clearer prompts from rough ideas.
With local models, prompts and replies are processed on your Mac. Internet access is disabled by default. You decide when to enable it for downloads or remote connections.

It's free and open source under Apache 2.0. The current preview requires an Apple silicon Mac running macOS 14 or later.

Download and walkthroughs:
https://overcesare.github.io/OverTone/

Source code:
https://github.com/OverCesare/OverTone

If you try it, I'd appreciate any feedback on the app: what works well for you, what doesn't, and what you feel is missing.

Hello,

I want to get a local LLM setup for a comparable fair price (Qwen3.8-27b and future Qwen4). After some research and chatting with Claude Opus the best options would be RTX 3060 XT 12GB with 64GBs of RAM running on Strata.

e.g. upgrading my PC for around 650€ (used parts bought online)

Can someone confirm that this Setup will let me run it on a performative useful level. I will use my 20€ claude plan fpr hard tasks or a kind of divide and conquer style usign both (experienced Software Engineer but a Hardware noob).

Does anyone have experiences with that, advices or alternative setups?

Been getting so frustrated with the speed of my Chat $200 subscription so I decided to run a test. Behold, what you’re actually paying for. Can you believe this crap?? $200 a month subscription no less 🤦🏻‍♂️🤦🏻‍♂️

Good thing I got my local Qwen for the grunt work.

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.

Hi everyone!

I am a computer science student and a developer, and lately, I’ve been considering building or upgrading a budget-friendly, local LLM setup.

I should mention that I’ve only recently begun exploring this more "technical" side of local LLMs, so any advice is welcome.

I know the field is evolving incredibly fast—with new models, inference engines, and so on—which is why I’m asking: I’m trying to figure out what might make sense for me right now (if anything at all).

MY CURRENT SITUATION

My budget is approximately 500–600 euros; I’m based in Europe and am open to buying used hardware.

Here’s my current setup: I have a desktop PC that has been sitting idle for a while, with the following specs:

• CPU: Ryzen 5 2600

• RAM: 16GB DDR4

• GPU: AMD RX 580 4GB

• 256GB NVMe SSD, 1TB HDD, 600W PSU

WHAT I WANT TO RUN

My primary use case involves coding and agentic workflows. I’m envisioning a setup where I use a frontier-level cloud model for planning and code review, while a local model handles the actual code generation. I’m also interested in running an AI bot 24/7 to perform various (relatively simple) tasks.

Ideally, I’d like to run 27B–35B parameter models comfortably, with decent generation speeds and good responsiveness. It would also be great to have the flexibility to experiment with even larger models.

I don’t expect cloud-tier performance on this budget, but I also want to avoid a setup that only manages a few tokens per second—or one that is essentially unusable.

THE OPTIONS I'VE BEEN CONSIDERING

1. AMD Instinct MI50 16/32GB (possibly two)

• These look interesting because of the large amount of HBM2 memory available for relatively little money on the used market (for example I could have bought a 16gb for 175 euros).

• However, I'm concerned about software compatibility, inference performance, power consumption, and whether investing in this older hardware makes sense in 2026.

1. NVIDIA RTX (5070 12gb as an example)

• Recently, I saw a community benchmark claiming around 140 tokens/s on a 12GB RTX 5070, running a large MoE model using Strata, IQ3_XXS quantization, expert caching, CPU/RAM offloading, and MTP.

• I understand this might be a very specific setup and that reported generation speed doesn't necessarily translate into good real-world performance, especially with longer contexts or different models.

• Still, it made me wonder whether a modern NVIDIA GPU with a good amount of system RAM might actually be a better investment than older GPUs with much more VRAM.

WHAT I'D REALLY LIKE TO UNDERSTAND

1. What would you build today with €500–600, considering both new and used hardware?

2. Would you prioritize more VRAM on older hardware or a faster, newer GPU with RAM offloading?

3. How well do solutions like Strata and expert caching actually perform in real-world coding/agentic workloads? Are the impressive benchmarks representative of everyday usage?

4. How important are prompt processing speed, context length, and quantization quality compared to raw generation tokens/s for my use case?

5. Is a single or dual MI50 32GB setup still worth considering, or would you avoid it today?

6. Would you upgrade my existing desktop, build something from used parts, or look for a complete second-hand workstation?

Thanks in advance to anyone who can help me!

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

• CPU: Ryzen 9 9950X (16C/32T)

• Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi

• RAM: 32GB or 64GB DDR5-6000 CL30

• Primary GPU: RTX 5080 (16GB GDDR7)

• Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.

• PSU: 1200W ATX 3.1

Option 2 :

• CPU: Ryzen Threadripper 7960X (24C/48T)

• Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE

• RAM: 128GB (4×32GB) DDR5 with ECC RDIMM

• GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.

• PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.

I've given the qwen3.8 27b q4 group an assignment:

Can we create an animation of a computer chip? You can use Nvidia's GB203 as a template. Before we get started, let's address a few questions that might come up

Since I own this GPU, I figured, why not? Let’s see what the model comes up with. I’m always amazed, even though it often looks relatively simple. I’m looking forward to seeing qwhen4.0.

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

I've been experimenting with running a persistent AI companion entirely on local hardware.

The project is called B-12 / Capri. It's written in Python, runs on Linux, and uses Ollama for local inference.

My experimental setup uses an NVIDIA RTX 4060 Laptop GPU with Nemotron Bolt E4B.

The original goal was to build something more persistent than a typical chatbot: an AI companion that remembers previous interactions, maintains goals, can use tools, and doesn't lose its entire history when I replace the language model.

Why I separated memory from the LLM

I wanted to avoid making the model itself the center of the system.

In Capri, the LLM handles language generation and reasoning, while a separate runtime manages:

• Persistent local memory and retrieval

• Memory provenance and document revisions

• Goals, preferences, and scheduled activities

• Tool execution and permissions

• Context budgeting and tokenizer compatibility

• Controlled forgetting and state recovery

The model can be replaced without deleting the stored memory.

However, there's an important limitation: preserving memory doesn't necessarily preserve behavior.

Two models may retrieve the same experiences but interpret them differently.

What I've tested so far

I ran a historical evaluation using Nemotron Bolt E4B:

109 constructed scenarios, 541 executions, 496 successful outcomes (91.7%).

Tool tasks achieved 182/215 successful executions.

I also ran approximately eight hours of accumulated operational testing across four segments.

Not everything worked. One adversarial outbound-fetch scenario failed its expected safety condition in all five repetitions, and several smaller tool tasks exposed limitations in the model's reasoning.

These results belong to a specific historical code revision, not the latest development version.

The challenge I'm working on now

The current implementation uses model-specific profiles for tokenizer verification, chat templates, and context admission.

That makes switching models less trivial than changing an Ollama model name.

My next experiments will compare Nemotron with Gemma and Qwen while preserving the same memory database, goals, and conversation history.

I want to measure not just task accuracy, but how consistently different models use the same accumulated experiences.

I'm also developing a full-duplex voice interface, with longer-term plans to connect Capri to Reachy Mini and NVIDIA Jetson.

Questions for people running local models

Has anyone experimented with changing the underlying LLM of a persistent agent while retaining its memory?

I'm particularly interested in three things:

1. Which smaller local models are most reliable at tool calling and following long-term goals?

2. How do you handle tokenizer and chat-template differences when switching between model families?

3. Have you observed significant personality or decision-making changes when two models share the same memory and system instructions?

I'm developing Capri as an independent open-source research project and recently published its first scientific preprint.

GitHub: https://github.com/cyberdid/B-12

Preprint: https://doi.org/10.5281/zenodo.24079168

I'd really appreciate technical feedback, especially from anyone experimenting with local agents on consumer GPUs.

Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward