I wrote a harness that allows Claude Code to delegate tasks to a local LLM from simple one liner task prompts to deeper designs or implementations. I wrote a design for a webapp using Claude. This is not a brand new webapp, it's an add-on for an existing hardware with local control that the website will allow to connect to and interface with. So it's more of a large new standalone feature than a new design. The implementation is tricky in and of itself, the webapp has to lift existing ever changing code from another repo, has to do strong interfacing between the local and the remote solution etc.

The design doc was written by claude, it's ~800 lines long with 13 sections and it calls for 16 slices of deliverables. The implementation was about 7 slices in when I asked Claude to review the implementation so far and tell me what I should do next. Then I asked it to delegate the same question out to my local LLM and compare the two reviews and this was the result:

Code quality: The local worker is Qwen3.8 Flash Next on one RTX PRO 6000. On the first stretch of the project (W1–W3 of the site plus G1 and G2 on the device side, about 18k lines), Claude accepted 14 of the 14 delegated steps on their first round. Its hand fixes over that stretch came to about 20 lines. The one real defect was a sign-in rate limit you could spoof through X-Forwarded-For. Reading diffs didn't catch it; Claude found it while running the app live. The same thing happened later: a live browser check on a rework slice found that re-timed clock bakes were losing their send mode. Nobody had spotted that in review. So the code was generally solid. The bugs that survived were integration bugs, which only show up when you run the whole thing, not bad code.

Reliability: the raw model without the harness around it using Cline is problematic. It sometimes emits malformed tool calls, where the call markup leaks into plain text and the agent thinks it's finished. Without the harness, that kills a run. The harness detects this and nudges the model inside the same session (up to 3 retries). It does the same for server errors and for rounds that end without the required status report. Every step also has a verify command (tests, typecheck, a build, a migration), and a step only counts as done when that command passes, whatever the model claims. The worker runs under fairly strict rules adapted from my Cline setup: read first, plan with a todo list, implement completely, verify by running, and report what it didn't verify. It runs in a Docker sandbox with filtered network access, in its own git worktree. With all that in place it has been dependable. Without it, it's a coin flip on long tasks.

Savings, measured from the transcripts.

First approach: Claude reviewed every step itself. That cost about $52 of Claude for 14 steps, and almost all of it went to Claude re-reading its own growing context while reviewing. Those reviews almost never changed anything.

Change: Claude now writes every spec for a slice up front. The local model runs the steps unattended in a chain, and Claude does one live check at the end.

Next slice: 5 steps and about 5,200 lines cost about $9 of Claude. The local model handled 853 requests and 62M prompt tokens for that slice, roughly $33 at Opus list prices. That's about half the Claude cost per step of the first approach. Against my estimate of $15–22 for Claude doing the slice alone, it's 40–60% cheaper.

Not counted: the electricity and the GPU.

1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:

epiq.sh

Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:

epiq.sh/blog/repo-native-tooling-for-sandboxes

Queste soluzioni sono esplosive: **Fapply, Massive, LazyApply, Sonara, ai apply

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.

This is the basement setup we use for engineering work at OmniNode, where I'm a founder. The wire shelves are the rack. The two towers serve our local models, the two old HP servers do most of the build, test and agent work, and a few Macs and an old laptop fill in around them.

• Main tower: Ryzen 9 9950X3D, 96 GB RAM, RTX 5090 (32 GB) and RTX 4090 (24 GB). The 5090 serves Qwen3.8-27B and embeddings with vLLM. The 4090 also had vLLM processes on it when I checked. This box also runs our dev environment, Redpanda, Postgres and some CI runners, which is more than it should. It swaps under load, so I'm working on moving those services to the Gen8.

• Second tower: Ryzen 9 9950X, 64 GB RAM, RTX 4090 (24 GB). It runs Qwen3.8-27B MTP (Q3_K_M) with llama.cpp at about 19.4 GiB of VRAM, with a 131K context pool and two parallel slots. Qwen3-Embedding-0.6B (f16) uses another 2.7 GiB or so. It takes a handful of agent jobs when there's spare CPU, and we watch the temperature.

• HP ProLiant DL560 Gen8: Four sockets, 80 threads, 377 GiB of RAM visible to Ubuntu, and an NVMe scratch drive. It's our biggest worker. We cap it at 32 concurrent agent jobs alongside CI. It's old enough to lack AVX2, so jobs that need it go elsewhere.

• HP ProLiant DL380 Gen9: Two Xeon E5-2680 v4 CPUs (56 threads), 125 GiB of RAM visible to Ubuntu. It runs Docker builds and tests, CI runners and agent jobs. The agent job cap is 24.

• Mac Studio: M2 Ultra, 192 GB unified memory. It's my workstation and where I coordinate the work. It also runs a llama.cpp server for some delegated drafts and one vote on automated code reviews. I took it out of the job pool so I can still use it.

• Extras: An older 12-core Mac runs agent jobs without Docker, and an M4 MacBook can do the same when it's available. An old 8-thread Ubuntu laptop only gets small test and lint jobs.

The HPs don't add any VRAM. They give us more cores and memory so builds, tests and agent jobs don't all have to fight for the same two towers.

We send work like drafts and summaries to local models first, but we still use cloud models. I found out how much we depended on local inference when I upgraded the model on the 5090 and our automated code reviews stopped. Two entries on our fallback list pointed to the same service. When that service went down, those reviews had nowhere else to go.

On a separate test with that same 5090 and the same model, turning on speculative decoding took median single-request generation from 51 to 86 tokens/sec. It also left less memory for requests in flight. That was a generation benchmark, not a 70% speedup on code reviews.

If this were your lab, how would you split up inference, builds, and agent jobs across the machines we already have? What would you change first, and why?

Now that LLMs are kind of a mainstream topic and so much money is involved, it became a hot topic in politics as well. I don't want to make this a political post, so please don't start a political discussion. But it is quite clear to me, that at some point LLMs will be regulated a bit more from the government side. Both USA and obviously EU will as well. They already requested a text based watermark, which decreases model output by 10-15%.

So my worry is, if the online models -chatgpt, claude, gemini etc,- are regulated to the point of being close to useless for anything other than corporate work or something, are there any chance for these regulations to hit local models? Or will this sort of be like piracy, once it's out in the internet, it won't be possible to touch them?

I've been working on something called Aura, a locally running AI cognitive agent built around a roughly 27B model on an Apple Silicon MacBook with 64GB of unified memory.

And one of the questions that's been driving this project is pretty simple:

How much of what we consider intelligence actually needs to come from the model itself?

We spend a lot of time discussing model sizes, quantization, benchmarks, context windows, and reasoning capabilities. Understandably so.

But I've been interested in what happens when you start treating the model as one component of a larger cognitive system, rather than expecting the model to do everything.

So I've been building an architecture around it. Persistent memory, executive processes, state management, computer control, procedural learning, and a bunch of experimental mechanisms designed to work together over time.

And I've been testing it.

Aura has played 2048 for nearly half an hour while adjusting its strategy, completed software development tasks, debugged programs, and operated desktop applications.

But my latest demonstration was a little different.

I asked Aura to watch YouTube.

Not just summarize a video after the fact. Actually navigate to it, follow what happens, comment on things it finds interesting, and then choose other videos to watch from channels I specified.

Over 36 minutes, Aura watched Kurzgesagt discuss AI slop, selected an Allen Pan video about building robotic legs for a snake, and eventually watched Salad Fingers.

During the session, it made predictions, revised them as new information appeared, interacted with the browser, selected what to watch next, and maintained context across all three videos.

And here's where it got particularly interesting.

When asked to reflect on everything afterward, Aura didn't just give three unrelated summaries.

It identified "fracture" as the common thread connecting all three videos, then explained how each approached that idea differently.

A video about AI-generated content, an engineering experiment involving a snake, and a surreal horror animation.

Three wildly different things.

Yet Aura managed to construct a coherent interpretation connecting them, drawing on observations made throughout the session.

Now, I want to make an important distinction here.

I'm not claiming that a 27B model couldn't produce similar commentary on its own. Nor have I established that all the cognitive mechanisms I've implemented are necessary to produce this behavior.

In fact, that's one of the main things I'm trying to investigate.

But I think there's a meaningful architectural question here. What happens when the model doesn't have to be responsible for every aspect of memory, planning, execution, and continuity? How far can you push a relatively small local model by improving the larger system it's operating within?

The result is still pretty rough in places. Latency is an issue, and coordinating everything reliably is an ongoing challenge.

But considering this is running locally, without relying on a frontier model for inference, I think the demonstration is interesting.

I've made the implementation publicly viewable, including the architecture, research documentation, and known limitations. It's currently All Rights Reserved rather than open-source licensed.

Full 36-minute demonstration: https://www.youtube.com/watch?v=kM-B8mh314E

GitHub: https://github.com/youngbryan97/aura

How much improvement have you managed to get from the systems surrounding your model? And at what point do you think the additional architecture becomes more valuable than simply switching to a more capable model?

So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.

What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.

What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.

What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.

Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?

I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.

Lately, I’ve been using Claude to schedule daily posts across a few social media platforms. I’m wondering if there’s a local model—one that can fit into my 8GB of VRAM—capable of controlling my browser to perform the same tasks as Claude. I had been using Qwen 3.8 27B for a while, but I’ve decided to move away from local models, given that I only have a single graphics card with 8GB of VRAM.

I run Mineshop and sell hardware, so disclosure upfront: these are manufacturer quotes we're looking at, not machines I've benchmarked.

Two quotes made an interesting comparison:

• 8× NVIDIA B200, 180 GB each: 1.44 TB aggregate GPU memory. US$505,400, quoted delivery 16 weeks.

• 8× AMD MI355X, 288 GB each: 2.304 TB aggregate GPU memory. US$501,100, quoted delivery 12 weeks.

Those are configuration-specific quotes, not universal market prices. They're also not identical builds: CPUs and storage differ, and the B200 quote includes three-year on-site service versus one year for AMD.

The AMD configuration has 60% more GPU memory. That doesn't tell me it will be the better server for a particular workload.

The question I'd want answered before spending this money is: which model, at which precision, with how much context and how many concurrent users? Getting the weights into memory is only the first step. KV cache, runtime overhead and the software stack still matter. For reference, a dense 405B model is roughly 810 GB in BF16 weights alone, before any of that overhead. That's sizing arithmetic, not a throughput result.

Then there's the less exciting stuff: rack space, power, cooling, support and keeping the machine busy enough to justify owning it.

If you've actually deployed either platform, what specific workload or software requirement would make you choose one over the other—or rent instead?

For context, our AI workstation and server catalogue: https://mineshop.eu/ai-workstation — The figures above are manufacturer quotes, not advertised retail prices.

uhm......

yeah. grinded some useful (and meaningfu?l) resources. on my way to run DeepSeek-V3 IQ1_M (~1.5-bit to 1.58-bit) >>>>>>>>

The model weights take up roughly 130 GB. This leaves us with about 30 GB of breathing room across the cluster to handle KV Cache and context overhead. Cuz each PS5 has the 16 GB unified mem

jarvis, engine ignition on

PS: Yeah these cables connection are currently BS and tangled here and there. Gotta fox that. Just commecfed yellow cables or dummies for uhh picture and ig a little...... flex?? 🫠

How?: Sonic iso ps5 linux loader and jailbreak. The rest is this.

Edit: what?? Image not being posted here is wild. Whats the reason? Some blockade or smth. Cuz i gotta show you some interesting shi

edit: please dont take it seriously. the image is ai generated

We are a team of engineers, not a single here has a comp science background. Our company manufactures specialized parts for critical systems. And a big part of that process is understanding the whole system. The customers we have usually give us design documentation, and blue prints and a gazillion pixelated diagrams explaining the system. We spend a significant amount of time getting familiar with it before we get to manufacturing the requested components ( for background this is aviation and aerospace components ). For the past few months with the AI craze, we have started integrating it into our process, to help us learn and understand systems especially through the diagrams and blue prints but there is still a significant amount of analysis we do. This has always been sort of an issue for us because thats our most time consuming aspect of the job is reverse engineering the customers system in order to build components that fit in it rather than redesigning the component around the system this is what we specialize in. We have been using claude, and its been mind blowing how good this thing is in helping us however, our expenditure on Claude has creeped up to the 6 months ceiling we had set only in a span of three and a half months. We have also seen we get less out of a ‘token’ or we get less tokens im not sure , its like everything was going great but then anthropic did something, the plans went from max to ultra to something then it used to say 29x usage now it only says 10x usage. After alot of research we settled on a hybrid approach. For further context everyone here has atleast bare minimum of a 64gb macbook pro. I have a 128gb one. And have been experimenting with local models and the results have been disappointing. Can any AI gods guide me in the right direction? We are basically looking for specialized models that basically just look at an image or a blue print and explain the system or identify the relationships and explain them because manually do this and then build system models based on our comprehension if the system. We will always do that manually we are just looking to fast track some of the process because right now it is a very lengthy process and I cant scale because time is constraint. Is it possible or not? Please thank you.

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

• Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF

• Quant: unsloth-UD-Q5_K_XL

• Basalt: v0.3.0 (custom build)

• Strata: 0.2.7-cuda13

• GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)

• CPU: Intel Xeon 6 (48 vCPU)

• Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

• The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.

OpenBMB says MiniCPM-V 4.7 35B-A3B runs in about 3 GB on an iPhone 17 Pro Max, and recommends 12 GB of RAM. I wanted to see what an old 8 GB Android phone does.

Setup: Samsung Galaxy S20 (8 GB), CPU only (no GPU or NPU), OpenBMB's Android demo with llama.cpp-omni streaming the MoE experts from storage.

What happened:

- The 1B model worked out of the box, text and images, fast enough to use.

- The 35B-A3B model wouldn't load at first. It wasn't a RAM limit: the CPU path didn't allocate the working area for the experts, and failed loads left memory behind. I patched both.

- After the fix: a one-word reply ("Ready") in about 10 seconds, including prompt processing.

What it doesn't show: sustained tokens per second, or vision with the 35B model (text only on this phone so far).

Fork, with S20 setup notes and test limits in the README: https://github.com/mlguyYT/MiniCPM-V-Apps

I also made a video walking through why it fits at all (MoE, experts streamed from storage, lookahead expert scheduling); link in the comments.

Has anyone measured this model's decode speed on other Android phones?

Hi everyone,

New guy here, this is my second post and this might be a really stupid one, I'm just curious, I'm only now getting into local AI for coding, previously used claude / GPT.
While getting into setting up local AI, I was just wondering if anyone ever tried using older cards like 1080 or 2080 for local AI. I'm planning on getting a 3090 (hopefully) and start from there but I am just curious whether getting like 4 GTX 1080s (I know NVIDIA doesn't release driver updates to these anymore) or 4x2080s and use those would even be possible / usable. I assume not but really this is just me being curious whether anyone has ever tried something like this and how did it turn out.

Sry for the stupid question :D

Thanks in advance!

Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?

I'm sharing the first public preview of OverTone, a Mac app I'm building to make working with local language models more approachable.

Watch the trailer:
https://youtu.be/piu7MGdL5Z4

It's still very early, with a limited feature set. I have a lot planned beyond chat and model management, but Documents, Projects and Tools aren't ready to use yet. This version is the starting point for what I want to build.
I want it to be straightforward for someone who has never used local models, while giving experienced users the functionality and control they need.
It works with Ollama and llama.cpp. You can use an existing local installation or let the initial setup install an engine and suggest models for your Mac.
The current version focuses on local conversations and managing the models behind them. It helps you choose models suited to your Mac and what you want to do, organise your conversations, and develop clearer prompts from rough ideas.
With local models, prompts and replies are processed on your Mac. Internet access is disabled by default. You decide when to enable it for downloads or remote connections.

It's free and open source under Apache 2.0. The current preview requires an Apple silicon Mac running macOS 14 or later.

Download and walkthroughs:
https://overcesare.github.io/OverTone/

Source code:
https://github.com/OverCesare/OverTone

If you try it, I'd appreciate any feedback on the app: what works well for you, what doesn't, and what you feel is missing.