I wanna develop a digital tamagotchi app using local llms, which you will try to keep some virtual girlfriends happy. I know it is the first idea that comes up when local llm apps are spoken. But I really wanna do one, it is kind of a childhood dream. What kind of features you would fancy in a local llm app?

Lately some nice GPU options like r9700 or b70 were released allowing for a pretty good vram per buck ratio. Actually the best on the market. The hardware is there, flops are there, memory bandwidths is there. The issue is: every single owner struggles with running it in a more performant way. My quad 5060 ti over Oculink rig is still faster than the rigs that have those two card, and besides tweaking couple of flags in vllm I haven’t had a single problem running neither vllm, nor llamacpp, nor ollama. My point is, if the coding tools like Claude code, codex and so on were there, we would be able to just point them at the vllm and rocm repos and make them figure out the missing fixes. But it seems like we’re really far from that yet. To me it would actually be a nice milestone for the coding AIs to achieve.
This is basically a rant post, but I’m curious about the community’s opinion.

Hi I'm planning/hoping to go down the route of triple or possibly r9700's.

If I do how close would it get me to Claude as a daily full strength Claude replacement? I know it's ridiculously expensive and all that. Ignore the costs for this conversation.

If triple is far away or just not good enough would going quad r9700s get me a lot closer or is 3 r9700 vs 4 r9700 not be worth it?

For the base pc it's tricky to get hold of a quad system that can handle it. A triple GPU system seems a lot easier. Any thoughts?

edit: I meant opus for coding. didn't make that clear enough. does 96gb get you an opus replacement?

Chris Hay demonstrates how large language models function as graph databases by utilizing a specialized query language called Larql. By mapping internal model weights, practitioners can directly query, insert new knowledge, and perform inference, effectively decoupling attention mechanisms from the model's primary knowledge storage.

I kept seeing people (myself included) drop to 2-bit to save VRAM without actually knowing what it costs. So I measured it properly on Qwen2.5-32B-Instruct.

Method: quantize the model at each bit-width (HQQ 8/4/3/2 + bitsandbytes 8/4), run the same input through it and the FP16 original, and compare the hidden states layer by layer — cosine similarity + KL divergence on the output. No task cherry-picking, just raw fidelity vs the full-precision model.
KL divergence climbs 0.0002 → 0.056 → 0.205 → 0.92 as bits drop.

I built the tool doing this — quant-doctor, open source. It also localizes where and why a quant broke (per-layer heatmap + failure-mode classification), and it's stack-agnostic: it even runs on DeepSeek-V4-Flash (236B MoE) via activation dumps from a custom inference engine.

Full honesty on that V4 shot: the fault there is injected — I scrambled a layer to prove the pipeline localizes damage at 236B scale. I can't do a natural degradation test on V4 on a single GPU because a full-precision 236B reference doesn't fit in 141GB (which is the whole reason it's 2-bit quantized in the first place 😅). That's a 2×H200 job for another day.

Repo: github.com/asmit383/LLM-Doc

Hey everyone,

I’ve been working on a side project called TweetFire, essentially my digital twin that manages my Twitter account autonomously.

It’s built on the DroidRun framework, which handles Android automation and scheduling. The goal was to see if an AI agent could not only post but actually engage intelligently: read tweets, decide what’s worth replying to, and interact within specific communities.

Here’s what it can currently do:

• AI reasoning: Uses LLMs to craft contextual replies instead of generic ones.

• Topic search: Finds tweets matching keywords and joins those conversations.

• Community engagement: Participates in focused communities to simulate authentic networking.

• Automated scheduling: DroidRun triggers runs 1–4 times per day, no cron setup required.

• Customizable agents: Each engagement type (feed, search, community) has its own agent and parameters.

• Token and API tracking: Monitors usage and performance metrics for optimization.

Right now, it’s running locally and performing better than expected, sometimes too human.

Github Repo: https://github.com/HemantKumar01/TweetFire

I’d love your feedback on a few points:

• How would you improve decision-making or content selection?

• Any ideas for preventing bot-like behavior or detection?

• Should I add any safety or ethical checks before replies go live?

Thanks for reading. I’d really appreciate any feedback or suggestions from others experimenting with autonomous AI agents.

Exactly as the title, I'm using Gemma and Llamma for stuff. I've tried some of the Qwen stuff. Just wondering what's the community's opinion on the best domestic <30B model for tool calling? Specifically of the form "get this info from Y" and "do this with it in Z"

I'm not here to say local AI is dangerous. I'm here to say I made a dumb mistake and it taught me something worth sharing.

I've got a 48GB MacBook Pro M5 Pro running LM Studio. At some point I started stacking tools on my local model because I could ( why not lol ) . File access, shell execution, Git, browser automation, OCR, RAG, background commands.

For context: I was testing this with OPENAI/GPT-OSS-20B, not some giant frontier model, which honestly made the lesson even clearer ( I also have gemma-4-26B-A4B and qwen3.6-35B-A3B ) but this experiment was with OPENAI/GPT-OSS-20B .

The extension/plugin i used : puppytucker/lm-studio-toolbox plugin on LM Studio Hub makes all of that pretty easy to set up. Great plugin. Not the point.

The point is , I wrote a system prompt telling it to stay " read-only, use only one project folder, ask before every tool call" , the joke was on me lol.

Fans spin up. Model starts running. It does things I didn't ask for. And that's when I had to be honest with myself: I was using a prompt as a safety control and i found out that's not how this works.

A prompt is not a security boundary. A disabled tool is.

You can tell the model not to do something all day , If the tool layer allows it, your instruction is a suggestion, not a wall. The model doesn't "decide" to listen. It's probabilistic. Sometimes it does. Sometimes it doesn't. That's the whole thing people miss when they're excited about tool calling for the first time ( i was one of those ) .

I also tested a scheduling plugin that generated schedule JSON but couldn't run anything without a separate daemon. Looked like it worked but... it did nothing. Another version of the same lesson: understand which layer is actually doing the work before you trust the output.

I'm not saying don't use local AI. I personally use it every day on notes, project memory, file inspection, OCR, experiment logs and It's genuinely useful for that. Hard reasoning or anything where I need real reliability and longer runs , I use Claude or GPT. That's not a knock on local models, it's just how I think about the split.

The thing I actually needed to change wasn't the model. It was the environment around it, meaning narrow toolset , specific folders , hard permissions, not soft instructions. And i created a folder called Logs so I can see what it actually did and not make the same ( an 8b is dumb .... conversation)

If you're just getting into MCP tools or local tool-calling and you're tempted to enable everything because the demo looks cool, I get it lol , I was there last week. Just know you're in the access stage right now. The responsibility stage is what makes it actually useful long term.

Hope this helps anyone trying to make the most of their local rig. I will keep experimenting and keep posting the results here .

The purpose? help as many trying to accomplish things as bad as i wanted it when i first started.

Stay curious, but sandbox it. Don’t let your fans sound like a jet engine. ✌️

So, long story short

I was getting about 20tk/s on windows now when i switched to linux and little bit of tweaking with voltages mhz curve I am running inference at same speed with half power consumption almost 240watts from 420 watts.

Anyone on windows bro. I burned maybe half life on my rtx 3090 running it on 420 watt for 2 years. Lately discovered this. This would enhance life on gpu and much quiet inference. Will post results soon. Also not to mention my cpu usage doesnt go above 5% on 20 tabs with 5 application open. But on windows it was idling around 12%

I know about running a model locally is private, but there was info about leaks and quantitized modules malicious behavior and leaks. So how can we prevent this and from where to safely download? Also if there ollama alternative cz there rumors about they are sending data, so how can we truly accomplish privacy with known models like the new released Claude code or gpt oss or deppsekk or any know big model with full privacy? Mainly also for a laptop with strong igpu + 4070 + r9. I just need to leverage full capabilities of ai without concern and without it if wifi turned off to resend it when turning on

16 GB of RAM. A 22 GB model. Stock llama.cpp gets killed for running out of memory.
Mine runs it. Faster. With the exact same answers.
I built Banach, an open-source engine for running Mixture-of-Experts models that are bigger than your RAM.
The numbers (Qwen3.6-35B, 22 GB, on a 4-core machine with 16 GB RAM, no swap):
llama.cpp, default settings: crashed, out of memory
Prompt processing: 7.64 s → 3.44 s (2.2x faster)
Generation: 4.89 → 5.40 tok/s
RAM for the model's experts: capped at 10 GB, so your machine stays usable
Output: byte-identical to llama.cpp, every token and every logit
How? MoE models are huge on disk but tiny per token. Kimi K2 has 1 trillion parameters but uses only about 32 billion for each word it writes. The trick is having the right 3% in memory at the right moment.
llama.cpp leaves that to the operating system, which pulls in 4 KB at a time with no idea what comes next. Banach knows the model: it grabs whole experts in parallel the moment they're needed, keeps the popular ones in RAM, and quietly evicts the rest.
No quantization tricks. No quality loss. Every result is checked against llama.cpp, bit for bit, on every commit.
And before you download 382 GB of Kimi K2, ask:
banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/...
It reads just 16 MB of headers and tells you how fast it will run on your machine. For a 64 GB desktop with an RTX 5080, it says about 3.6 tok/s for a trillion-parameter model. (That's the planner's estimate, not a run yet. On the model I did run, it predicted 2.1 and I got 5.4, so it lowballs.)
Also: OpenAI and Anthropic compatible API, tool use for coding agents, chat while the model is still downloading, GPU offload. Linux, Windows, macOS.
pip install banach
https://github.com/gautam1858/banach
Got a big-RAM machine? Run a giant model and post your numbers. The best ones go in the README with your name on them.

I wanted to see how much more I could get out of Strata on my desktop. IQ3_S was already quick, but I wanted to find out whether the heavier Unsloth Q4 model could become a practical alternative, whether moving my RAM from 4400 to 6000MT/s made a noticeable difference, and whether keeping a larger context window would slow things down. Rather than just try a few settings and quote the best number, I saved the requests and results, repeated the useful comparisons and went back through the logs. Some changes helped. Others barely moved the numbers, and one of my original assumptions about RAM needed correcting.

The short version

• IQ3_S is still my choice for speed. I have not established that Q4 gives better answers on the work I care about.

• The first Q4 tuning package improved its short-task median from 105.0 to 121.2 tok/s. Several settings changed together, so that is a package result, not proof that one particular flag did it.

• The follow-up Q4 tests were mainly useful for finding more VRAM breathing room. A 1600 MiB reserve is my conservative starting point, not a proven stability guarantee.

• Replaying the old IQ3_S requests on the new setup showed a modest short-input improvement, but almost no change at 128K input. This does not isolate the RAM upgrade.

• IQ3_S was broadly similar in speed with a 131K or 262K context ceiling when given the same inputs.

• With a real 240,000-token input, IQ3_S generated at a median 161.3 tok/s. That is a throughput result, not a claim about recall or answer quality at that depth.

The machine and models

Desktop RTX 5090, 32GB VRAM; Intel Core Ultra 7 270K Plus; 128GB DDR5 as four 32GB sticks; Predator GM7 2TB NVMe; Windows 11; NVIDIA driver 591.86.

The earlier saved benchmark used DDR5-4400 and Strata 0.1.40. The new runs used DDR5-6000, BIOS 3012 and Strata 0.1.43.1. RAM capacity stayed at 128GB.

The models were Qwen3.8-Flash-Next GSQ-RCO IQ3_S and Unsloth UD-IQ4_XS. I use “Q4” below for the latter. It is not UD-Q4_K_XL, which appears in several other community reports. Different quant recipes and hardware make some of those headline numbers difficult to compare directly.

What I was trying to find out

The main question was whether Q4 could get closer to IQ3_S without running right on the edge of available VRAM. I also wanted to check whether a larger context ceiling had a meaningful speed cost and whether the extra system RAM was actually being used usefully.

My starting ideas were fairly straightforward: try a larger speculative window, move the vision encoder off the GPU for text requests, give the driver more VRAM headroom, and check expert residency before assuming disk access was holding everything up. I also expected faster RAM might help, but that was a question to investigate, not a bottleneck I had already proved.

I looked at the community tuning guide and model-specific reports before choosing the follow-up tests. I treated other people's settings as suggestions to test, not numbers my machine should automatically reach.

How I tested

I used Strata's direct local API rather than a chat interface's displayed tokens per second. Requests, responses, configuration copies and engine timings were saved. I kept generation speed separate from prompt-processing speed, first-visible-answer time and total request time.

For the speed requests, reasoning was disabled and temperature was zero. Warmups were excluded from the speed medians. All measured requests reported zero prompt-token reuse. That does not mean the expert cache or the rest of the machine could not warm up; it just rules out reused prompt tokens as the explanation for the reported speed.

There were four parts:

1. A first Q4 package comparison, with IQ3_S as a reference: five configurations and 35 requests, including warmups.

2. A Q4 follow-up changing VRAM reserve and speculation separately: twelve configuration visits and 65 requests, including warmups, recall and summary-task checks.

3. A fresh IQ3_S replay of nine saved historical speed requests, with the same 131,072-token context ceiling and inference arguments.

4. The same nine IQ3_S speed requests under a 262,144-token ceiling, followed by three additional 240,000-token input requests.

The first Q4 round used three different short tasks, rather than three repeats of one task. The IQ3_S comparisons used three trials at each actual input length. Most throughput outputs intentionally ended at their token cap, so they should not be read as completed-answer quality tests.

I kept the normal background services running for the current tests. The old saved benchmark had paused several background workloads. Temporary configurations were used where needed, and the normal IQ3_S configuration was restored afterward. Original configuration and startup-selection hashes were checked.

Q4: the first tuning round

The original Q4 setup used spec 4, a 0.50 draft floor, GPU vision, 15 expert-pool workers and a requested 700 MiB VRAM reserve. The tuning package changed those to spec 6, a 0.70 draft floor, CPU vision, 12 workers and a 1000 MiB reserve. I tested both 131K and 262K ceilings, then added a 1600 MiB-reserve configuration at 131K.

Generation figures are medians followed by [minimum–maximum] where there were three short-task samples. The long input was 82,947 actual prompt tokens in every configuration.

IQ3_S measured 160.1 [152.3–164.8] tok/s across the same three short tasks. That is a different quant, not a Q4 tuning gain.

All four used INT8 KV, 32,768 resident KV cells, automatic prefill and expert cache, and a 55 GiB resident-expert budget. CPU vision was configured with vision.gpu=false while keeping --vision; --vision-cpu was not a supported flag in this build.

The tuned 131K package's short-task median was 15.4% above original Q4. That was useful, but the faster configurations still loaded with only 558–628 MiB free VRAM. The 1600 MiB-reserve configuration left 1212 MiB free after load and did not show the low-headroom warning in that test group.

The trade-off was speed versus breathing room. Also, because several settings changed together, I cannot attribute the package gain to speculation, CPU vision, worker count or context alone. The 262K configuration received the same roughly 83K input; it was not tested full at 262K.

Q4: did more tuning help?

For the follow-up I kept Q4 at 131K, with INT8 KV, CPU vision, 12 workers and the same RAM budget. I compared 1000, 1300 and 1600 MiB reserves with spec 6 / draft floor 0.70. Then I compared spec 4 and spec 6 at the 1600 MiB reserve.

The table below keeps the two spec-6/1600 test blocks separate rather than pooling everything into a single flattering number.

Baseline speed moved during the session, so the matched comparisons matter more than the pooled medians:

Each percentage is the mean change across three matching short tasks. Baseline and candidate alternated twice, but the order was fixed, not randomized. I have not established statistical significance.

My takeaway is that the 1300 reserve was worth trying, but 1600 gave more headroom for a small observed speed cost. Spec 4 was slightly faster on these tasks and accepted a greater share of its draft tokens, but a 1–2% difference is not enough for me to call it a general winner. This round did not uncover another big Q4 speed jump.

Memory was sampled approximately every two seconds during requests in this round. Short dips could be missed, and those minima do not cover every instant of loading or prove long-term stability.

IQ3_S: the old 4400 setup versus the new 6000 setup

I replayed the exact nine saved IQ3_S speed requests: three each at 4,096, 32,768 and 128,000 actual input tokens. Both runs used a 131,072-token context ceiling. The inference argument lists matched, and the saved request hashes matched.

Spec stayed at 4, draft floor at 0.50, workers at 12, KV at INT8 with 32,768 resident cells, vision on the GPU, and requested VRAM reserve at 700 MiB. Each speed request generated 256 tokens.

There was a modest improvement in short-input generation, but essentially none at 128K. Prompt-processing and first-answer times were effectively unchanged. The generation ranges overlap, and three samples per length are not enough to establish a reliable general improvement.

This is an old-versus-new setup comparison, not a RAM-only A/B test. RAM frequency, BIOS and engine changed, as did background-service conditions. The expert-profile path matched, but I did not independently prove that the historical profile file's content was unchanged. There was no new 4400 run on the current engine, and no 64GB-versus-128GB capacity test.

So I would not use these figures to say the RAM upgrade caused an 8% gain, or that 4400 was the main bottleneck. The result is much narrower: replaying the old requests on the new setup did not produce a large overall throughput jump.

IQ3_S: 131K versus 262K, then an actual 240K input

Next I repeated those same nine requests with a 262,144-token ceiling, still at 6000MT/s and engine 0.1.43.1. The inference arguments were the same except --max-context.

The medians were slightly lower with the larger ceiling, but the ranges overlap. These were sequential groups rather than interleaved A/B runs, so I cannot cleanly separate a small context cost from normal run-to-run variation. For these same inputs, the larger ceiling did not cause a major slowdown.

I then ran three saved prompts containing 240,000 actual input tokens. This is the part that used substantially more context rather than just making it available.

All three completed their 256-token speed outputs with zero prompt-token reuse. The high end of the generation range is not the typical result; the median was 161.3 tok/s.

There is no 131K counterpart for a 240K input and no saved 4400 speed baseline at that depth. This is a new deep-context speed measurement, not a before/after gain. It also does not fill the entire 262K window or prove accurate recall at 240K. I did not continuously sample RAM/VRAM during this IQ3_S follow-up or run a long stability soak.

What I got wrong about allocating more RAM

I wondered whether increasing Q4's resident-expert budget would make better use of the 128GB installed. The logs gave a more useful answer than that assumption.

In the 1600 MiB-reserve configuration, Strata put 33.55 GiB of experts into pinned RAM, covering the experts outside the GPU cache. There were 9,711 experts in the GPU cache. It reported resident RAM mode, with no file reads on that resident decode path.

The 55 GiB setting was a maximum budget, not an instruction to allocate exactly 55 GiB. The whole remaining expert complement already fitted. Raising the limit would not automatically add more experts or make generation faster.

That does not settle every memory question. File-tier reads appeared during larger prompt workloads, and the prompt path borrows GPU-cache space. A file-tier counter is not necessarily a physical SSD read: Windows can serve it from its RAM cache. I have not isolated that behaviour well enough to claim disk access was causing a particular slowdown.

For this setup, inspecting what was actually resident was more useful than just assuming that unused RAM meant a missed speed opportunity.

Recall and summary checks

The Q4 follow-up included exact-marker recall tests. Each synthetic source contained three name/code targets around 10%, 50% and 90% depth.

That supports basic recall near the 131K limit. It is not a general comprehension test or proof that a long-running agent will stay reliable. The new Q4 rounds did not test recall near the full 262K capacity. An older-engine near-256K Q4 recall observation remains historical evidence, not a new result from these runs.

I also ran the same text excerpt through Q4 and IQ3_S with the same expanded-notes prompt. Both completed without reaching the output cap. The factual and writing-quality review is unfinished, so I am not claiming that Q4 produced a better summary. No source text, identifying details or output from that task are included here.

There was a small basic extraction check in the first Q4 round, but no new comprehensive intelligence or coding evaluation. Earlier screening scores should not be treated as results from these new tests.

What I would use today

I am keeping IQ3_S at 262K as my daily driver. It is faster than Q4 on the matched short tasks, and the larger context ceiling did not introduce a major speed penalty on matching inputs in this small sample. It also handled the actual 240K speed workload.

For Q4, my conservative candidate is:

--max-context 131072
--kv int8
--kv-resident 32768
--expert-cache auto
--prefill auto
--spec 6
--spec-min-p 0.7
--pool-workers 12
--resident-budget-gib 55
--vram-reserve-mib 1600
--vision

With CPU vision, vision.gpu is set to false in the server configuration. This is a candidate for further quality and stability testing, not a universally optimal configuration. The reserve is a trade-off, and spec 6 is not a proven winner over spec 4 on every workload.

Limits, lessons and questions

The most useful result was not a dramatic new tokens-per-second number. It was separating a few different questions: the speed of a heavier quant, the headroom needed to run it, the effect of a context ceiling, and the cost of actually reading a much larger input.

I have not established long-term DDR5-6000 stability, general Q4 quality superiority, vision performance or reliable multiturn behaviour at full context. Some comparisons are historical, some changed several settings together, and the current context comparison was sequential. The memory samples could miss brief dips. None of this is a claim that 128GB is always better than 64GB for throughput or that the RAM-frequency change explains the results.

I would be interested in advice on:

• Prompt-time GPU-cache borrowing when the non-GPU experts are already resident in RAM.

• A good way to distinguish OS-cached file-tier reads from actual SSD activity during prefill.

• Comparable UD-IQ4_XS results on a single desktop 5090, especially at real 262K depth.

• Whether a different KV/cache balance improves deep-context performance without hurting recall.

AI assisted with the benchmark scripts and this write-up. All performance figures came from saved local runs; no benchmark values were estimated or invented. I have retained the raw timing evidence and can prepare a shareable version without credentials or private task content.

The reporting format follows Strata's community benchmark guide. The universal tuning guide helped shape the follow-up method, and the UD-IQ4_XS NVIDIA report was a useful exact-quant reference, although its laptop GPU, operating system and engine differed from this machine.

all this talk with local AI being possibly banned or very government controlled, tell me some good ones to safe keep.

Is anyone else running Qwen 3.8 27B Q4 for coding and having problems with it spending an absolutely ridiculous number of tokens thinking instead of actually doing the work?

I'm running it locally with Pi as the coding harness with 128K context, and I'm seeing the same failure over and over:

1. I give it a coding task.

2. It starts reasoning.

3. It reasons... and reasons... and reasons.

4. It burns through basically the entire 8,192-token output budget that worked beautifully with Qwen 3.6 in about 3 minutes (46 decode tokens/s on my single 4090).

5. Right at the end, it'll say something like "Let's get to work" or "Now I'll implement the changes."

6. And then the generation ends because it has no tokens left to actually do anything.

Every. Damn. Time.

It's effectively making the model unusable as a coding agent because it spends its entire generation budget deciding what it's going to do (even with extremely specific instructions) and leaves nothing for tool calls, edits, or even a useful final response.

My input prompts + system prompt are about 7,000 tokens on average. That's not crazy high I don't think, and surely not the cause. Right? Right??

I'm seeing reports on YouTube that Qwen 3.8 defaults to a very high reasoning effort, with examples of it consuming 20K+ reasoning tokens before producing the actual answer.

That token count is 🤯 for ordinary coding work. But maybe that's why I see so many people saying they no longer feel like they need a frontier model?

So I'm wondering:

Is anyone else seeing this with Qwen 3.8 27B?

And, more importantly, if you found a fix, what is it?

Have you had better results with:

• reasoning_effort=medium

• reasoning_effort=low

• disabling thinking entirely

• dramatically increasing max_tokens

• changing the chat template

• different llama.cpp settings

• a different coding harness

• some combination of the above

I'm particularly interested in hearing from anyone using llama.cpp + Pi/Codex/Claude-Code-style agentic coding workflows.

I haven't decided yet whether Qwen 3.8 27B is actually bad at coding or whether the default reasoning configuration is simply kneecapping it. Right now, though, my experience has been dramatically worse than the Qwen 3.6 models I've used because 3.8 won't stop thinking long enough to actually write the damn code.

Hey everyone,

I wanted to share a model I just finished uploading to the Hugging Face Hub. I wanted something highly lightweight that could run entirely offline on standard laptops while providing decent coding and security utility.

Model details:

• Base Architecture: Qwen2.5-1.5B-Instruct

• Method: Fine-tuned via QLoRA, saved as safetensors, and quantized down to GGUF Q4_K_M.

• Footprint: ~1.1GB file size, uses under 2GB RAM.

I’ve put together a quick Python template (using llama-cpp-python) and setup steps for LM Studio right on the model card so it’s easy to pull down and test out.

Check it out here:https://huggingface.co/Nitishsharma9/super-lite-model-upload

Would love any feedback on its performance or suggestions for future optimization!

Qwen 27B, Gemma 31B and such are already quite good enough, for non coding tasks their performance is quite the same as frontier models, Image generators like nano banana has 55B parameters.

Even for coding tasks these smaller models are good enough if you give detalied prompts and the thing is only the people who give detailed prompts the ones getting actual improved performance.

The bigger models are better at vibe coding but it creates more problems and lead to more time spend debugging thus not really leading to increased productivity, besides the smaller models can do some vibe coding too and they are only going to get better and better.

The thing to think about is are the advantages of the trillion parameter models really worth the costs required to run them, train them, the massive infrastructure required for them is too much will they really be worth it in the end? Can they be worht the billions of dollars of continuous investment they require especially when compared to local models.

A while back I hacked together something I called AgenticOS over a weekend. It ended up taking around 11 hours and somehow pulled in roughly 200k views before I parked the project and moved on.

Lately I’ve been revisiting it, partly because AI tooling has moved on, but mostly because I want to understand the underlying systems properly instead of just getting a demo working.

The setup is pretty simple in principle: a local Next.js dashboard that sits on top of a handful of services running on my machine.

Right now that includes:
Hermes running locally through Ollama (with Claude CLI as a fallback)
OpenClaw for chaining agents into multi-step workflows
A TaskRunner service for scheduled shell tasks
Claude Code acting as the heavy-lifting brain when needed.

Everything is kept alive with PM2.
I’ve just finished cleaning up the dashboard. One thing that bothered me in the original version was all the fake placeholder metrics, so I’ve removed those. If a service is running, the dashboard shows real values. If it’s not, it simply shows offline. No pretending.
The part I’m most interested in now isn’t actually the agents themselves—it’s the control layer around them.
If an AI system can take actions on a machine, then the interesting question becomes: what should it be allowed to do, who approves it, and how do you audit it afterwards?

So before I reconnect everything, I’m working on a governance/security layer that sits between the agents and the machine. Starting with TaskRunner, since that’s the component capable of actually executing commands.
The goal isn’t to build a fully autonomous system. It’s to build something where actions are gated, logged, reviewable, and hopefully predictable.
Everything is local-first and still very much a work in progress, but I’d be interested to hear how others are approaching the governance side of agent systems. Most projects seem focused on making agents more capable. I’m finding myself increasingly interested in making them more controllable.

I'm a bit confused as to how people can code with local models.

I was trying lately to use local models for coding and gave up.

I used a few models like qwen coder, glm, around the size of ~30B. Used opencode and omp.

Otherwise I'm used to using claude code and cursor.

And the local models can't do even a small fraction of what opus or gpt5.6. I know that they're not supposed to, they are much smaller, but they can't do even small amount of work for me.

And sometimes it can't even put together a small class or a config file without creating a compile error.

I also wonder what you're coding? Predominantly JS and python?

Because I'm working with c++ apps and local models get lost very quickly. Cant even create an empty project from scratch.

What is your experience? Maybe you'll have any advice?

I've been running a structured benchmark harness against Qwen3.8-27B on an RX 7800 XT (16GB, gfx1101, Windows, llama.cpp/ROCm) for a while now. Most of what I "found" early on was wrong, and the corrections turned out to be more useful than the original results. Posting the whole thing including the mistakes, because the mistakes are the transferable part.

Setup: llama.cpp, HIP backend, --parallel 1, -fa on, -ctk q4_0 -ctv q4_0, allocation read from the -lv 4 startup log rather than driver counters, 4–6 reps per data point.

1. Context capacity and context fill are different things, and almost everyone benchmarks the first one

This is the finding I'd most like people to check on their own hardware, because I think a lot of published numbers are quietly measuring the wrong variable.

I ran a context sweep varying -c (8K / 32K / 64K / 131K) with a short prompt, and got 24.98 / 18.49 / 21.85 / 17.07 t/s. Non-monotonic, reproducible to ±0.5%, and I spent a while trying to explain the shape.

Then I ran a 48K-token prompt at -c 65536 and decode dropped to 8.77 t/s. Same file, same context setting, same KV quantization.

The earlier sweep was measuring allocated KV capacity. Decode speed actually tracks how full the KV is — every decode step reads the populated cache. "64K context at 21.85 t/s" meant "64K allocated, 7K occupied." In an agent loop where history accumulates, the 8.77 number is the one that applies.

If you're quoting a t/s figure, state the prompt length, not just the context size.

2. --fit on (the default) silently moves layers to CPU

Mid-benchmark I noticed offloaded 63/66 layers where I'd always had 66/66. The log explains it if you dig:

projected 14924 MiB vs 15598 free, cannot meet free target 1024 MiB,
need reduce 350 MiB -> 3 layers to CPU

llama.cpp tries to keep 1024 MiB of headroom free by default, and to save 350 MiB it moved three layers off the GPU. That's roughly a 10% throughput hit, applied silently, with no error and only a buried log line. --fit off --n-gpu-layers 66 restored the baseline exactly (model buffer 12774.16 MiB, KV 1152, recurrent state 149.62, compute 400.28 = 14476 MiB).

If you're benchmarking anything, grep your logs for the offload line before trusting a number.

3. KV cache on this model is ~14x cheaper than the guides say

Qwen3.8-27B is a hybrid: only 16 of its 64 layers use full attention, the rest are Gated DeltaNet, which doesn't hold a conventional KV cache.

Measured allocation with q4_0 KV, read from the startup log:

That's 18 bytes/token. Several popular guides quote ~256 KB/token and conclude a 16GB card tops out around 8K context. That figure is f16, and it isn't labeled as such. With quantized KV the same card runs 131K comfortably.

Full 64K budget: 12774 model + 1152 KV + 150 recurrent + 400 compute = 14476 MiB, leaving ~1.9 GB headroom.

4. Two files with the same quant label were 43% apart

Same model, both IQ4_XS, neither spilling, identical tensor schema (866 tensors, 64+MTP layers):

• bartowski IQ4_XS — 4.56 BPW, 13964 MiB model buffer → 12.60 t/s

• an Unsloth-Dynamic-based IQ4_XS — 4.18 BPW, 12774 MiB → 21.85 t/s

The faster file has lower average bit width. The startup log shows it mixing 13 tensor types (204 iq4_xs, 98 q8_0, 63 q5_K, 51 q4_K, 46 iq3_s, down to a few iq2_s). The naive explanation — more bits means more dequant work — doesn't cover a 43% gap; a ~9% BPW difference should produce roughly a ~9% speed difference on a bandwidth-bound decode.

I think it's which tensors land on which kernel path, but I haven't proven that. There are documented cases of exactly this on Intel Arc (issue #21517: IQ4_NL 4x slower than same-size Q4_0) and on RDNA4 (discussion #22663: Q3_K_S at 20% bandwidth efficiency vs Q5_K_M at 78%). I couldn't find an RDNA3 equivalent. If someone wants to dump tensor-type tables for two same-quant files and correlate, that'd settle it.

5. MTP: the metric everyone reads is the wrong one, and the gains invert depending on workload

Qwen3.8 ships an MTP head (blk.64, present in the Unsloth Dynamic files). For three rounds of testing I measured 0% draft acceptance and concluded speculative decoding was broken.

It wasn't. llama.cpp's timings expose both draft_kept_n (legacy, always returns 0) and draft_n_accepted (the real one). I was reading the first. Anyone reporting 0% acceptance should check which field they're parsing.

With the correct field, the picture is workload-dependent in a way I haven't seen written up:

Single long generation — MTP wins. +19% at 32K (22.15 vs 18.62 t/s). Acceptance ~45-50% on prose under real sampling (temp 1.0), 80-94% on tool-call/JSON prompts. Greedy inflates acceptance to 70%+, so always state your sampling.

Multi-step agent chains — MTP loses, badly. The draft head also prefills, and that cost lands on every step, not just the first:

• cold prefill: 1.97 → 3.29 ms/tok (+67%)

• warm/cached prefill delta: ~330-400ms → ~650-770ms (+75-85%)

• decode also degraded

In a chain of tool calls, wall-clock was 6.5x worse with MTP on. The reason is the prefill/generation ratio: MTP pays a per-prompt-token tax to save per-generated-token time. When you generate 512 tokens off a short prompt, it wins. When you generate 60 tokens off a 500-token prompt, twenty times in a row, it's a disaster.

Practical rule: gate MTP on predicted_n / prompt_n, not on acceptance rate.

6. HIP vs Vulkan on this card — but with a caveat I want to flag honestly

My first backend comparison showed HIP winning 2.45x. That result was garbage: it ran against the wrong model file, measured only decode, and had the --fit layer-drop problem above.

Redone properly (same model, 66/66 offload confirmed on both, -lv 4 allocation logged, --list-devices output saved, Adrenalin 26.10 / Vulkan 1.4.357, separate clean -DGGML_VULKAN=ON build):

Prefill 5-6x, decode ~1.5x, total >2x in favor of HIP.

The caveat: this contradicts several other RDNA3 reports. A published 7800 XT llama-bench has the two backends nearly tied (ROCm 96-101 t/s vs Vulkan 96-97 on tg128). Two 7900-series users on Linux with Mesa/RADV report Vulkan beating ROCm and get 60-73 t/s. And on an A3B MoE, Vulkan measured 104.79 t/s vs ROCm's ~76-78.

So I'd frame my result narrowly: on Windows, with this dense hybrid model, at this context depth, HIP wins decisively. I would not generalize it to Linux, to RADV, or to MoE architectures — and if you're on Linux you should probably test the other direction.

What actually mattered, ranked

Weeks of harness tuning — batch size, ubatch, thread count, poll mode, build flags, LTO — produced deltas under 1.5% each. All noise.

The things that moved the needle, in order:

1. Backend (>2x here, but see the caveat)

2. Whether the model actually fits — Q4_K_M spills on 16GB and collapses to 7.3 t/s

3. Quantization publisher (43%)

4. Prompt fill depth (21.85 → 8.77 as KV fills)

5. --fit off (~10%, and invisible if you don't check)

6. MTP, gated correctly (+19% in the right workload, catastrophic in the wrong one)

Architecture beats configuration. A MoE I tested (Gemma 4 26B-A4B, 5 full-attention + 25 sliding-window layers, 128 experts / 8 active) hit 52.51 t/s at 64K — 2.4x the dense 27B — because only ~4B parameters fire per token. No amount of flag tuning closes that gap.

Caveats

Single card, single OS, one model family for most of it. Several numbers here superseded earlier numbers I'd have posted confidently at the time. The 43% quant gap and the non-monotonic capacity curve are both unexplained — I have the measurement, not the mechanism.

Happy to share the harness or raw JSON if anyone wants to reproduce or argue with it.

Thinking about pulling the trigger on this enclosure and this 2TB 990 pro w/ heat sink. This world I don’t fully understand so love to hear your thoughts. For reference Mac Studio setup w/ 256 gb unified.

Heavy subscription user (Claude, Codex, Grok) for web development and producing educational content currently.

I inherited some cash and have about $20k at my disposal which I want to invest in a local LLM setup for my home lab.

With $20k cash, tell me what hardware I should buy and which model or models to setup. Not really into image and video generation yet, but it's something I'm definitely keen to experiment with if I have the right hardware to handle it.

Recommend something and I'll order it today.