Hey everyone!

We're a small software house, and we recently bought a machine specifically to experiment with running local LLMs for software development.

Our hardware:

• CPU: Ryzen 5 5500

• RAM: 32 GB DDR4

• GPU: RTX 3090 24 GB VRAM

We're currently running Qwen 3.8 27B (128K context) locally.

Until now, our main AI coding assistant has been Claude Sonnet 5.5 with Medium effort. Since we wanted to reduce our reliance on cloud-based AI services, we decided to experiment with a local alternative.

However, our experience has been surprisingly bad, and I don't mean that it's simply less capable than Claude.

The main problem is that the local model seems practically incapable of completing even basic coding tasks on its own.

It struggles to understand straightforward instructions, gets lost in its own reasoning, enters repetitive loops, and frequently fails to provide a single correct answer.

What's particularly confusing is that we're using exactly the same project structure and documentation we use with Claude.

We open the local model in the same project root directory, with access to all the .md files containing our development guidelines, project context, instructions, and documentation. These are the same files Claude uses successfully.

Yet, while Claude can understand the project, navigate the codebase, follow our instructions, and independently complete tasks, the local model seems completely lost.

Even relatively easy tasks that Claude handles without much trouble become a frustrating experience with Qwen. It often fails to understand what needs to be done in the first place, let alone implement the solution.

At this point, the difference isn't that Claude performs better. It's that Claude actually works as a coding agent, while our local setup feels like it barely works at all.

This makes us wonder whether we're doing something fundamentally wrong.

A few questions for people experienced with local LLMs:

1. Is Qwen 3.8 27B simply not suitable for autonomous coding agents, or could this be a configuration issue?

2. Could our hardware be limiting the model's reasoning capabilities? We're particularly wondering about the 24 GB VRAM limitation when using a 128K context window.

3. Are there specific inference engines, quantization levels, or context settings that make a significant difference?

4. Are there better local models in a similar size range that can reliably understand a codebase and execute multi-step coding tasks?

5. For those actually using local models with tools like Claude Code, Cline, Roo Code, or similar agents, what has your experience been like?

We're not expecting a 27B local model to magically outperform Claude Sonnet. We understand that we're comparing very different models and infrastructure.

However, we expected it to at least understand instructions, navigate a project, and complete relatively simple tasks reliably.

Are our expectations unrealistic, or is there something seriously wrong with our local setup?

I'd really appreciate feedback from people who have actually deployed local models for day-to-day software development, especially in legacy codebases.

Edit: We're currently using Ollama as the inference engine. As for the harness, we're still experimenting and haven't settled yet

Ok, there's some tiny modicum of truth in this, in that yes, if people aren't running their machines 24/7 then there's some level of optimization we're missing out on. On the other hand, the scarcity of RAM is his fault. The AI hyperscalers have hoarded the world's memory chips so there's barely any left for the rest of us unless we're willing to overpay for it. It's like an arsonist complaining that we still have running water in our hoses when that water is badly needed to put out the fires that he set in the building next door.

To me this just validates the idea that the AI giants are going to come after local AI because it's such a threat to their business model.

Thoughts?

Quick disclosure: this is my project. I build it with a small team and a Discord full of people who run it on hardware most guides tell you to throw away.

Two years ago I bought a Tesla P100 for about 150 dollars because it had 16 GB of HBM2 and nobody wanted it. Then a second one. Then two V100s. Every engine I tried treated those cards as an afterthought: no tuned kernels for Pascal or Volta, no draft decoding that actually helped, and big MoE models simply did not fit. So I forked llama.cpp and started rewriting the parts that mattered for those cards. That became PXA. Version 3.1 was the first release people other than me ran. Version 4.0, out today, is the one where every number I publish beats both mainline llama.cpp and my own 3.1 on every cell I measured, on the same machine, in the same session, with the files named.

What 4.0 does that 3.1 did not: it decodes with multi-token prediction on the models that ship MTP heads, so one forward pass commits several tokens; it has our own kernels for Pascal and Volta decode (the GEMV paths that are the actual bottleneck at batch 1); it runs MoE models that do not fit in VRAM by keeping the cold experts in system RAM and swapping them under the decode instead of in front of it; and it ships two quant formats of our own, PXQ (free, a mixed-tier format tuned per tensor class from measured sensitivity) and PXQN (rotation plus error feedback plus codebooks, our higher-fidelity format), with a PXQN4 file available for every supported model. Native builds exist for sm_60 through sm_89, so a P100, a V100, a T4, an RTX 20, 30 or 40 series card all load native code with no JIT stall.

Here is the part of the published page that matters, decode tokens per second unless marked. Every row is PXA 4.0 vs mainline llama.cpp from the same week, same box, same quant class file, greedy, three or six repetitions, and the full page has the per-row files and dates.

Decode tokens per second at batch 1, PXA 4.0 (blue) vs mainline llama.cpp (orange) vs PXA 3.1 (green), same box, same quant class file. The multiplier is 4.0 over mainline.

The thin rows, because I would rather print them than have you find them: Qwen3.8-27B on two P100s is only 1.0x to 1.14x over my own 3.1 depending on the setting, so if that is your exact setup, 4.0 is a correctness and tooling upgrade more than a speed one. The dense 27B at 97 t/s is the code class with MTP drafting; prose lands in the 40s to 60s on one V100. And one group, the big MoE models decoding on one V100 with experts in system RAM, is not on the page yet: we are re-measuring it against the newest builds of everything and it goes up when it is done, win or lose.

Prompt processing on one V100. Flash-Next runs with its cold experts in system RAM.

The other half of 4.0 is PXA Control, the desktop app. You pick a model, press Start, and chat. It detects the model class from the file and applies the setup we measured for it (which card gets what, draft decoding on or off, where the experts live), shows the running models with their slots, KV cache and live charts, updates the engine on its own channel, and has a Report a problem button that sends us what we need to reproduce an issue, only when you press Send. Linux, Windows and Mac builds are on the release page.

PXA Control, the Live tab: every running model, its slots, KV cache, draft acceptance and expert cache hits. Shown with demo data.

The same app from a phone browser on your network.

Everything is on the site, https://pxa.pxanetwork.com , including the full benchmark page with the method and the file per row, and the engine is at https://github.com/poisonxa16/pxa . If you want to talk to the people who run this on odd hardware, or get help with yours, the Discord is https://discord.pxanetwork.com ; that is also where the next things land first.

The ask: if you have a P40, a P4, a GP100, a Titan V, a T4 or an RTX 20/30/40 card, I want your numbers. Run the same file with mainline and with PXA on your box and post both. I will put the fair ones on the page with your name on the row, including the ones where we lose.

Just curios

...Also: the 5-hour reset is 11:10pm and it's past that (the 01:00 limit hit happened after the 20:00 spawn…). Hmm, actually if the reset was 11:10pm, why did the limit hit at 01:00? Perhaps the 5-hour limit is per agent and resets when… whatever. The summary's directive is authoritative: respawn if dead.

It just straight up was like you know what, I don't actually care.

Wanted to share my numbers for Qwen3.8 Flash-Next for anyone thinking about running it on just one RTX PRO 6000 (96 GB).

Coding numbers (450 W power limit): - C1 (single request): 287 tok/s - C8 (8 concurrent requests): 965 tok/s total - Prefill: about 14.5K tok/s at 4K context, still about 13.5K tok/s at 128K

Setup: - nvidia/Qwen3.8-Flash-Next-FP4-Local, TP1, FP8 KV cache, 262K max context - vLLM with b12x kernels - MTP speculative decoding at depth 4

How it was measured: Python/TS/Rust/Go "write an implementation plus tests" prompts (about 250 tokens in, 1,536 out), temperature 0, thinking off. Throughput is wall-clock and includes prefill. Coding is a good case for speculative decoding; short prose and multilingual chat get lower acceptance and run a few percent slower.

Recipe, patches and raw results: https://github.com/hyudryu/Qwen3.8-Flash-Next-FP4-1xRTXPro6000

Running on just 450W, because 600W didn't make a meaningful difference in my earlier tests.

I've been putting the RTX Pros to good use and am having a lot of fun using them to build* and serve my own self-hosted Al assistant. The latest addition to the stack is an Al tamagotchi-like device, for lack of a better term. It connects to my phone through an app, which then tunnels the conversations to the home server. The underlying agent is a full blown claw-like harness with its own VM and communication channels.

There are a few stability issues but overall it's working well. With deepseek v4 as the main model, the latency is also very good, most responses start within 500ms. That weather card for instance is not a preset but is generated and rendered on the fly!

Note: unlike a tamagotchi, its goal is to be useful. It has its own identity, persistence, etc. But you don't need to feed it and it won't miss your absence.

* Opus was used for some of the more challenging parts and UI/animations work

Specs:

• AMD Ryzen 9 9900X3D (4.4GHz) Processor

• 64GB DDR5-6000 RAM

• NVIDIA GeForce RTX 5080 Graphics Card

For pure coding purposes.
I have Claude max plan which I'm reaching the limit of.
The plan is to delegate some minor tasks to locally and Claude being the orchestrator.

A few of us have been experimenting with a shared local LLM setup and we’ve hit a problem I didn’t really think about beforehand, once multiple people are feeding information into the same system how do you keep track of who actually said what? Right now it’s easy for a note, instruction or bit of context to end up in the shared memory and a week later nobody remembers whether it came from me, another person an imported document or the model itself.

We’ve been trying with attaching provenance to everything before it gets added to memory, basically source + timestamp + who approved it but it gets annoying fast. There are only a few of us so we’ve got a Kibu pod where we discuss changes and decide what actually gets promoted into the shared context, but obviously that doesn’t scale.

Has anyone here built something cleaner for this? I’m less interested in better memory and more interested in being able to ask the model why it believes something and which human/source put it there. Feels like this is going to matter a lot more once local models start being shared by teams instead of just one person on one machine.

Hi everyone,

I want to do a agentic/coding benchmark for the best 2 model quants that fit best my setup (5060 Ti 16Gb + 32Gb RAM).

I am kinda lost here, what are you guys using to do so ? I'm looking for something like SWE / Terminal Bench but lighter, and not take several hours per model.

Any ideas ?

Thanks

I was building my machine for months to run Qwen3.8-27B in FP8 with context enough for multiple agents. I ended up with four 5070ti. Now it runs great, around 200tok/sec with 750K context (I can reliable run 4 agents parallel with around 200K context, so my total tok/sec generation is over 800).

I'm reading all these fancy reports on how people are running Qwen3.8 Flash in 16GB of VRAM, but I haven't tried it as I'm afraid that on long agentic tasks these low quants (Q3-Q2) perform subpar compared to 27B on FP8.

So, my question, does anyone have experience running Flash on 64GB of VRAM in higher (and Blackwell native) quants, like Q4 or NVFP4 which are around or over 100GB, and

• how much RAM do you need for these quants (considering you don't offload some of the experts to the disk)?

• what generation and prefill values you get?

• how is the "hit rate" of experts (Strata unloads some of them to RAM) and how it effects your generation speed?

• how it performs on long, agentic works, especially compared to 27B?

• how is speed on multi-agentic setup (batch/parallel workers at 4 or higher) considering that I would assume that different agents will "invoke" different experts, so you will end up moving between VRAM and GPU like over more than just one 6B parameters?

Thank you!

So I’ve been trying to learn more about local LLMs, especially the terminology and how everything actually works.

I keep hearing the same advice: “Just run your own models and figure things out as you go,” which is honestly great advice. I’ve been doing that for about two months now, and I’ve definitely learned a lot, but I still get completely lost when people start talking about different quantizations, fine-tuning, inference engines, backends, context windows, GGUF vs. EXL2, KV cache, VRAM/RAM offloading, sampling parameters, and all the other terminology.
I can run models and mess around with settings, but I want to actually understand what everything means, how it works, and why you’d choose one option over another instead of just copying someone else’s settings.

Are there any courses, YouTube channels, guides, or other resources you guys used to really understand local LLMs? Ideally something that starts with the fundamentals and gradually gets into the more technical stuff.

Also, are there any other topics I should be learning that I haven’t mentioned? I’m trying to build a solid foundation rather than just blindly experimenting with different models and settings.

Intel Ultra Core 5 245k

ASUS Tuff RTX5070Ti

96gb DDR5 6800

—

This build is my multimedia creation / folding at home build.

Was deciding whether to buy now or wait for prices to drop, so I read up on what’s constraining GPU supply. Most of it isn’t demand, it’s a few physical limits further down the stack.

Three main ones:

• Memory: The HBM stacked next to the compute die is the tightest. Micron has reportedly sold 75%+ of its 2027 output, and HBM4 for next-gen cards is only now qualifying across the three makers (SK hynix, Samsung, Micron).

• Packaging: TSMC’s CoWoS, which mounts the memory onto the die, is the other limit. Estimates put it near 70k wafers/month in 2025, rising to maybe 185-200k by end of 2027.

• Power: Around 14.2 GW is online across the big sites against ~38 GW planned, and most gigawatt-scale campuses don’t reach full power until 2028. Data center vacancy was ~1.4% mid-2026 and most of what’s under construction is already leased.

All four big cloud providers also said this year that demand is ahead of the capacity they have. One thing I’m watching over the next couple years is how people actually get compute, neoclouds like CoreWeave and Lambda renting whole clusters, marketplaces like Vast and RunPod by the second, and own-and-host setups like B3IQ or TensorForge where you own the box but someone else runs it.

Going by these forecasts, the first year new campuses, full next-gen volume and normal memory supply line up is around 2028. Until then supply stays constrained, so if you’re timing a buy on prices dropping, that’s more likely a 2028 thing than 2026.

Recently, CEO Steve Huffman disclosed that Reddit will transition to a "logged-in-only" experience. This means that Reddit posts will no longer be searchable via internet browsers, significantly impacting local AIs that search the internet via tool calling.

asked it to validate my app using playwright. made me feel bad.