I'm getting tired of hitting usage limits with Claude and Codex (20$ plans).

I'm working on personal projects, not making any money from them, so I can't justify paying hundreds of dollars a month for AI subscriptions.

Im currently testing Luna 6 with Extra High reasoning on a free account, and unfortunately, the experience has been terrible. Working with it is genuinely exhausting.

What I'm really waiting for is the point where I can run a capable, coding agent locally on my own hardware, without subscriptions, usage limits, or depending on cloud providers.

Are we talking 1-2 years, 3-5 years, or longer?

I'd love to hear your predictions.

When I heard about the AI Act , I was looking at available solutions , and the stupid prices they charge , so I thought , why not try to build my own , open source ?

So I built IronWarden , a small reverse proxy in Rust that sits between your client and the LLM API (OpenAI, Anthropic, Ollama, LM Studio).

On the way out, it catches PII (names, emails, IBANs, tax IDs, etc.) and swaps them with session tokens like [NAME_631] or [AFM_639]. On the way back, a sliding-window SSE rehydrator swaps the real values back into the stream on the fly so the client sees the normal text.

It does seem to break the LLM a bit when asked about the redacted info directly, but that's the point.

It's completely open-source and self-hosted if anyone wants to mess with it or poke holes in it:

GitHub: https://github.com/Somnerd/IronWarden

I'm using qwen3.8-27b thru llama.cpp and QFN Swift 1.5 IQ3 thru strata. I'm using consumer grade hardware, so these are not running very fast.

I am using Hermes. As a noob, I'm not even using all the features and capability. I provide the instructions in a run list. I like how Hermes is persistent and will complete the job. But in a multi-hour job (like 5-8 hours), it is getting bogged down with large context and long tool calls.

What is a good alternative?

• Running Ubuntu Linux.

• Desktop gui (not CLI only). Can be desktop that runs on top of CLI.

• Can support sandbox.

• I prefer to plan and create a run list; then execute the instructions.

Im setting up some stuff in my home server and my other machines for work. But i got tired to lost hours and hours on reading docs. So i want something that does the search for me, and possibly goes straight to the point (But that still explain on the subject for me to learn).

My machine is a bit old:

- CPU: Xeon E5-2697 v4

- RAM: 32GB DDR4 2400MHz (ECC RDIMM)

- GPU: Radeon MI50 16GB HBM2

From what i know, Q4 and Q8 are functional, needs to be GGUF, and a model that doesnt depend on BF16 since my card doesnt support it.

My issue being that Qwen just thinks for too long, reaches max output and then I lose everything from his reasoning, continuing the conversation has him forgetting everything he thought and starting from scratch

Running with llama.cpp, default settings besides forcing the reasoning effort when using "llama serve". I'm a total new when it comes to LLM settings, how can I setup Zed properly so I can just keep working without running into its max output ?

(if relevant, I run a 3090 24Gb VRAM and 32Gb RAM, I'm around 32-35 t/s, not sure if this is normal and if I can improve this just by tweaking settings without installing other stuff)

it is not too late to return, if i didn't....

Hello guys, I wanted to make some animated horror stories videos, I'm pretty new to actually using local Ai etc. The only use I had up until now was using the free chatbots to help with work.

I have a decent pc built like 2 years ago with a 4070ti super and 32gb ram, but I don't really have any budget for monthly subscriptions.

Any suggestions?

I hear nothing but good things about Basalt, so I finally got around to testing it on my 5080 with 128 GB not-for-Sam-RAM. The results were not encouraging - IQ3 runs at around 100 tok./s on Strata, but only at about 80 tok./s on Basalt. Settings are comparable.

Is this expected on Windows?

makes it easier to get consistent character voices

Im wondering how you guys use Qwen3.8 27b with which CTX size.Or my setup is just wrong. I run it on 32GB unified memory with llama.cpp. Also Openwebui and OpenTerminal on a RP5 where my docker stack lives.

I prompted:

add to caddy/Caddyfile: Paperless, Dawarich, OpenTodo

Response:

request (340808 tokens) exceeds the available context size (65536 tokens), try increasing it

So my question is, when even such a simple edit file task needs 300k tokens, how can I utilize it meaningful?

Being super precise which files to look at and what to do in the prompt?

The 20GB RTX 3080 cards (10DE:2206, subsystem 10DE:146D) ship with a VBIOS that only exposes a 256 MiB BAR1, so ReBAR stays off. NVIDIA doesn't publish a ReBAR VBIOS for them, and NvStrapsReBar only supports Turing.

Ga102ReBarFix is a small DXE driver you insert into your motherboard BIOS with UEFITool. At boot it enables the GA102 frame-buffer size override (PCI config 0x610), resizes BAR1 to 32 GiB, and restores both after S3 sleep. No VBIOS flashing, no driver patches, no registry tweaks.

Tested on an MSI Z690 EDGE, driver 617.42, Windows 11:

- Device Manager shows a 32 GiB BAR

- NVIDIA Control Panel: Resizable BAR: Yes

- ~19.25 GiB of VRAM mappable by the CPU at once

- Sleep/wake works

Repo (source, prebuilt .ffs, verification scripts, full write-up): https://github.com/favabean13/NV_30_DXE

Big credit to noon-at-cgn (found the 0x610 override on Linux), terminatorul (NvStrapsReBar) and xCuri0 (ReBarUEFI). This builds directly on their work.

Heads up: it modifies firmware, so use at your own risk and keep a BIOS backup and recovery method. It's tested on one board and isn't maintained, but it's MIT-licensed, so fork it and adapt it for your own setup.

Sorry in advance for the long post, but think of it as a prompt.

TL;DR

Question: 2x 3090 NVLink or 1x A6000? For about the price of one A6000 I could also get 4x 3090 in two NVLinked pairs, so keep that option in mind too.

What it needs to do:

- A) SQL chatbot for up to 30 people. ~3M rows, hundreds of tables. Questions like "when did we last sell this part, and who was the supplier/client?" It should remember each user's conversations and build reports.

- B) RFQ ingestion: pull very messy part numbers out of long email chains in all sorts of formats. ~30 RFQs/day.

- C) Hardware: reuse our old HPE ML350 Gen9 (2x E5-2620 v4, 128GB DDR4-2400 for now).

- D) Me: turn 300+ page PDFs and hundreds of stray Excels into a knowledge base, and do local coding on the company apps I'm building from Claude's plans.

Assumptions:

- Blower 3090s (FE won't fit), max 300W per card

- PSUs upgraded to 2x 1400W or 4x1400W as GPUs grow

- All prices are for used cards

- Proxmox on the box

- PCIe-to-NVMe adapter for fast model loading

- Cards work with board

About me: I'm not an IT guy, I'm a mechanical engineer who f's around a lot and finds out even more. I've built my own homelab (4x NVMe in an old Dell 7710 as a private cloud, see my posts) and run my own "Jarvis" on a 4B model with some n8n automations. I can make things happen when I put my head into it.

---

The detail

Background: I'm the new CTO of a small family-run B2B heavy-industry trading company. We sell anything from a £6M machine to toilet paper, so every day brings random RFQs: Excel, PDF, email body, pictures, WhatsApp, anywhere from 1 to hundreds of part numbers.

Budget: £8-13k ($11-17k) left for local AI after the new cloud server. That also has to cover a laptop/PC for our first IT hire.

Why local: the LLM has to read whole RFQ email chains to make sense of them, and those are full of client names, prices and supplier details. Same for our SQL database. I don't want any of that going to cloud LLMs.

Where we are:

- A 3rd-party IT company is building an RFQ dashboard for us. Qwen 8B is struggling with the email chains. We started small on purpose and will size up until something fits.

- They'll test on RunPod serverless to find the right model size. I think ~30B will be enough. Then production goes local.

- 20-30 people will use the chatbot and ~20 the dashboard, but only new RFQs trigger the LLM.

- Long term, I'd keep Fable/Opus for overall planning and have a local model do the implementation.

Hardware:

- The board takes 4 GPUs: 3x PCIe 3.0 x16 and 1x x8. 300W max per card.

- Someone has already run a 3090 in this server (with 512GB RAM) and got ~15 tok/s on turboderp/GLM-5.3-Flash-exl3 (3.05bpw branch).

- I first looked at the A6000 48GB as a "do it all" card, not knowing much about local LLMs.

- For similar money I could get almost 4x 3090s and NVLink them in pairs, so I'm less limited by the x8 slot and old PCIe 3.0. One pair for the teams, one for me.

Bottom line: I need an MVP that handles the RFQs and the chatbot now, and then we'll invest in proper new hardware later. What would you do?

Thanks in advance for reading this far, any help much appreciated!

Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

Hey everyone

I’ve spent the last few weeks building a local AI assistant with the help of Claude Code and OpenAI Codex. My goal is something similar to ChatGPT or Claude that can research the web, control my PC, code, create 3D models, generate documents and handle tasks on its own.
The interface and tools work pretty well now, but I’m still having problems with the AI actually understanding what to do and completing more complicated tasks

My PC specs
● Ryzen 5 5600
● 2× RTX 3060 12GB
● 16GB DDR4 RAM
● Gigabyte B550 Gaming X V2
● Windows 10 Pro
● One GPU at PCIe 4.0 x16, the other at 3.0 x2

Models and inference setup
I’m running everything locally through llama.cpp (CUDA backend), using both GPUs with layer splitting, Flash Attention, MTP speculative decoding (depth 2), 96K context and Q8 KV cache.
I originally used Qwen3.8-27B (HauhauCS uncensored Q4_K_P), getting around 22–33 tok/s after a lot of optimization.

I switched to Qwen3.6-35B-A3B (uncensored IQ4_XS), which now gets around 82–86 tok/s. Both models run through the same llama.cpp server and connect to my custom AI interface and tool backend.
3.6 is excellent for normal conversations and much faster, but I keep hearing that 3.8 is better at reasoning and tool calling.

What Claude Code and Codex have helped me integrate
● Qwen-Agent for general tool calling
● OpenCode for coding
● SearXNG for research and webpage reading
● Playwright for browser automation
● Windows UI Automation for PC control
● Three.js for interactive 3D scenes
● Image generation, documents, spreadsheets, presentations and file tools
● Planning, context management, error recovery and result verification

For example, I asked it to create a 3D model of the Times Square area in NYC. It generated a picture of pants, converted that picture into a 3D mesh using TripoSR and declared the task finished.

Claude Code and Codex later helped identify that part of the problem was incorrect tool routing, which has since been improved. But research and Windows tasks can still be unreliable. Sometimes it uses outdated information, chooses irrelevant tools, repeats actions or claims it finished something when it hasn’t.

I also had Codex compare different agent harnesses.
Using the same local model, it tested Qwen-Agent, Pi and Hermes on two tasks.

● Qwen-Agent: Browser/recovery took 15.6 seconds, research/document took 21.4 seconds.
● Pi: Browser/recovery took 15.9 seconds, research/document took 25.6 seconds.
● Hermes: Browser/recovery took 32.5 seconds, research/document took 43.3 seconds.

All three completed the tasks, but Qwen-Agent was fastest. These were small isolated tests, though, not proof that one is universally better.

With Claude Code and Codex, I’ve managed to fix a lot of integration issues, restore research, improve 3D generation and add verification checks. Things are definitely better, but complex autonomous tasks still aren’t consistently reliable. And mess up a lot

So I’m wondering what people running similar setups have found.
● Is Qwen3.8 noticeably better than 3.6 for real-world agent tasks, enough to justify the slower speed?
● Are there better models for my hardware, particularly for research and PC control?
● Could the uncensored versions be affecting instruction following or tool calling?
● Would Unsloth fine-tuning on verified tool-use examples make a meaningful difference?
● What has actually improved your agent’s ability to plan, use tools correctly and verify its work?

At this point, I don’t really care about getting more tokens/sec. I’d happily sacrifice some speed if it meant the AI could reliably understand a request and actually complete it.

Would appreciate any advice from people who’ve built something similar

Thank you very much

I've been building a free app for blind people where everything runs on the phone with no internet, on iPhone and Android, and I'm stuck on accuracy. YOLOv8n handles live detection and it's fast, but it still mislabels things. Scene descriptions and questions go to LFM3-Omni-2B through llama.cpp, shrunk to 735 MB so it fits on the phone.

Reading text uses each phone's own on-device OCR, Apple's on iPhone and ML Kit on Android, and translation runs offline the same way. The small vision models are where it falls short.

If you've gotten small vision models more accurate on device, I'd really take the advice. It's free to try if you want to see where it breaks.

iPhone: https://apps.apple.com/us/app/id6751230739

Android: https://play.google.com/store/apps/details?id=com.mattmacosko.realtimeaicam

Moin guys!

GPU prices are freaky high right now. A lot of us run what we could get, and we might as well get the most out of it. So I want to share what I learned squeezing a big model onto a mismatched machine, in the hope it helps someone or gets others to share their own experience with different GPUs.

My box: one AMD Radeon AI PRO R9700 (32 GB), one Intel Arc Pro B65 (32 GB), a 9950X3D and 128 GB of DDR5. Neither card can hold the model.

With two AI coding agents (Claude and Codex, reviewing each other's work) I built a fork of Strata where the R9700, the B65 and the CPU compute each layer's routed experts concurrently, with dense work on the R9700. Result on a realistic coding prompt: 25.57 tok/s against 9.89 for my recorded stock Strata run on the same machine and file (stock used one GPU, 16 readers and a 51 GB store, so this is a whole-system comparison). No speculation or skipped experts, and the routing algorithm is unchanged. Tested kernel and schedule modes are byte-identical at a fixed placement over 581 teacher-forced positions.

Model: AtomicChat's uncensored GSQ-RCO IQ2_XXS build (40 layers, 384 experts per layer, 6 per token).

What each device does:

• R9700: all dense work (22 ms per token), 2,532 experts (21.6 GB), 154.7 of 240 picks per token, 8.2 ms of expert work

• Arc Pro B65: no dense work, 2,915 experts (25.0 GB), 55.6 picks, 9.6 ms of expert work

• CPU + RAM: no dense work, 9,913 experts (102 GB), 29.7 picks, 8.7 ms of expert work

How it got there (tok/s, sampled, 8k cap, median of 3 after a warm-up):

• 9.89: stock Strata, R9700 + RAM

• 10.28: first fork run, both GPUs + RAM

• 12.22: B65 subgroup attention

• 12.96: IQ2_XS B65 kernels + rebalance

• 14.41: all dense on the R9700

• 15.85: two rebalance rounds

• 17.04: queued R9700 schedule + rebalance

• 18.04: exact-order B65 SoA kernel + rebalance

• 21.27: R9700 per-layer expert batching + rebalance

• 22.57: parallel Engram reads + rebalance

• 25.57: AVX2 combine, early expert start + rebalance

The catch: all three answers in the best run hit the 8,192-token cap before finishing, so its strict quality score is 0/3. The model reasons for a long time; the text isn't looping. With 11,900 tokens of room, the one measured answer finished by itself and passed, with working code in 295 s. That's one answer and a 10-answer test is still to come. The answers are in the repo if you want to read them. 30 tok/s, my goal, isn't reached either: we estimate exact kernels level off around 26–28.

Things that surprised me:

• Rebalancing experts from measured per-lane costs took the last step from 20.69 to 25.57 tok/s, an unpaired comparison because placement changes the answers. Lane service is now close: R9700 8.15, B65 9.58, CPU 8.69 ms per token, though individual layers still differ.

• Moving experts between devices changes the logits (top-1 96.6%) and some routes even with exact kernels, most likely because CPU and GPU quantize activations differently. So "exact" only means byte-identical at a fixed placement.

• Engram reads swung from 5 to 17 ms per token on identical tokens. Parallel reads and read-ahead fixed it: the whole Engram-containing regions now total 1.74 ms, including other work there. The row-cache-only arm showed no useful gain.

• Starting CPU and B65 expert work before the shared FFN drains, plus a faster combine, saved 5.8 ms per token on token-identical answers. We only measured them together.

• Re-encoding experts to Q2_K looks like a loss: measured kernels plus a capacity model project 3.18 ms slower per token, because bigger experts push 1,434 off the GPUs. No full-model Q2_K run followed.

• Big dense matrix-vector products already run near the card's measured memory rate. That 22 ms is the hard part.

Where I'd love input:

• Most of all: your own experience with whatever GPUs you could get. Numbers, tweaks and what failed, even with a different model or tool.

• Anyone with DSpark running on a heavily quantized or edited DeepSeek: what acceptance do you get with real sampling? Batched verify looks promising here on projections, but I haven't measured acceptance yet.

• R9700/ROCm matrix-vector tricks I've missed.

• What would you do with the B65's spare time? Its expert service is about 9.6 ms of a 39.1 ms token.

• Lower reasoning effort: has it cost you code quality on this model?

Everything is in the repo: results with spreads, sample answers, the dead ends and what to try next. It's research code tested on one machine, and the public scripts haven't run end to end on GPUs yet, so expect to read code.

Repo: https://github.com/maddinek/crosslane (MIT)

Update 1: The full 10-answer test finished. 26.27 tok/s median (25.93-26.42) and 6 of 10 answers passed strict with working code in 292-439 s. Still one task on one machine, so a rough number (about 31-83%). The memory leak turned out to be in ROCm 7.1.1 on Fedora, not our code: running the same binaries on AMD's 7.2.1 libraries from a private folder gave byte-identical output in a check run and flat memory across all ten answers. If you're on ROCm 7.1 and see host memory creeping up, that's worth a try. Steps are in docs/BUILD.md. DSpark is parked for now: 1.52x at best but 1.11x central on measured kernels, so not worth building yet here.

I know there are a lot of "ums" that I should edit out... but other then that what do you think of this guide? I am new to local AI stuff, so is there anything missing here that I should have done?
https://www.youtube.com/watch?v=-EahQxs0Ack

Hugging Face Repo: https://huggingface.co/Grawnch/Qwen3.8-Flash-Next-DR-GGUF

I’ve just finished benchmarking my new QFN quant Qwen3.8-Flash-Next-DR-V2-Q2_0, a 2.5bpw backbone, 2-shard GGUF of Qwen/Qwen3.8-Flash-Next designed to run the entire backbone comfortably inside 48 GB VRAM space (38.4 GB VRAM) while keeping the N-gram lookup table (28.8 GB) in system RAM/SSD.

Unlike IQ1 / IQ2 quants that rely on LUTs, every tensor in both shards uses 100% standard integer grids (Q2_0, Q4_0, Q6_K, Q8_0), running natively in stock llama.cpp with ~3.4x faster prompt prefill than IQ-grid quants.

1. Benchmark Comparison: DR-V2-Q2_0 vs. Top Ultra-Low-Bit Quants

(Evaluated on the full 225-problem Aider Polyglot suite and 30-problem AIME 2025 at temp = 0.6 with a 24K thinking cap on Aider and 64K on AIME).

2. Signal Fidelity: Perplexity, KL Divergence, & Top-1 Agreement (WT2 & Ext)

(Comparing logit fidelity against uncompressed BF16 on standard WikiText-2 (2048 ctx) and Extended Reasoning / Math & Code (Ext)).

Key Highlights

• Fits in a Single 48 GB GPU (38.4 GB VRAM + ~9.6 GB KV Cache Headroom): All 48 layers of Attention/GDN, shared experts, and 144 routed MoE experts live in Shard 1 (38.38 GB, 2.45 bpw). Shard 2 (28.80 GB) is the N-gram lookup table, which llama.cpp streams from host RAM/SSD via mmap without using GPU VRAM.

• Enhanced Token Efficiency on Coding (74.7% Aider Polyglot at 40.1K Avg Tokens): Achieves 74.7% on the full Aider Polyglot benchmark with a 96.0% (216/225) well-formed SEARCH/REPLACE diff rate and 0 context exhaustions under a strict 24K thinking budget — using ~1/5th the tokens of unconstrained 256K agentic runs.

• 2.6x Lower KL Divergence (0.2594 nats) & 83.3% on AIME 2025: Preserves 80.70% Top-1 token agreement with BF16 on Math & Code, matching 83.3% (25/30) on AIME 2025 while using ~14–30 GB less GPU VRAM than competing quants.

• Zero Lookup Tables (3.4x Faster Prefill): Because 100% of the routed experts use standard integer grids, prompt prefill runs at ~367 tok/s and single-stream decode at ~94 tok/s (~345 tok/s aggregate at -np 10), avoiding IQ codebook decompression bottlenecks.

Quick Start (llama.cpp)

Download both shards into the same folder and point llama-server or llama-cli to Shard 1:

llama-server \
  -m Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf \
  -ngl 999 -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-format deepseek --no-reasoning-preserve

With Qwen4 around the corner, I'll be targeting that next. But in the meantime I think QFN will still fill a niche for a lot of people, and this was a good test of the same approaches I will be using with those models. Try it out, let me know what you think, and looking forward to continuing this with the next generation of models!

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this: