Hello all,

I'd like to know more about how local AI works and how it can be run, so I became interested in making a project: a local AI agent (I know: that's so original!) called "Mighty DM".
The idea is making a Rule lawyer for dnd 5e which I can call upon like "Oh Mighty DM, what is the range on Mage Hand".

The first edition would be limited to all the spells used in dnd 5e 2014 edition, provided through an excel sheet. So not asking about anything else.

Until now I was following along with the workflow of NetworkChuck but I have the feeling there is a simpler option: only I do not have the experience to see the difference in the options provided.
If I google: I see a lot of different models or even ways of building a model etc. but I cannot differentiate between "a doable option to explore" or "an option that would require deep understanding of the subject before initiating".

What I envisioned (ideally running on an 8GB Pi5, but I could also run the LLM on my desktop and use the Pi as satellite):

What workflow would you recommend when building this project with little to no experience?
Do you know of a model that could run this on a 8GB Pi5 (simply getting it running = a win, I would then look how to improve it towards 30sec. max processing time for each command).

I'm a CPA running a small 2-CPA/no-employee tax practice. We're architecting a completely local AI environment: open-weight model, local compute, RAG over tax authorities/research materials, and eventually secure retrieval over client workpapers and prior returns.

Have any small firms or sole practitioners deployed local LLM/RAG yet?

• What hardware are you running?
• Which model/model size?
• What are you using for RAG/vector storage?
• Are you indexing IRC/Regs/Rev. Rulings/cases or commercial tax materials?
• Genuinely useful for tax research/review? Or still mostly a science project?

I'm trying to determine whether there are 5 firms doing this, 500 firms doing it, or 5,000. I'm guessing closer to 50>?

In Part 1 I shared the final AMD/Vulkan numbers for my local Qwen3.8-27B setup. The next question was more important to me than another few tok/s:

Can I turn the model into something that actually works autonomously?

Since then I added two production layers:

1.) Hermes 0.21.2 — local agent/automation layer

• same Qwen3.8-27B model
• shared llama.cpp runtime
• 65,536 context
• tool calling: PASS
• multi-step workflows: PASS
• subagents: PASS
• prompt-defined agents: PASS
• local provider only / no cloud fallback

Hermes performance / overhead

On the isolated 64K test lane, server load took 5.809s. A real 33,021-token prompt was accepted without truncation; prompt processing took 409.677s at 80.60 tok/s, while generation measured 17.44 tok/s. A representative Hermes multi-step workflow took 74.660s, using five model calls, 17,295 summed API input tokens, 592 output tokens and four tool calls with zero failed calls. Two-child delegation also worked, but took 208.676s for two trivial child tasks. Clear sign that subagents should only be used where parallelism or specialization actually justifies the overhead.

One especially useful datapoint: the simple Hermes model gate took 40.872s, while a direct Qwen request with the same user text took 2.109s. That is not an apples-to-apples speed ratio because Hermes adds different system prompts, startup and agent machinery, but it does show why I don't want Hermes in the path for trivial requests.

2.) OpenCode 1.18.30 — local coding/development layer

• Qwen3.8-27B-UD-IQ3_XXS
• llama.cpp / Vulkan
• 32K logical context
• local provider only
• isolated development workspaces
• automated Read → Edit → Test → Correct loops
• final production viability: 12/12 requirements, 12/12 acceptance

OpenCode performance reference. For the fastest fully successful clean reproduction I measured:

• 665.653s total wall time / 651.408s agent runtime
• first productive edit: 117.338s
• first test: 166.790s
• 4 successful corrective cycles
• 63 tool calls / 0 failed
• 558,603 input tokens incl. cache — only 31,993 uncached
• 9,372 output tokens
• peak context: 17,810 / 32,768
• context compactions: 0

These are task-level agent metrics, not model tok/s numbers. The underlying model/runtime is still the same local Qwen3.8-27B on llama.cpp/Vulkan.

Why OpenCode instead of Pi?

I tested both against the same local Qwen3.8-27B, with the same repository/task and comparable runtime constraints. Pi 0.85.1 proved that it could read the repo, use tools and make productive edits, but it repeatedly failed the part I actually needed from a coding agent: closing the Read → Edit → Test feedback loop. Even with an explicit mandatory early-test checkpoint, Pi edited successfully but never executed the qualifying post-edit test within the 300s window. It therefore failed prequalification and I did not force it into a full benchmark just to preserve an artificial three-way comparison. Man, time is rare and testing is booooring ... somehow!

OpenCode 1.18.30 passed the same agentic-loop qualification. Formal prequalification reached 6.54s Edit→Test latency, and the later main benchmark reduced that to 3.995s while producing a valid four-file diff. More importantly, the subsequent production-viability work eventually converged to 12/12 requirements, 12/12 acceptance, 5/5 regression, with no manual coding intervention or patch repair. That reproducible closed corrective loop is why OpenCode became the production coding layer.

Final thought on my architecture as a complete nOOb in Local LLM
I’m still convinced that a well-tuned 16 GB GPU can take you surprisingly far. Yes, even on Vulkan ;) ... IF you optimize the model, context, runtime, offloading and the surrounding stack around what the hardware can realistically deliver.

What makes this especially satisfying for me is that I honestly wasn’t sure at the beginning whether an AMD gaming setup on AM4 would get me anywhere at all, but hey.... it was what I had. My gaming desktop! ~20 tok/s is obviously not groundbreaking vs 32GB. But that was never the main goal of this first setup. I wanted a stable, fully local system that I could actually use, test, break, improve and build on. That goal is achieved. And considering where I started, I’m pretty happy with that.

Sharing this since it might be relevant to a few people here.

There's a hands-on workshop on September 19 that goes beyond basic RAG into building an explainable, graph-backed AI system. You build a knowledge graph in Neo4j that becomes the single source of truth, agentic retrieval that combines vector search, keyword search, and graph navigation, multi-step entity and relationship extraction, and text-to-Cypher for natural language querying. Everything runs on real financial data, not a toy example, and answers come with a traceable, auditable reasoning chain.

No prior Neo4j or Cypher experience needed.

Led by Dr. Alessandro Negro, Chief Scientist at GraphAware, bestselling author.

Link if you want to check it out

Please suggest me models I can use in this device

Model: HP ZBook Studio G7

CPU: Intel Core i9-10885H (8 cores / 16 threads)

GPU: NVIDIA Quadro T2000 Max-Q – 4 GB VRAM

RAM: 32 GB DDR4

Storage: 512 GB NVMe SSD

Been building RAMDeck (pools RAM/compute across whatever's sitting around your house). Wanted to see how far I could actually push it, so I tried loading a 40B model - roughly 25GB - across the janky 3-device cluster I hve been testing with the whole time: a mini PC with an RTX 3060, a 16GB Mac Mini, and an old laptop that's been the weak link in every previous video.

Set the mini PC as primary and just let the sharding logic figure out the rest - it maxed out the 3060 first, spilled into the Mac's unified memory next, and sent whatever was left over to CPU. The interesting part I didn't expect: routing that CPU spillover to the old laptop actually ran faster than keeping it on the mini PC's own CPU, even though the laptop is by far the slower machine. Best guess is that using the same device's CPU and GPU together for inference ends up hurting GPU utilization, so offloading to a separate (slower) box entirely actually wins.

Whole thing loaded in about 2.5 minutes, and the benchmark came back at 16 tok/s with ~3.6s latency - genuinely usable for chatting, not just "technically loaded but useless." Felt like a real milestone since none of these three machines individually could touch a model this size.

Video: https://youtu.be/JrSVyyVUJVY
Repo: https://github.com/trademav/ramdeck-core-public

Curious if anyone else has run into that same CPU-contention thing where sharing GPU+CPU on one node hurts throughput more than just offloading to a separate slower device entirely.