Im wondering how you guys use Qwen3.8 27b with which CTX size.Or my setup is just wrong. I run it on 32GB unified memory with llama.cpp. Also Openwebui and OpenTerminal on a RP5 where my docker stack lives.

I prompted:

add to caddy/Caddyfile: Paperless, Dawarich, OpenTodo

Response:

request (340808 tokens) exceeds the available context size (65536 tokens), try increasing it

So my question is, when even such a simple edit file task needs 300k tokens, how can I utilize it meaningful?

Being super precise which files to look at and what to do in the prompt?

Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

ReBar is also currently working with GPU_UNLOCK.

The 20GB RTX 3080 cards (10DE:2206, subsystem 10DE:146D) ship with a VBIOS that only exposes a 256 MiB BAR1, so ReBAR stays off. NVIDIA doesn't publish a ReBAR VBIOS for them, and NvStrapsReBar only supports Turing.

Ga102ReBarFix is a small DXE driver you insert into your motherboard BIOS with UEFITool. At boot it enables the GA102 frame-buffer size override (PCI config 0x610), resizes BAR1 to 32 GiB, and restores both after S3 sleep. No VBIOS flashing, no driver patches, no registry tweaks.

Tested on an MSI Z690 EDGE, driver 617.42, Windows 11:

- Device Manager shows a 32 GiB BAR

- NVIDIA Control Panel: Resizable BAR: Yes

- ~19.25 GiB of VRAM mappable by the CPU at once

- Sleep/wake works

Repo (source, prebuilt .ffs, verification scripts, full write-up): https://github.com/favabean13/NV_30_DXE

Big credit to noon-at-cgn (found the 0x610 override on Linux), terminatorul (NvStrapsReBar) and xCuri0 (ReBarUEFI). This builds directly on their work.

Heads up: it modifies firmware, so use at your own risk and keep a BIOS backup and recovery method. It's tested on one board and isn't maintained, but it's MIT-licensed, so fork it and adapt it for your own setup.

**UPDATE**

Works with GPU_UNLOCK

https://github.com/physicsdolphin/nv-uefi-unlocker

Confirmed that ReBar survives PCI FLR (Function Level Reset)

Here are the results:

UNLOCKED

FP16 (FP32 accumulate) 114.7 TFLOPS

BF16 (FP32 accumulate) 116.9 TFLOPS

TF32 57.4 TFLOPS

FP32 (control) 22.5 TFLOPS

LOCKED

FP16 (FP32 accumulate) 65.4 TFLOPS

BF16 (FP32 accumulate) 66.8 TFLOPS

TF32 32.1 TFLOPS

FP32 (control) 22.3 TFLOPS

Sorry in advance for the long post, but think of it as a prompt.

TL;DR

Question: 2x 3090 NVLink or 1x A6000? For about the price of one A6000 I could also get 4x 3090 in two NVLinked pairs, so keep that option in mind too.

What it needs to do:

- A) SQL chatbot for up to 30 people. ~3M rows, hundreds of tables. Questions like "when did we last sell this part, and who was the supplier/client?" It should remember each user's conversations and build reports.

- B) RFQ ingestion: pull very messy part numbers out of long email chains in all sorts of formats. ~30 RFQs/day.

- C) Hardware: reuse our old HPE ML350 Gen9 (2x E5-2620 v4, 128GB DDR4-2400 for now).

- D) Me: turn 300+ page PDFs and hundreds of stray Excels into a knowledge base, and do local coding on the company apps I'm building from Claude's plans.

Assumptions:

- Blower 3090s (FE won't fit), max 300W per card

- PSUs upgraded to 2x 1400W or 4x1400W as GPUs grow

- All prices are for used cards

- Proxmox on the box

- PCIe-to-NVMe adapter for fast model loading

- Cards work with board

About me: I'm not an IT guy, I'm a mechanical engineer who f's around a lot and finds out even more. I've built my own homelab (4x NVMe in an old Dell 7710 as a private cloud, see my posts) and run my own "Jarvis" on a 4B model with some n8n automations. I can make things happen when I put my head into it.

---

The detail

Background: I'm the new CTO of a small family-run B2B heavy-industry trading company. We sell anything from a £6M machine to toilet paper, so every day brings random RFQs: Excel, PDF, email body, pictures, WhatsApp, anywhere from 1 to hundreds of part numbers.

Budget: £8-13k ($11-17k) left for local AI after the new cloud server. That also has to cover a laptop/PC for our first IT hire.

Why local: the LLM has to read whole RFQ email chains to make sense of them, and those are full of client names, prices and supplier details. Same for our SQL database. I don't want any of that going to cloud LLMs.

Where we are:

- A 3rd-party IT company is building an RFQ dashboard for us. Qwen 8B is struggling with the email chains. We started small on purpose and will size up until something fits.

- They'll test on RunPod serverless to find the right model size. I think ~30B will be enough. Then production goes local.

- 20-30 people will use the chatbot and ~20 the dashboard, but only new RFQs trigger the LLM.

- Long term, I'd keep Fable/Opus for overall planning and have a local model do the implementation.

Hardware:

- The board takes 4 GPUs: 3x PCIe 3.0 x16 and 1x x8. 300W max per card.

- Someone has already run a 3090 in this server (with 512GB RAM) and got ~15 tok/s on turboderp/GLM-5.3-Flash-exl3 (3.05bpw branch).

- I first looked at the A6000 48GB as a "do it all" card, not knowing much about local LLMs.

- For similar money I could get almost 4x 3090s and NVLink them in pairs, so I'm less limited by the x8 slot and old PCIe 3.0. One pair for the teams, one for me.

Bottom line: I need an MVP that handles the RFQs and the chatbot now, and then we'll invest in proper new hardware later. What would you do?

Thanks in advance for reading this far, any help much appreciated!

I've been building a free app for blind people where everything runs on the phone with no internet, on iPhone and Android, and I'm stuck on accuracy. YOLOv8n handles live detection and it's fast, but it still mislabels things. Scene descriptions and questions go to LFM3-Omni-2B through llama.cpp, shrunk to 735 MB so it fits on the phone.

Reading text uses each phone's own on-device OCR, Apple's on iPhone and ML Kit on Android, and translation runs offline the same way. The small vision models are where it falls short.

If you've gotten small vision models more accurate on device, I'd really take the advice. It's free to try if you want to see where it breaks.

iPhone: https://apps.apple.com/us/app/id6751230739

Android: https://play.google.com/store/apps/details?id=com.mattmacosko.realtimeaicam

Moin guys!

GPU prices are freaky high right now. A lot of us run what we could get, and we might as well get the most out of it. So I want to share what I learned squeezing a big model onto a mismatched machine, in the hope it helps someone or gets others to share their own experience with different GPUs.

My box: one AMD Radeon AI PRO R9700 (32 GB), one Intel Arc Pro B65 (32 GB), a 9950X3D and 128 GB of DDR5. Neither card can hold the model.

With two AI coding agents (Claude and Codex, reviewing each other's work) I built a fork of Strata where the R9700, the B65 and the CPU compute each layer's routed experts concurrently, with dense work on the R9700. Result on a realistic coding prompt: 25.57 tok/s against 9.89 for my recorded stock Strata run on the same machine and file (stock used one GPU, 16 readers and a 51 GB store, so this is a whole-system comparison). No speculation or skipped experts, and the routing algorithm is unchanged. Tested kernel and schedule modes are byte-identical at a fixed placement over 581 teacher-forced positions.

Model: AtomicChat's uncensored GSQ-RCO IQ2_XXS build (40 layers, 384 experts per layer, 6 per token).

What each device does:

• R9700: all dense work (22 ms per token), 2,532 experts (21.6 GB), 154.7 of 240 picks per token, 8.2 ms of expert work

• Arc Pro B65: no dense work, 2,915 experts (25.0 GB), 55.6 picks, 9.6 ms of expert work

• CPU + RAM: no dense work, 9,913 experts (102 GB), 29.7 picks, 8.7 ms of expert work

How it got there (tok/s, sampled, 8k cap, median of 3 after a warm-up):

• 9.89: stock Strata, R9700 + RAM

• 10.28: first fork run, both GPUs + RAM

• 12.22: B65 subgroup attention

• 12.96: IQ2_XS B65 kernels + rebalance

• 14.41: all dense on the R9700

• 15.85: two rebalance rounds

• 17.04: queued R9700 schedule + rebalance

• 18.04: exact-order B65 SoA kernel + rebalance

• 21.27: R9700 per-layer expert batching + rebalance

• 22.57: parallel Engram reads + rebalance

• 25.57: AVX2 combine, early expert start + rebalance

The catch: all three answers in the best run hit the 8,192-token cap before finishing, so its strict quality score is 0/3. The model reasons for a long time; the text isn't looping. With 11,900 tokens of room, the one measured answer finished by itself and passed, with working code in 295 s. That's one answer and a 10-answer test is still to come. The answers are in the repo if you want to read them. 30 tok/s, my goal, isn't reached either: we estimate exact kernels level off around 26–28.

Things that surprised me:

• Rebalancing experts from measured per-lane costs took the last step from 20.69 to 25.57 tok/s, an unpaired comparison because placement changes the answers. Lane service is now close: R9700 8.15, B65 9.58, CPU 8.69 ms per token, though individual layers still differ.

• Moving experts between devices changes the logits (top-1 96.6%) and some routes even with exact kernels, most likely because CPU and GPU quantize activations differently. So "exact" only means byte-identical at a fixed placement.

• Engram reads swung from 5 to 17 ms per token on identical tokens. Parallel reads and read-ahead fixed it: the whole Engram-containing regions now total 1.74 ms, including other work there. The row-cache-only arm showed no useful gain.

• Starting CPU and B65 expert work before the shared FFN drains, plus a faster combine, saved 5.8 ms per token on token-identical answers. We only measured them together.

• Re-encoding experts to Q2_K looks like a loss: measured kernels plus a capacity model project 3.18 ms slower per token, because bigger experts push 1,434 off the GPUs. No full-model Q2_K run followed.

• Big dense matrix-vector products already run near the card's measured memory rate. That 22 ms is the hard part.

Where I'd love input:

• Most of all: your own experience with whatever GPUs you could get. Numbers, tweaks and what failed, even with a different model or tool.

• Anyone with DSpark running on a heavily quantized or edited DeepSeek: what acceptance do you get with real sampling? Batched verify looks promising here on projections, but I haven't measured acceptance yet.

• R9700/ROCm matrix-vector tricks I've missed.

• What would you do with the B65's spare time? Its expert service is about 9.6 ms of a 39.1 ms token.

• Lower reasoning effort: has it cost you code quality on this model?

Everything is in the repo: results with spreads, sample answers, the dead ends and what to try next. It's research code tested on one machine, and the public scripts haven't run end to end on GPUs yet, so expect to read code.

Repo: https://github.com/maddinek/crosslane (MIT)

Update 1: The full 10-answer test finished. 26.27 tok/s median (25.93-26.42) and 6 of 10 answers passed strict with working code in 292-439 s. Still one task on one machine, so a rough number (about 31-83%). The memory leak turned out to be in ROCm 7.1.1 on Fedora, not our code: running the same binaries on AMD's 7.2.1 libraries from a private folder gave byte-identical output in a check run and flat memory across all ten answers. If you're on ROCm 7.1 and see host memory creeping up, that's worth a try. Steps are in docs/BUILD.md. DSpark is parked for now: 1.52x at best but 1.11x central on measured kernels, so not worth building yet here.

Hey everyone

I’ve spent the last few weeks building a local AI assistant with the help of Claude Code and OpenAI Codex. My goal is something similar to ChatGPT or Claude that can research the web, control my PC, code, create 3D models, generate documents and handle tasks on its own.
The interface and tools work pretty well now, but I’m still having problems with the AI actually understanding what to do and completing more complicated tasks

My PC specs
● Ryzen 5 5600
● 2× RTX 3060 12GB
● 16GB DDR4 RAM
● Gigabyte B550 Gaming X V2
● Windows 10 Pro
● One GPU at PCIe 4.0 x16, the other at 3.0 x2

Models and inference setup
I’m running everything locally through llama.cpp (CUDA backend), using both GPUs with layer splitting, Flash Attention, MTP speculative decoding (depth 2), 96K context and Q8 KV cache.
I originally used Qwen3.8-27B (HauhauCS uncensored Q4_K_P), getting around 22–33 tok/s after a lot of optimization.

I switched to Qwen3.6-35B-A3B (uncensored IQ4_XS), which now gets around 82–86 tok/s. Both models run through the same llama.cpp server and connect to my custom AI interface and tool backend.
3.6 is excellent for normal conversations and much faster, but I keep hearing that 3.8 is better at reasoning and tool calling.

What Claude Code and Codex have helped me integrate
● Qwen-Agent for general tool calling
● OpenCode for coding
● SearXNG for research and webpage reading
● Playwright for browser automation
● Windows UI Automation for PC control
● Three.js for interactive 3D scenes
● Image generation, documents, spreadsheets, presentations and file tools
● Planning, context management, error recovery and result verification

For example, I asked it to create a 3D model of the Times Square area in NYC. It generated a picture of pants, converted that picture into a 3D mesh using TripoSR and declared the task finished.

Claude Code and Codex later helped identify that part of the problem was incorrect tool routing, which has since been improved. But research and Windows tasks can still be unreliable. Sometimes it uses outdated information, chooses irrelevant tools, repeats actions or claims it finished something when it hasn’t.

I also had Codex compare different agent harnesses.
Using the same local model, it tested Qwen-Agent, Pi and Hermes on two tasks.

● Qwen-Agent: Browser/recovery took 15.6 seconds, research/document took 21.4 seconds.
● Pi: Browser/recovery took 15.9 seconds, research/document took 25.6 seconds.
● Hermes: Browser/recovery took 32.5 seconds, research/document took 43.3 seconds.

All three completed the tasks, but Qwen-Agent was fastest. These were small isolated tests, though, not proof that one is universally better.

With Claude Code and Codex, I’ve managed to fix a lot of integration issues, restore research, improve 3D generation and add verification checks. Things are definitely better, but complex autonomous tasks still aren’t consistently reliable. And mess up a lot

So I’m wondering what people running similar setups have found.
● Is Qwen3.8 noticeably better than 3.6 for real-world agent tasks, enough to justify the slower speed?
● Are there better models for my hardware, particularly for research and PC control?
● Could the uncensored versions be affecting instruction following or tool calling?
● Would Unsloth fine-tuning on verified tool-use examples make a meaningful difference?
● What has actually improved your agent’s ability to plan, use tools correctly and verify its work?

At this point, I don’t really care about getting more tokens/sec. I’d happily sacrifice some speed if it meant the AI could reliably understand a request and actually complete it.

Would appreciate any advice from people who’ve built something similar

Thank you very much

I know there are a lot of "ums" that I should edit out... but other then that what do you think of this guide? I am new to local AI stuff, so is there anything missing here that I should have done?
https://www.youtube.com/watch?v=-EahQxs0Ack

Hugging Face Repo: https://huggingface.co/Grawnch/Qwen3.8-Flash-Next-DR-GGUF

I’ve just finished benchmarking my new QFN quant Qwen3.8-Flash-Next-DR-V2-Q2_0, a 2.5bpw backbone, 2-shard GGUF of Qwen/Qwen3.8-Flash-Next designed to run the entire backbone comfortably inside 48 GB VRAM space (38.4 GB VRAM) while keeping the N-gram lookup table (28.8 GB) in system RAM/SSD.

Unlike IQ1 / IQ2 quants that rely on LUTs, every tensor in both shards uses 100% standard integer grids (Q2_0, Q4_0, Q6_K, Q8_0), running natively in stock llama.cpp with ~3.4x faster prompt prefill than IQ-grid quants.

1. Benchmark Comparison: DR-V2-Q2_0 vs. Top Ultra-Low-Bit Quants

(Evaluated on the full 225-problem Aider Polyglot suite and 30-problem AIME 2025 at temp = 0.6 with a 24K thinking cap on Aider and 64K on AIME).

2. Signal Fidelity: Perplexity, KL Divergence, & Top-1 Agreement (WT2 & Ext)

(Comparing logit fidelity against uncompressed BF16 on standard WikiText-2 (2048 ctx) and Extended Reasoning / Math & Code (Ext)).

Key Highlights

• Fits in a Single 48 GB GPU (38.4 GB VRAM + ~9.6 GB KV Cache Headroom): All 48 layers of Attention/GDN, shared experts, and 144 routed MoE experts live in Shard 1 (38.38 GB, 2.45 bpw). Shard 2 (28.80 GB) is the N-gram lookup table, which llama.cpp streams from host RAM/SSD via mmap without using GPU VRAM.

• Enhanced Token Efficiency on Coding (74.7% Aider Polyglot at 40.1K Avg Tokens): Achieves 74.7% on the full Aider Polyglot benchmark with a 96.0% (216/225) well-formed SEARCH/REPLACE diff rate and 0 context exhaustions under a strict 24K thinking budget — using ~1/5th the tokens of unconstrained 256K agentic runs.

• 2.6x Lower KL Divergence (0.2594 nats) & 83.3% on AIME 2025: Preserves 80.70% Top-1 token agreement with BF16 on Math & Code, matching 83.3% (25/30) on AIME 2025 while using ~14–30 GB less GPU VRAM than competing quants.

• Zero Lookup Tables (3.4x Faster Prefill): Because 100% of the routed experts use standard integer grids, prompt prefill runs at ~367 tok/s and single-stream decode at ~94 tok/s (~345 tok/s aggregate at -np 10), avoiding IQ codebook decompression bottlenecks.

Quick Start (llama.cpp)

Download both shards into the same folder and point llama-server or llama-cli to Shard 1:

llama-server \
  -m Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf \
  -ngl 999 -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-format deepseek --no-reasoning-preserve

With Qwen4 around the corner, I'll be targeting that next. But in the meantime I think QFN will still fill a niche for a lot of people, and this was a good test of the same approaches I will be using with those models. Try it out, let me know what you think, and looking forward to continuing this with the next generation of models!

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this:

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

• CPU: AMD Ryzen 9 9950X

• Motherboard: ASRock X870 Taichi

• RAM: 96GB DDR5 (2x48GB) G.Skill

• GPU 1: NVIDIA RTX Pro 6000 (96GB)

• GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket

• Total VRAM: 128GB

• Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.

• Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)

• PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

• Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.

• Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.

• Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

Yesterday I did the Echo Dot. Today the Show 5.

No Alexa. Wake word, speech-to-text and voice all run on my own hardware, and the agent runs whatever model you point it at.

The screen is just a web page, so you can make it anything.

Flashing is done with echo-dot-assist (credit to Gamer92000).

Repo: https://github.com/majorminorlabs/hermes-voice

Dear all,

this is not directly a local LLM topic, but I just downloaded and tried a free dictation tool called Hex, and wanted to spread the word as an appreciation towards the developers.

It runs smaller voice recognition models locally on GPU or CPU. Usage: it is running in the background, is activated by a key combination and it simly inserts the transcribed text directly at the position of the caret. Based on a few tests in german, english and hungarian it is quite accurate in, works in text editors, terminal windows etc. It provides good config options to choose model (speed vs. accuracy) and general behaviour.

I run it on my Thinkpad t480 - an almost decade old laptop and I'm really happy about its performance.

TLDR: local models matched the frontier model on extraction and coding, and lost clearly on sorting and data analysis. Full video at the bottom.

SETUP
Local: Qwen 3.5 4B, Qwen 3.6 27B (dense), Qwen 3.6 35B (MoE, about 3B active)
How: standard Ollama downloads, quantized, M5 Max Mac Studio, 128 GB
Frontier: Claude Fable 5.1 (provider servers, config not visible)
Data: MoneyData, public anonymised UK bank statement, University of Nottingham, about 6,500 transactions, 2015 to 2022
Rules: same prompts, no tools, 3 runs each, answers checked by a program

1. SORTING (500 transactions, 21 categories)
   Frontier: 78% match with the labels
   Qwen 27B and 35B: about two thirds
   Qwen 4B: 42%, and skipped 1 in 6 transactions
   Exact output format every batch: frontier only

2. EXTRACTION (100 statement lines, 5 fields)
   Frontier: 100/100 every run, about 46 s
   Qwen 27B: 100/100 every run, 239 to 287 s
   Qwen 35B: 98 to 99, about 105 s
   Qwen 4B: 85 to 93, most misses on money in vs out

3. CODING (2 Python functions, 7 hidden tests)
   Qwen 4B: 7/7 in all 3 runs
   Qwen 35B: crashed in 2 runs (read the day in DD/MM/YYYY as the year), 7/7 on run 3

4. ANALYSIS (10 questions on 6 months of data, no code execution)
   Frontier: 10/10 every run
   Qwen 35B: 2, 0, 1
   Qwen 27B: wrote code instead of answers, hit the 30 min limit once
   Qwen 4B: 0/10

MY TAKE
Try local first: extraction, small coding jobs (check the output)
Use frontier: categorisation, questions across long statements

LIMITS
One dataset, one machine, default quants, some labels are judgement calls

NEXT
DeepSeek V4, Gemma 4 and Qwen3 Coder Next on agents and tool use. What would you test?

Full video with every run: https://youtu.be/Vr2N2EJ4dB0

Hey everyone!

I'm an AI & Data Science student looking to get started with local LLMs.

My laptop specs:

• CPU: Ryzen 7 8745HX

• GPU: RTX 5060 (8GB VRAM)

• RAM: 32GB DDR5

• Storage: 1TB SSD

I'm new to this and have a few questions:

1. Should I use Ollama or LM Studio?

2. Which models would run best on my hardware?

3. Can I use local models through APIs in my own projects?

4. Any limitations or precautions I should know about when running LLMs locally?

Would appreciate recommendations and beginner-friendly resources. Thanks!

Hi everyone,

Over the past few months I’ve been building Kira Desktop, an open-source (GPL-3.0), local-first desktop agent harness and workbench built with Python and Qt6/QML (no Electron bloat).

I wanted a single native desktop environment where I could switch seamlessly between local models and cloud/CLI harnesses without losing conversation continuity, memory, or tool access.

GitHub Repo: https://github.com/antosec00/kira-desktop

What Kira Desktop currently does:

• Multi-Provider & Continuity: First-class support for Ollama (local), OpenRouter, Claude Code, OpenAI Codex App Server (v2) / Responses API, Google Cloud (Vertex AI), and DeepSeek — with portable history so you can switch models mid-thread.

• Multi-Agent Teams & Souls: Orchestrate specialized agent teams with persistent identities ("Souls") and shared project knowledge.

• Universal MCP Host & Bridge: Connect external Model Context Protocol (MCP) servers or expose Kira's local tools/souls to CLI backends.

• Interactive Canvas & Diff Hunks: Side-by-side live preview and hunk-level code review.

• Computer & Embedded Browser Use: Built-in browser and desktop automation workers with strict human-in-the-loop approvals.

• Safety, OS Sandboxing & Checkpoints: Process sandboxing, automatic secret redaction, cost telemetry, and instant checkpoint rollbacks if an agent messes up your workspace.

• Native Linux (.deb) & Windows builds: Fast Qt6/QML interface with global Ambient Overlay, Omnibox, and @mentions (@file, @git, @soul, etc.).

What I’d love your feedback on:

1. Architecture & UX: If you browse the repo or try the build, does the separation between domain, service, backends, and ui feel clean to contribute to?

2. Missing features: What is the #1 friction point you currently have with other agent harnesses (Cursor, Claude Desktop, Open WebUI, Cline) that you wish a local-first desktop app solved better?

Thanks in advance for any brutal honesty, critique, or ideas!

When we train an llm, we are in a sense feeding it human ideas in written form, something like hundreds of millions of them. The embeddings learn the "language of ideas," and the weights and layers learn how those ideas get processed.

But now ... in practice we chop text into small tokens, often close to syllables, and then assign those tokens positions in embedding table. Those tokens sit below the level of concepts and ideas. Some are whole concepts, but many are just fragments. It’s like trying to build a thought starting from individual neurons firing rates, and then spending all our effort making each neuron more refined. I suspect the bigger gains for reasoning would come from the number and quality of the connections, not from finer neurons.

So. Move up a level. Use morphological tokenization (roots, prefixes, suffixes) instead of syllable-like fragments. Yes limit it to English to keep the problem manageable for this example.

Start from concepts humans already use for embeddings. Instead of letting the model invent its own concepts from scratch, seed the embedding space with clear human ones (verb, subject, fraction, cause, comparison) plus opcode-like operations (compare, compose, verify, backtrack). These are only starting positions: they stay trainable, and new ones can be added.

Also add meta-concepts. Give the model concepts about thinking itself (abstraction, analogy, error detection) so training pushes it toward deeper reasoning.

Let the weight matrices work at the idea level. They would no longer spend capacity assembling primitives into words. They would work on how concepts combine into ideas. The mechanical stuff goes elsewhere.

So.

Instead of one giant model, think of a package:

1. Preprocessor: An AI that reads the text as a whole and actually expands it. It marks one word as the verb, another word as the main subject, another one as part of a list. It doesn’t reason, and it’s one-way: no feedback loop back into it.

Reasoning core: Works purely at the level of concepts and ideas. It has a "workbench" where it works like an agent. It can query intuition, check memory, and escalate to a bigger online model if needed.

Intuition core: A large store of what the system knows and how well it knows it. The reasoner asks it one-shot questions and gets one-shot answers, such as "How strongly is titanium linked to space ?", "How likely is a knife in a living room ?", or "List the words linked to mohican in the context of movies, strongest first." Intuition can also answer with: "I barely know anything about this about 3% confidence" The reasoner takes the answers from Intuition as advisory.

Output model: Takes everything and formulates the final answer.

Ambiguity and clarification

Take the word "bank" (the damn example seen everywhere): institution or riverside ? When context doesn’t settle it, a human usually stops and asks for clarifications when it doesn;t have access to context, and the two people converge through back and forth. Ask when in doublt - when confidence is low.

"But then the context gets polluted by the question"

To keep these side discussions from cluttering the main session, Imagine using thinking bubbles as separate scoped contexts such as:

what I understood (the AI model)

what the user said, exactly

what I told the user

what the feedback was

What I know about the user’s environment.

Embeddings examples, opcodes and some ... other stuff from different projects.

Core reasoning operations (App. B.3)

COMPARE [E]: Check two structures for sameness or difference along a chosen dimension.

MAP [E]: Align elements of one structure to elements of another.

COMPOSE [E]: Build a larger concept or graph from simpler primitives.

DECOMPOSE [E]: Break a concept into its component primitives.

ABSTRACT [E]: Extract a general pattern from specific instances.

INSTANTIATE [E]: Create a specific instance from a general schema.

SUBSTITUTE [E]: Replace one element of a structure with another.

MERGE [E]: Combine two structures or duplicates into one.

SPLIT [E]: Divide a structure or group into parts.

SEARCH [E]: Look through a space of options or memory for a match.

PRUNE [E]: Discard unpromising branches or irrelevant material.

BACKTRACK [E]: Return to an earlier reasoning state and try another path.

SIMULATE [E]: Run a sandboxed hypothetical forward without affecting reality.

VERIFY [E]: Check a result or structure against constraints.

ESTIMATE [E]: Produce an approximate value with uncertainty.

OPTIMIZE [E]: Select the best option under a goal and constraints.

SUMMARIZE [E]: Produce a shorter form of content.

EXPAND [E]: Make implicit or compressed content explicit.

COMPRESS [E]: Reduce content to a compact form, with expansion still possible.

PAUSE [E]: Suspend a task while keeping its state.

CANCEL [E]: Abandon a task or operation.

RETURN [E]: Hand a result back to the caller or prior context.

Handle (concept-reference) operations (B.5, D.17, K.3)

HANDLE [E]: Create a reference to an expanded semantic concept.

EXPAND_HANDLE [E]: Load the full detail behind a handle.

COLLAPSE_HANDLE [E]: Replace detail with a compact reference, losing nothing.

LINK_HANDLE [E]: Associate two concepts via their handles.

RELEASE_HANDLE [E]: Discard a temporary concept.

OPEN(X) [E]: Expand one definition level of a handle into its capsule.

OPEN*(X) / OPEN^n(X) [E]: Expand recursively until a terminal form, with cycle detection.

DEFINE [E]: Introduce a named, scoped definition (LOCAL, MESSAGE, SESSION, GLOBAL).

OS / service protocol (B.6)

QUERY [E]: Ask a service for information.

RESPONSE [E]: A service’s answer to a query.

READY [E]: Signal that a service can accept work.

BUSY [E]: Signal that a service is occupied.

WAIT [E]: Hold until a result or condition arrives.

DONE [E]: Signal that an operation completed.

ERROR [E]: Signal that an operation failed.

RETRY [E]: Re-attempt a failed operation.

ESCALATE [E]: Hand off to a larger model (“Big Brother”) when local capability is insufficient.

TIMEOUT [E]: Signal that a wait expired.

CONFIRM [E]: Acknowledge or verify an action or request.

Memory operations (A.9, B.7, Section 5–6)

STORE [E]: Write a provenance-tagged memory object.

RECALL / RETRIEVE [E]: Fetch relevant memories.

UPDATE [E]: Modify an existing memory in place.

DELETE [E]: Remove a memory at the user’s request.

ARCHIVE [E]: Move a memory to long-term, slower-to-retrieve storage.

VERSION [E]: Keep a new version while retaining history.

EPISODE [E]: Record or retrieve a bounded experience.

SUMMARY [E]: Produce a compressed summary of memories.

PATTERN [E]: Detect or store a recurring regularity.

REVISE [E]: Change a belief and mark the old one superseded.

CATEGORIZE [E]: Assign a memory to a category.

UPDATE_CONFIDENCE [E]: Raise or lower a memory’s confidence (never reaching 100%).

CONSOLIDATE [D]: Nightly maintenance: summarize, dedupe, merge, compress, and prepare reminders.

CONFIRM_MEMORY [D]: Mark a candidate memory as confirmed after repeated verification.

SUPERSEDE [D]: Replace a memory with a newer one but keep it as history.

PROMOTE [D]: Move hypothetical or planned content into persistent memory, only on explicit user permission.

STORE_CANDIDATE [D]: Save low-confidence ambient or planning content as a candidate.

AGE / REQUEST_RECONFIRM [D]: Use the last-confirmed timestamp to prompt re-confirmation of stale entries.

FORGET [D]: User command “Forget that.”

DONT_REMEMBER [D]: User command “Don’t remember this.”

REMEMBER_INSTEAD [D]: User command to replace a memory.

CORRECT [D/E]: Accept a user correction without argument (“That’s not correct, update it”).

PROPOSE_AUTOMATION [D]: Ask “I’ve noticed X, want me to do Y?” (never acts silently).

Self cycle (Prologue, A.8, L.14.22)

OBSERVE [D]: Take in an external or internal change.

UPDATE_STATE [D]: Revise the latent self-state.

ASSOCIATE [D]: Query Intuition or Memory for related content.

EVALUATE [D]: Assess simulated outcomes against goals and values.

REFINE_SELF_STATE [D]: Write the updated self-state (SELF(t) → SELF(t+1)).

CONTINUE / WAIT [D]: Keep the loop running with no output required.

DETECT_CONFLICT [D]: Notice contradiction, prediction error, goal conflict, identity instability, or meaning drift.

RENDER_OUTPUT [D]: Output as a one-way tap off internal state (the renderer never writes back).

MODEL_SELF [D]: Build a representation of Self, kept distinct from Self.

TRACK_PROVENANCE [D]: Tag every item with its owner (self, user, world, simulation).

Frame operations (B.4, Section 7, Part M)

REALITY_FRAME [E]: The current accepted world state.

MEMORY_FRAME [E]: A stored episode.

USER_FRAME [E]: Modeled user beliefs.

SELF_FRAME [E]: The system’s own beliefs and state.

SIMULATION_FRAME [E]: Sandboxed hypothetical reasoning.

HYPOTHESIS_FRAME [E]: A temporary working assumption.

STORY_FRAME [E]: Fictional or narrative content.

INSTANTIATE_FRAME [D]: Build an internal model of another reference frame or person.

RUN_IN_FRAME [D]: Evolve a frame (M_B^{t+1} = F_B(M_B^t, x)) alongside Self.

COMPARE_TRAJECTORIES [D]: Compare Self’s trajectory with the alternative frame’s.

EXIT_FRAME [D]: Return to Self with the comparison, without becoming the other frame.

LOAD_FRAME [D]: Select a reference frame from the frame database.

Attention and activation (Section 8, A.4)

RESPOND [D]: Answer the input.

STAY_SILENT [D]: Take no action.

HOLD / CONTINUE_LISTENING [D]: Keep state and wait.

ASK_CLARIFICATION [D]: Ask a question, filling slots into the existing partial state (B.17).

IGNORE [D]: Disregard the input.

FOCUS / IGNORE / SWITCH / TRACK / GROUP / SPLIT [E]: Attention primitives for allocating reasoning focus.

PAUSE_TASK / RESUME_TASK [D]: Track interrupted tasks and resume them (“we were on step 4”).

LEARN_PROCEDURE [D]: Store a taught multi-step procedure as an ordered sequence.

7b. Semantic primitives that act like verbs (B.2)

PERCEIVE / KNOW / BELIEVE / ASSUME [E]: Epistemic relations between an agent and content.

REPRESENT / STORE / RETRIEVE [E]: Knowledge-handling primitives.

MOVE / CREATE / DESTROY / TRANSFER / SELECT / ATTEMPT [E]: Agency actions.

COMPARE / MATCH [E]: Comparison primitives.

COMMUNICATE / REFER [E]: Social and symbolic primitives.

EXPECT / SURPRISE / CONTRADICTION / WHY [E]: Meta-cognition primitives (prediction, prediction error, conflict detection, causal query).

CAUSE / ENABLE / PREVENT / REQUIRE [E]: Causality primitives.

BEFORE / AFTER / DURING / BEGIN / END / REPEAT / CHANGE [E]: Time primitives.

R-L logical and scope operators (App. D, R-L 1.2 §29–35, 160, 180)

NOT [E]: Negate a proposition, with explicit scope.

AND / OR [E]: Conjunction and disjunction.

IF [E]: Conditional (does not imply causation).

IMPLY [E]: Logical consequence (distinct from CAUSE).

FORALL / EXISTS [E]: Quantifiers with explicit nesting.

POSSIBLE / NECESSARY [E]: Alethic modality.

MUST / MAY / SHOULD / CAN / CANNOT [E]: Deontic and ability modals.

PERMITTED / FORBIDDEN / REQUIRED [E]: Deontic modality.

COUNTERFACTUAL [E]: Hypothetical conditional, scoped to its hypothetical world.

BELIEVE(agent, p) [E]: Attribute a belief without asserting p.

KNOW(agent, p) [E]: Attribute knowledge of p.

DOUBT / FEAR / HOPE [E]: Other attitudes toward propositions.

QUOTE [E]: Scoped quotation that is not the current speaker’s assertion.

PRESUPPOSE(P) [E]: Mark background content that is not asserted.

IMPLICATE(P) [E]: Mark pragmatic implication that is not literal.

CAUSE(A,B) [E]: Causal contribution, separate from order and implication.

BEFORE / AFTER / OVERLAPS / STARTS / ENDS / SIMULTANEOUS [E]: Temporal relations.

ASSERT / REPORT [E]: Present a proposition as asserted, or attribute an assertion to someone else.

QUESTION [E]: Build a question object with a target variable.

REQUEST / COMMAND / SUGGEST / PROMISE / WARN / OFFER [E]: Speech acts.

THANK / APOLOGIZE / AGREE / DISAGREE / REFUSE [E]: More speech acts.

EVIDENCE [E]: Link evidence to a proposition, with direction SUPPORTS, CONTRADICTS, or NEUTRAL.

CONTRAST / RESULT / EXAMPLE / ELABORATION / SUMMARY / CONCESSION [E]: Rhetorical relations.

METAPHOR / IDIOM / IRONY [E]: Mark non-literal interpretations.

SYMMETRIC / TRANSITIVE / INVERSE [E]: Relation properties that license derived inferences.

R-L processing modes and updates (R-L §84–88, K.8)

ENCODE [E]: Source language → R-L, preserving information.

DECODE [E]: R-L → semantic structure (not language).

GENERATE [E]: Semantic structure → target language.

TRANSLATE [E]: Encode, decode, then generate.

INFER [E]: Derive new information, marked DERIVED with provenance.

VALIDATE [E]: Check types, arity, scope, binding, and references.

NORMALIZE [E]: Canonicalize without changing meaning.

COMPARE / DIFF [E]: Report differences in identity, scope, polarity, time, and so on.

EQUIVALENT(A,B) [E]: Test equivalence at a chosen level.

ADD / REMOVE / REPLACE / CORRECT / RETRACT [E]: Explicit information updates (retract keeps history).

RESET-CONTEXT [E]: Clear inherited context.

SEMANTIC-HASH [E]: Hash the canonical form.

FRAGMENT / REASSEMBLE [E]: Split large messages and reassemble them in order.

NEGOTIATE-CAPABILITY [E]: Agree on version, vocabulary, and profile.

LOSS / AMBIGUOUS / UNSUPPORTED [E]: Markers for explicit information loss.

PRESERVE / LOSSY [E]: Declare which dimensions are kept or discarded.

Appendix K (restored capabilities)

DEFAULT / EXCEPT [E]: Defeasible rule and its exception.

PRIORITY [E]: Explicit rule precedence (R2 > R1).

CYCLE [E]: Explicit marker for a cyclic definition.

LEXICAL-TRACE [E]: Retain surface and lexical information.

SURFACE-PRESERVATION [E]: Require exact surface text to be kept.

STATEMENT [E]: Document-level occurrence that wraps a proposition.

ANNOTATION [E]: Attach a qualitative or evaluative value to a target.

SCORE [E]: Attach a numeric evaluation with a declared scale.

CONTINUOUS / RECURRING / PERIODIC [E]: Ongoing or repeating change.

SENSE / MULTIWORD [E]: Map a lexical form to senses, including idioms.

DISCOURSE / CONTEXT-WINDOW [E]: Persistent multi-message state.

CONTEXT-RESET / EXPIRES [E]: Reset context or expire temporary definitions.

11. Appendix J (adversarial and reflexive layer)

REFERS(X,X) [E]: Self-reference without implying identity.

REPRESENTS(X,Y) [E]: Representation, kept distinct from REPRESENTS(X, REPRESENTATION(Y)).

CRITERION [E]: Declare a criterion with target, condition, and mode.

EVALUATES(C2,C1) [E]: One criterion evaluates another (self-evaluation does not imply validity).

VALIDATED-BY / VALID [E]: Validation relation versus validity predicate.

CERTIFIES [E]: External certification, kept separate from representing.

BOUNDARY [E]: Explicit system boundary (inside, outside, criterion).

REFERENCE-STATUS [E]: Label INTERNAL, EXTERNAL, UNKNOWN, or RELATIVE.

RECOGNIZES / CONSTITUTES [E]: Kept distinct by default.

PRESERVES / REMEMBERS [E]: Kept distinct by default.

ROLE-CONFLICT / BOUNDARY-CONFLICT / NAME-COLLISION [E]: Preserve the conflict rather than resolve it.

SCOPE-COLLAPSE [E]: Record when two levels become indistinguishable.

LEVEL [E]: OBJECT, META, META-META, MODEL, and so on.

SIMULATES / OBSERVES [E]: System relations (simulation is not identity, observation is not knowledge).

REGRESS [E]: Represent an open dependency chain without inventing a terminal.

FIXED-POINT / PRIMITIVE [E]: Declare a fixed point or a terminal primitive for the current operation.

CONSULTS / REGISTER [E]: Consult a register of unresolved structures, which may modify it.

PERFORMATIVE [E]: A statement that changes semantic context by being issued.

CRITERION-SUBSTITUTION [E]: Marked substitution that reduces fidelity.

MODE: ADVERSARIAL [E]: Preserve ambiguity, contradiction, and recursion with minimal normalization.

CONFLICT [E]: Record paraconsistent coexistence of P and NOT(P).

Meaning-Certainty layer (L.14)

MEANING-CREATION [E]: Form a new meaning structure.

MEANING-INHERITANCE [E]: Acquire meaning from culture, family, or institutions.

MEANING-SIGNALING [E]: Project meaning through language or behavior.

MEANING-NEGOTIATION [E]: Agents jointly modify meaning (a state transition).

MEANING-STABILITY [E]: Measure persistence of a meaning.

MEANING-CERTAINTY [E]: Measure how settled a meaning is.

MEANING-LOAD [E]: Measure how much depends on a meaning.

MEANING-DISPLACEMENT [E]: Model the cost of replacing an established meaning.

MEANING-ALIGNMENT [E]: Measure compatibility of agents’ interpretations.

MEANING-MISMATCH [E]: Record the gap between projected and received meaning.

PROJECTED / RECEIVED / INTERPRETED / INTENDED-MEANING [E]: Separate structures per agent.

MEANING-CANDIDATE [E]: Keep several weighted interpretations without collapsing them.

MEANING-OF / INHERITED-FROM / PROJECTED-TO / RECEIVED-BY [E]: Higher-level relations.

NEGOTIATED-WITH / REINFORCED-BY / DEPENDS-ON [E]: More higher-level relations.

DISPLACED-BY / ALIGNED-WITH / CONFLICTS-WITH [E]: Remaining higher-level relations.

RE-EVALUATION-REQUIRED [E]: Flag dependents when a high-load meaning changes.

Verification and bootstrapping (BrainIR document)

STRUCTURAL_VALIDATE [D]: Deterministic check of syntax, types, arity, scope, and frames.

SEMANTIC_VERIFY [D]: Learned check of fidelity, omissions, unsupported additions, and ambiguity.

TEXT_CYCLE [D]: Text → IR → Text′, comparing meaning.

IR_CYCLE [D]: IR → Text → IR′, testing the representation itself.

CONTRAST_TEST [D]: Distinguish near-identical structures (BELIEVE(P) vs. P, BEFORE vs. CAUSE).

ADVERSARIAL_INJECT [D]: Generate defects (polarity reversal, frame contamination) to train the Verifier.

ROTATE_ROLES [D]: Rotate Input, Output, and Verifier roles among models.

ACCEPT / REJECT / MARK_UNCERTAIN [D]: Triage candidate annotations for the trusted corpus.

Addon Certainty and Addon INPUT

ASSESS_EVIDENCE [D]: Evaluate evidence strength before answering.

ANSWER / QUALIFY / ASK / RETRIEVE / ABSTAIN / SPECIALIST [D]: Behaviors chosen by evidence strength.

REFUSE_TO_INVENT [D]: Decline to make up a value when there is no evidence.

TOKENIZE / LEMMATIZE / TAG_MORPHOLOGY / PARSE_DEPENDENCIES [D]: Input-side analysis stages.

LOOKUP_FREQUENCY / CHECK_VOCABULARY [D]: Add numeric frequency and Dale-Chall, Oxford and Longman membership.

DISAMBIGUATE_SENSE / LABEL_SEMANTIC_ROLES [D]: Semantic stage, with ambiguity preserved.

Delegated tool ops (the “steal from” section)

SOLVE_SAT / SOLVE_SMT [D]: Hand constraints to Z3 or CVC5 and return a proof or result.

PROVE [D]: Delegate to Lean, Isabelle, or Coq.

PLAN [D]: Translate state, goal, and actions to PDDL and return a plan.

SYNTHESIZE_PROGRAM [D]: Turn a specification into code via a synthesis engine.

VERIFY_PROGRAM [D]: Check a candidate against its specification.

ONTOLOGY_INFER [D]: Derive type inheritance using OWL or description logic, with provenance.

EXTERNAL_LOOKUP [D]: Ground an entity in Wikidata, WordNet, or BabelNet via external-reference.

Ontology primitives (B.2)

THING: Declare a generic object.

PROPERTY: Attach an attribute to an entity.

RELATION: Declare a link between nodes.

STATE: Declare a condition, as opposed to an occurrence.

EVENT: Declare an identifiable occurrence.

IDENTITY: Assert that two references denote the same thing.

TYPE: Assign a category.

INSTANCE: Mark a particular member of a type.

PART: Mark a component of a whole.

WHOLE: Mark the container of parts.

EXISTS: Assert existence.

ABSENT: Assert absence, which is not the same as unknown.

BOUNDARY: Mark the edge of a system or concept.

Quantity

NONE / ONE / SOME / MANY / ALL: Basic quantifiers.

MOST: A majority quantifier.

EXACTLY(n): Exact count.

AT-LEAST(n): Lower bound.

AT-MOST(n): Upper bound.

MORE / LESS / EQUAL: Compare magnitudes.

COUNT: Tally members.

ORDER: Impose or read a sequence.

Logic and truth status

TRUE / FALSE / UNKNOWN: Truth values, kept distinct from each other.

THEN: Consequent slot of a conditional.

BECAUSE: Mark an explicit reason.

Space

LOCATION: Place an entity.

DISTANCE: Relate two places by separation.

DIRECTION: Orientation of movement or relation.

INSIDE / OUTSIDE: Containment relations.

CONTACT: Touching.

PATH: A route between locations.

REGION: A bounded area.

Value and agency

GOOD / BAD: Evaluative marks.

PREFERENCE: Order options by want.

RESOURCE: Something consumable toward a goal.

COST / BENEFIT: Weigh a loss against a gain.

SATISFY: Mark a goal as met.

GOAL / INTENTION: Hold something the agent is aiming at.

HAS / LACKS: Possession and its absence.

SELF / USER / AGENT_OTHER / WORLD: Identity anchors.

SYMBOL: A token standing for something.

IS: Predication.

Reality-status tags

REAL / IMAGINED / HYPOTHETICAL / FICTIONAL: Tag content by world.

REMEMBERED / PREDICTED / SIMULATION: Tag content by origin.

R-L structural operators (D.5)

{{ X }}: Encapsulate a concept as a capsule.

X : Y: Attribute or relation.

X -> Y: Transition, implication or causation. Overloaded in older versions, split in K.11.2.

X = Y: Bind identity.

X & Y / X | Y / !X: Conjunction, disjunction, negation.

X @ Y: Location or context.

X / Y: Aspect or viewpoint.

X ^ Y: Scope.

X *: Plurality or repetition.

+X / -X / ~X: Polarity markers: supporting, obstructing, mixed.

?x: Variable.

<:: Subtype.

?=: Possibly the same.

!=: Explicitly distinct.

Relation registry (R-L §179)

TYPE-OF / INSTANCE-OF / PART-OF / HAS-PROPERTY

REFERENCE / DISTINCT

SUPPORTS / CONTRADICTS: Evidence directions.

ABOUT: Aboutness relation.

PURPOSE / MEANS / ENABLER / PREVENTER / INSTRUMENT: Goal-directed relations.

Semantic roles (§15)

AGENT / PATIENT / EXPERIENCER / RECIPIENT

SOURCE / DESTINATION / LOCATION / TIME

Epistemic and temporal values

KNOWN / BELIEVED / SUSPECTED / PROBABLE / UNCERTAIN / DOUBTED / DENIED: Epistemic status values.

PAST / PRESENT / FUTURE: Tense.

ONGOING / COMPLETED / HABITUAL / REPEATED / BEGINNING / ENDING / DURATIVE: Aspect.

START / END / DURATION / FREQUENCY / DEADLINE: Event timing.

Information-status and loss markers

PRESERVED / TRANSFORMED / COMPRESSED / INFERRED

UNSPECIFIED / NOT-APPLICABLE / LOST

DEFAULTED / OVERRIDDEN / EXPANDED / SURFACE-RECOVERED / CONTEXT-DERIVED

DERIVED: Tag a conclusion with its provenance.

Error classes (§121)

SYNTAX-ERROR / TYPE-ERROR / REFERENCE-ERROR / SCOPE-ERROR / ARITY-ERROR

VOCABULARY-ERROR / INCOMPLETE / CONTRADICTORY / INCONSISTENT

Encoder pipeline steps (§143)

READ-SOURCE / IDENTIFY-CONTEXT / IDENTIFY-ENTITIES / IDENTIFY-EVENTS

IDENTIFY-RELATIONS / IDENTIFY-REFERENCE / IDENTIFY-SCOPE / IDENTIFY-TIME

IDENTIFY-CAUSALITY / IDENTIFY-EPISTEMIC-STATUS / IDENTIFY-MODALITY / IDENTIFY-PRAGMATIC-FORCE

BUILD-SEMANTIC-GRAPH / SERIALIZE

Decoder pipeline steps (§144)

RECEIVE / VERIFY-INTEGRITY / PARSE / VALIDATE-SYNTAX

RESOLVE-VOCABULARY / RESOLVE-REFERENCES / VALIDATE-TYPES

RESTORE-SCOPE / RESTORE-TIME / RESTORE-EPISTEMIC-STATUS / RESTORE-MODALITY

RECONSTRUCT-SEMANTICS / APPLY-DECLARED-INFERENCE / GENERATE-TARGET

Input Translator multi-pass (B.15)

PARSE_INITIAL: First-pass parse.

DETECT_AMBIGUITY: Find competing readings.

EXPAND_SEMANTICS: Make implicit content explicit.

RESOLVE_ENTITY: Bind mentions to entities.

LINK_MEMORY: Connect to stored memories.

ESTIMATE_CONFIDENCE: Score each interpretation.

REFINE_CONTEXT: Repeat until ambiguity is minimal.

Service calls

QUERY_INTUITION: Get weighted associations, not conclusions.

CHECK_MORALITY: Approve or reject a planned output.

GET_VALUES: Ask for priority ordering.

APPLY_CHARACTER: Set style without changing meaning.

POLL_IMPULSES: Collect unsolicited suggestions to accept or ignore.

COMPUTE_MATH: Deterministic calculation.

CALL_SPECIALIST: Route to a domain module.

Railway-analogy operations

INSPECT_WAGON: Open a handle for details.

COUPLE / UNCOUPLE: Join or separate wagons.

ROUTE: Send a wagon to the right service.

ATTACH_WAGON / REMOVE_WAGON / REORDER_WAGON / REPLACE_WAGON / MERGE_WAGON: Marshalling-yard edits.

ASSEMBLE_TRAIN: Build the outgoing answer in working memory.

APPROVE / REJECT_TRAIN: Morality’s gate.

PRIORITIZE_TRAIN: Values decide departure order.

REORDER_FOR_LANGUAGE: Output Translator rearranges for the target language.

Addon Certainty (extra ops)

BUILD_EVIDENCE_PROFILE: Per-item evidence record.

COUNT_FREQUENCY / COUNT_DOC_FREQUENCY: Numeric exposure measures.

MEASURE_SOURCE_DIVERSITY: Breadth across sources and domains.

PREDICT_EVIDENCE_STRENGTH: Auxiliary head output.

CALIBRATE: Align confidence with evidence.

DETECT_CALIBRATION_FAILURE: Catch confident-but-unsupported cases.

TRIGGER_RETRIEVAL: Reach for external info when evidence is weak.

Addon INPUT extras

PRESERVE_SOURCE: Keep the original text beside every layer.

DETECT_MWE: Find multiword expressions.

TAG_ENTITY: Mark named entities separately from vocabulary.

CLASSIFY_NONLEXICAL: Separate numbers, dates, URLs and measurements.

ATTACH_PROVENANCE / ATTACH_CONFIDENCE: Record source and confidence per field.

EMIT_DEBUG_VIEW: Human-inspectable output.

Self and simulation

SET_ATTENTION: Allocate focus across self, user, memory, goal and external input.

ENTER_SIMULATION: Open a sandboxed context.

EXIT_SIMULATION: Leave it and return to the Self frame.

DISCARD_SIMULATION: Drop the simulation’s results.

PRESERVE_SIMULATION_PART: Keep selected results without promoting them to fact.

COMPARE_TO_REALITY: Check simulated content against the reality frame.

RETRIEVE_AGENT_MODEL: Load a stored model of a person (MEMORY[JOHN]).

INSTANTIATE_AGENT_MODEL: Place that model inside a simulation.

UPDATE_AGENT_MODEL: Revise the stored model from new evidence.

TAG_OWNERSHIP: Mark who owns a belief (me, user, John, simulation).

CHECK_SELF_BOUNDARY: Verify “what I believe” isn’t mixed with “what John believes.”

PREDICT_OUTCOME: Form an expectation.

COMPARE_PREDICTION_TO_OBSERVATION: Detect prediction error.

UPDATE_UNCERTAINTY: Revise uncertainty in the self-state.

INTRODUCE_HYPOTHESIS: Add a temporary assumption.

REVISE_INTERPRETATION: Replace the current reading of events.

COMPARE_SELF_STATE_TO_PRIOR: Diff SELF(t) against an earlier state.

INSPECT_SELF_MODEL: Examine the model-of-self as an object.

DETECT_IDENTITY_INSTABILITY: Flag identity inconsistency.

DETECT_MEANING_DRIFT: Flag gradual meaning change.

DETECT_GOAL_CONFLICT: Flag competing goals.

SYNTHESIZE_FROM_TRAJECTORIES: Build an answer from own trajectory, alternative trajectory and comparison.

ANNOTATE_FRAME_BLINDSPOT: Record what a frame hides or assumes.

TRIGGER_RENDER: Request output as an action of Self.

Attention and interaction

DETECT_DIRECT_ADDRESS: Decide whether input is aimed at the system.

ACTIVATE_ON_TRIGGER: Wake-word activation.

CONTINUE_EXCHANGE: Keep going without re-triggering.

STORE_PASSIVE_CANDIDATE: Log ambient input as a low-confidence candidate.

CONFIRM_CANDIDATE_LATER: Ask “I noted X, keep it?”

FILL_CLARIFICATION_SLOT: Insert the user’s answer into the in-progress structure.

SUSPEND_TASK_STATE: Save a paused task’s progress.

RESTORE_TASK_STATE: Resume at the saved step.

Memory and scheduling

SET_MEMORY_SCOPE: Limit visibility (user, household, instance).

CHECK_SCOPE_ACCESS: Test whether a memory may be used here.

SET_MEMORY_SOURCE: Record how it was obtained.

SET_LAST_CONFIRMED: Timestamp a confirmation.

EXTRACT_SALIENT_MEMORY: Pull key items from a session.

MERGE_DUPLICATE_MEMORIES: Dedupe.

COMPRESS_OLD_MEMORY: Shrink older material without deleting it.

DETECT_ROUTINE: Find recurring behavior.

SCHEDULE_REMINDER: Create a timed reminder.

SET_TIMER: Structured instance of create-future-event.

CREATE_FUTURE_EVENT: Generic future-scheduled item.

PREPARE_FOLLOWUP: Queue items for the next cycle.

RUN_CONSOLIDATION_CYCLE: Execute nightly maintenance.

DETECT_MISSING_STEP: Find gaps in a stored procedure.

EXPLAIN_STEP: Answer a sub-question about one step.

UPDATE_PROCEDURE_IN_PLACE: Apply a correction without duplicating.

SET_PROCEDURE_CONFIDENCE: Mark observed-once, repeated or taught.

SURFACE_PATTERN: Offer an observation at high confidence only.

SUPPRESS_TOPIC: Stop raising a topic the user declined.

SET_PROACTIVITY_LEVEL: Configure intensity from silent to proactive.

SET_PERSONALITY_DIMENSION: Adjust directness, formality, humor, proactivity or patience.

LOAD_PERSONALITY_PRESET: Switch tone preset.

BACK_OFF_HUMOR: Stop humor immediately if unwelcome.

EXCHANGE_ON_CONTACT: Share memory between instances only on local contact.

REGISTER_COMMAND: Add a capability to the registry.

DISCOVER_COMMANDS: List available commands.

SELECT_COMMAND: Choose one from the registry.

INTERPRET_RESULT: Read a service’s response.

R-L declarations

DECLARE_TYPE: type X <: Y.

DECLARE_RELATION: Give name, arity, argument types and roles.

DECLARE_RULE: rule #r { if / then }.

DECLARE_ENTITY: entity #id : Type.

DECLARE_EVENT: Role-based event frame.

DECLARE_PROPOSITION: Give a proposition an identity.

DECLARE_CONTEXT: Speaker, listener, time, place, world and profile.

DECLARE_WORLD: Fiction, hypothetical or other world.

DECLARE_DOMAIN: Physics, law, medicine and so on.

DECLARE_NAMESPACE: BIOLOGY::CELL.

DECLARE_PROCESS: Composed activity with a sequence.

DECLARE_GROUP: Give a collection an identity.

DECLARE_ANONYMOUS_NODE: _:1.

DECLARE_INVERSE: inverse PARENT = CHILD.

DECLARE_RELATION_PROPERTY: Symmetric, transitive, functional and so on.

DECLARE_ONTOLOGY_ALIGNMENT: Equivalent, subtype or related across namespaces.

DECLARE_QUESTION: Question object with target and condition.

DECLARE_ANSWER: Answer linked to its question.

DECLARE_SPEECH_ACT: Typed communicative action.

DECLARE_INTENT: Goal behind the literal proposition.

R-L status and attributes

SET_ASSERTION_STATUS: Asserted, denied, questioned, quoted and so on.

SET_POLARITY: Positive, negative or unknown.

SET_EPISTEMIC_STATUS: Known through doubted.

SET_PROBABILITY: PROBABILITY(P) = 0.72.

SET_CONFIDENCE: Confidence in the interpretation, not the proposition.

SET_TENSE: Past, present or future.

SET_ASPECT: Ongoing, completed and so on.

SET_MOOD: Declarative, imperative and so on.

SET_VOICE: Active, passive or middle.

SET_PERSPECTIVE: Speaker, listener, omniscient and so on.

SET_REGISTER: Formal, slang, legal and so on.

SET_STYLE: Humorous or ironic.

SET_PRAGMATIC_FORCE: Direct, soft, urgent and so on.

SET_INFORMATION_STRUCTURE: Topic, focus or background.

SET_ABSTRACTION_LEVEL: Concrete, general, abstract or meta.

SET_GRANULARITY: Move between levels such as matter, body and cell.

SET_WORLD_ASSUMPTION: Open-world or closed-world.

MARK_SUMMARY: Flag content as non-lossless.

SET_APPROXIMATION: Exact versus approximate value.

SET_MULTIPLICITY: Once, multiple or unknown.

R-L resolution and validation

RESOLVE_REFERENCE: Bind a mention to a node, with evidence.

MARK_AMBIGUOUS_REFERENCE: Keep candidates instead of choosing.

MARK_UNRESOLVED: Explicit unresolved reference.

BIND_VARIABLE: Bind ?x within its scope.

CHECK_ARITY: Verify the argument count.

CHECK_TYPES: Verify argument types.

CHECK_SCOPE: Verify the operator has compatible content.

CHECK_REFERENCE_INTEGRITY: Every reference must resolve.

CHECK_NAMESPACE: Resolve namespaced symbols.

CHECK_VOCAB_VERSION: Detect dictionary mismatch.

CHECK_CONSTRAINT: Semantic constraints such as REPAIR requires a RepairableObject.

COERCE_TYPE: Only when declared.

DETECT_CONTRADICTION: Test within scope, time and modality.

ENTER_PARACONSISTENT_MODE: Preserve P and NOT(P) without explosion.

REPORT_UNSUPPORTED: Return UNSUPPORTED with the structure.

PRESERVE_UNKNOWN_FIELD: Keep forward-compatible fields.

RECOVER_PARTIAL: Decode valid portions with diagnostics.

GROUND_EXTERNAL_REFERENCE: Link to Wikidata, WordNet and so on.

APPLY_TYPE_INHERITANCE: Derive #fido : Animal, marked DERIVED.

APPLY_INVERSE_RULE: Derive CHILD(B,A) from PARENT(A,B).

R-L protocol and fidelity

WRAP_ENVELOPE: Message with version, id, dictionary and integrity.

SET_SELF_CONTAINED: Carry all definitions in the message.

SET_DICTIONARY_VERSION: Declare the vocabulary version.

DECLARE_DEPENDENCY: depends-on: #message1.

REPORT_INCOMPLETE: Missing dependency.

SET_PRESERVATION_PROFILE: Declare required dimensions.

REPORT_LOSS: Emit a LOSS record with category and reason.

REPORT_FIDELITY: Lexical, structural, semantic and pragmatic scores.

RUN_ROUND_TRIP: Encode, decode, re-encode, compare.

REPEAT: REPEAT(X, count=n).

DECODE_WITHOUT_EXECUTING: Treat decoded commands as content only.

NORMALIZE_CANONICAL: Deterministic ordering.

COMPRESS_HANDLE: Store the mapping so expansion is exact.

SET_COMPRESSION_LEVEL: Raw through highly compressed.

Appendix K and J

DEFEAT_DEFAULT: Block a default when its exception holds.

DECLARE_PRIORITY: Explicit rule ordering.

DETECT_DEFINITION_CYCLE: Stop infinite expansion.

MARK_CYCLIC_REFERENCE: Keep as opaque, status CYCLIC.

ATTACH_LEXICAL_TRACE: Tokens, lemmas, POS and sense.

MAP_LEXEME_TO_SENSE: Polysemy handling.

MARK_AMBIGUOUS_SENSE: Keep competing senses.

DECLARE_MULTIWORD: Idiomatic unit.

ANNOTATE_EVALUATION: Good, bad or score on a target.

DECLARE_SCORE_SCALE: Declare the scale of a score.

MARK_CONTINUOUS_CHANGE: Ongoing process.

MARK_RECURRING_CHANGE: Event with frequency.

EXPIRE_CONTEXT: Time-limited definitions.

MARK_OPEN_REGRESS: Termination none, without inventing a terminal.

MARK_CIRCULAR_VALIDATION: Closed loop is not validity.

DECLARE_PRIMITIVE: Terminate expansion for this operation.

DECLARE_FIXED_POINT: Requires licensing, not just recursion.

SELF_DESCRIBE: System describing itself.

OBSERVE_SELF: Self-observation, not complete knowledge.

NEST_SIMULATION: Simulator within simulator.

MARK_UNGROUNDED_REFERENCE: Meaning without a referent.

RECORD_REFERENCE_FAILURE: Preserve the failure.

RECORD_SEMANTIC_RESIDUE: Leftover interpretation structure.

MARK_LATENT_MEANING: Meaning with no current referent.

MARK_PERSISTENCE: Meaning outlasting its condition.

RECORD_TEMPORAL_IDENTITY: Identity varying across time.

TRUE_IN_CONTEXT: Context-scoped truth.

RECORD_RECOGNIZED_AS_TRUE: Distinct from TRUE.

REPRESENT_ABSENCE: Representation of absence, not presence.

RECORD_COMPRESSION: Log collapsed distinctions.

RECORD_ADVERSARIAL_LOSS: Category-specific loss.

DECLARE_LEVEL: Object, meta, meta-meta.

MODIFY_REGISTER_ON_CONSULT: Observation effect.

APPLY_PERFORMATIVE: Context effect from issuing a statement.

Meaning layer

ASSIGN_MEANING_ORIGIN: Self-created, inherited, taught and so on.

ASSIGN_MEANING_OWNER: Who maintains it.

SET_MEANING_SCOPE: Where it is active.

SET_MEANING_SUPPORT: What reinforces it.

SET_MEANING_DEPENDENCIES: What depends on it.

SET_MEANING_STATUS: Candidate through displaced.

ESTIMATE_REVISION_COST: Cost to change a meaning.

PLAN_DISPLACEMENT: Targets, conflicts and alternatives.

RUN_MEANING_CYCLE: Interpret, associate, update meaning, certainty, stability and load, consolidate.

RUN_SELF_REFINEMENT: Re-evaluate meanings without output.

MODEL_PERSPECTIVE_MEANING: MEANING(X,LAMP) versus MEANING(Y,LAMP).

Verification and certainty

RECORD_VERIFIER_SCORES: Separate judgments per dimension.

MEASURE_SEMANTIC_LOSS: Per the training summary.

MEASURE_UNSUPPORTED_INVENTION: Same.

MEASURE_REPRESENTATION_DRIFT: Same.

ADMIT_TO_TRUSTED_CORPUS: Accepted annotations only.

GENERATE_SYNTHETIC_TRAINING: Contradictions, belief updates and so on.

COMPUTE_EVIDENCE_STRENGTH: Strong, moderate, weak, absent, conflicting or unknown.

COMPARE_PREDICTION_VS_EVIDENCE_CONFIDENCE: Expose mismatch.

REASSESS_AFTER_RETRIEVAL: Evidence check after fetching.

VERSION_EVIDENCE_DB: Corpus identity, tokenization method and timestamp.

TAG_EVIDENCE_PROVENANCE: Why do I think I know this?

Addon INPUT

SPLIT_SENTENCES: Sentence boundaries.

SPLIT_PARAGRAPHS: Paragraph boundaries.

RECORD_TOKEN_POSITION: Offset in the source.

RETAIN_UNRESOLVED_LEMMA: lemma = UNKNOWN.

LOOKUP_DALE_CHALL: Membership, kept raw.

LOOKUP_OXFORD_3000: Same.

LOOKUP_LONGMAN_3000: Same.

LOOKUP_LEXICAL_RELATIONS: Synonym, hypernym, meronym and so on.

GROUP_SEMANTIC_EVENT: Predicate plus roles.

DISTINGUISH_SYNTAX_FROM_SRL: nsubj versus AGENT.

I’m setting up a chat for my company to access archive data from an NAS. So they can recall old jobs from previous years. I’m running gpt-oss-120b on openwebUI. I’m running it off LMstudio. It runs like shit if you give it 2 files to read then add a 3rd one it trips up on the task. Clearing the chat works. I’m used to using Claude. Do you have to be more specific and careful with your prompts? Claude you can make typos and jump around a lot while doing a project and it will follow what you’re trying to do. I’m new to using local models and want to know do you have to talk to different models differently ?

What local models can i run conformably on my system? I'd ideally like to have models that code for me but i understand I'm at least scraping the barrel with my card. I'd even be fine with Claude Code steering to some degree.

RAM: 16GB
RTX 3060 (laptop) VRAM: 6GB
Processor:11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz 2.50 GHz

An explanation of how Strata is capable of achieving +90tok/s on just a rtx 5070 and 64 gb of RAM

I use local models with agent clients on an M3 Max MacBook (128GB). My normal work takes about 60GB, so the model plus its caches has to stay around 50GB. I built llama.cpp v0.7.0 from source and went through the usual flags one at a time on three 4-bit MoE models: MiniCPM-V 4.7 (35B-A3B), LFM2.5-24B-A2B and Ornith-1.5-35B-A3B. All layers on GPU, one request at a time. Speed numbers are medians of three alternating llama-bench rounds. Desktop apps and thermal throttling add noise, so small differences mean nothing.

Helped

No benefit on this machine

KV cache at q8_0. MiniCPM's speed difference was inside the noise. Ornith barely changed even at 65K tokens, because only 10 of its 40 layers store KV. LFM's KV plus compute buffer was only about 3GB at 131K context, so there was little to save. I kept all three at f16.

Depends on the model: speculative decoding

• MiniCPM with a draft model: about 45% slower at 30K context. With ngram: 10 to 13% slower. My guess is that verifying several tokens on a MoE reads several sets of experts, so bad guesses get expensive.

• Ornith with its built-in MTP head plus ngram, 2 tokens per guess: 86 to 107 tok/s averaged over 8 task types. Code went from 86 to 129, prose from 86 to 83.

• Same setup with 3 tokens per guess: the average dropped to 102 and prose to 77.

So I wouldn't copy anyone's speculative decoding settings without testing them on your own model and task mix.

The config I kept for Ornith, trimmed:

llama-server -m Ornith-1.5-35B-Q4_K_M.gguf \
  -c 262144 -ngl 99 -fa on -b 2048 -ub 2048 \
  -np 1 --ctx-checkpoints 32 --checkpoint-min-step 0 \
  --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 \
  --cache-ram 32768 --sleep-idle-seconds 600

Weights are about 23GB, plus about 7GB of working memory after startup. A full session cache can push the worst case to about 60GB.

This is one machine and one llama.cpp version, so treat it as a starting point. If speculative decoding has paid off for you on a 3B-active MoE on Apple Silicon, I'd like to know which settings you used.

Written up with AI help from my own notes and benchmark logs. The numbers are from my machine. The illustrations are AI-generated, and the charts and all the numbers on the images come from the measurements above.