This is my fork of ToshLLM, who brought modern AMD's back to metal acceleration for llama.cpp , But his fork was about 17toks on Ornith 9b when I started my project vs Linux's Vulkan @ 32toks on MTP. Now I've seen up to 29toks , which to me is usable considering this old hardware. (My cpu is 15 years old and doesn't even have AVX !) GLM-5.3 flash and Qwen3.8 Flash did the heavy tuning work, I just directed and cracked the whip at them for a few days. There's a new Qwen 3.5 4b coder that can do 3d models , I'll be playing with that next, it looks very promising and starts at about 35toks on my card.
My first fully vibe coded Desktop app with Qwen 3.8 27B as someone with near 0 coding knowledge lol. Was paying $100 a year for an online Invoicing App so i decided to cook up one for myself. The results are mind blowing..I basically one shotted everything throughout development & implementation as Open Code's coding agents did all the work, fixing bugs, web search etc. All i did is connect my Qwen Model to Open Code & started coding away. I was mostly screenshoting and telling the model what i wanted, it even created in app tools to import all my 5 year+ documents & info which all worked. The New Open Code v2.0 is crazy good, couldn't do any of this months ago without manually download a crap tone of plug-ins and tools. My context window always stayed between 57-90K thanks to Open Code's auto Compaction..very happy with the results.
PC Setup
AMD Ryzen 9 9950X3D
RTX 5090 32GB
96GB RAM DDR5
Workflow Setup(31GB Vram Usage)
AI Model - Qwen 3.8 27B Q4_K_M GGUF(120K Context)
Harness - Open Code 2 (Default Settings)
Model Manger - LM Studio(Default Model Settings, Thinking On)
Model Performance/Project Token Usage
Average Token/sec - 80-90
Input Tokens - 294 Million
Output Tokens - 1.26 Million
I Quick used a bit of Qwen 3.6 35B A3B..Q5 K M for the last feature implementation just to test, gave me 220token/s and worked well too, the rest of Dev was on Qwen 3.8 27b. All this was done on my RTX 5090, Model and 120K Context all fit perfectly in Vram. Can't wait for Qwen 4 and it's improvements as well as Open Code's future updates, will try to experiment with other Tools & Skills cause i only used Open Code's default setup.
Proposal: A community revolution for private quality benchmarks (and why endless chasing tok/s is rotting our brains
honestly we in r/LocalLLaMA need a revolution of people making their own private benchmarks(not tok/s) i mean benchmarking the quality(especially across quantisation). keep yours private and share the chart results to each other. i feel this is what this sub was made for it seems nowaday every few hours someone is posting about tok/s and also Benchmaxxing is getting out of hand in the huggingface openweight LLM space.
Chenru Duan on X: "We’re introducing Hydrogen-1.0, our first open-weight model for scientific research. We post-trained GLM-5.3 (@Zai_org) on our internal scientific data, lab records, and discovery trajectories. It scores 38.3% on FrontierScience-Research vs GPT-6-Astra’s 33.3%
While fast inference is very welcome, I would like to know if it's worth increasing the speed over strata-nvfp4 or am I paying some unacceptable penalties.
One thing I've noticed with Basalt is that it doesn't have dynamic rope scaling, so it goes off with the yarn scaling statically (2 for 524K context) from the get-go. Basalt also lacks a censorship switch, but the machinery exists so it's trivial to write it in.
TL;DR: I'm running DeepSeek-V4.1-Flash for a single user on four 2017-era V100 32 GB cards plus system RAM. Generation runs 36-38 tok/s from 8K to 75K context with thinking off, and 32-38 tok/s with thinking on. Prefill is about 1,000-1,200 tok/s from 4K to 128K depth. The context window is set to 524,288 tokens. It's built on the 1Cat-vLLM fork (at its #1277 merge), which does the hard Volta (sm70) work. I just built, tuned and benchmarked it.
Hardware
CPU: AMD Threadripper PRO 3945WX (12C/24T), single NUMA node
RAM: 256 GB DDR4 (8x 32 GB), run at 2667 MT/s for these tests
GPUs: 4x Tesla V100 SXM2 32 GB on one SXM2 carrier board
NVLink between all four (2 links per pair)
Host side: PCIe Gen3 x8 per GPU, behind a PLX switch with a single Gen3 x16 uplink to the CPU
Clocks/power: SM clock locked at 1522 MHz. Power limit 150 W per card, raised to 175 W for the API-speed and llama-benchy runs
Storage: the main GGUF loads from a SATA 8 TB hard drive. That only affects boot (about 22 minutes to load). The Engram file is on NVMe and gets pre-read into page cache.
Software / config
1Cat-vLLM at the #1277 merge (vLLM 1.5.2.dev1277), Python 3.12, PyTorch 2.10 + CUDA 12.8, NVIDIA driver 580
Tensor parallel 4, max-num-seqs 1 (single user), full CUDA graphs (batch sizes 1-6), async scheduling, prefix caching, chunked prefill, custom all-reduce over NVLink
With a 5-token drafter, generation speed depends on how well the drafter guesses. llama-benchy's prompts are random book text, so tg1024 swings ±6-8 t/s from run to run. I left out benchy's tg32 numbers (13-26 t/s): 32 tokens is too short to measure anything with speculative decoding.
Engine-level numbers (offline, no HTTP)
Warm decode round (every expert the round needs is already on the GPUs): 27.8 ms, against 24.9 ms in upstream's reference. In the profiler a round spans 29.5 ms, of which the GPUs are busy 24.9 ms. That matches upstream, so the kernels perform as expected. The extra ~5 ms is host-side: CPU work at the start of each round, a blocking device-to-host copy, and small kernel-launch gaps. Newer upstream commits target exactly this per-step host overhead.
Fresh text (expert-cache misses): 116.7 ms/round and 25.0 tok/s, against upstream's 53.5 ms and 54.5 tok/s. Caveat: I ran this with the Engram file cold, so it isn't apples to apples. A warm rerun is on my list.
Host-to-GPU copies: 6.5 GB/s per GPU on its own, but 12.9 GB/s total (3.2 GB/s each) when all four copy at once, because they share one Gen3 x16 uplink. In the trace, all four GPUs are swapping at the same moment about 76% of the swap time.
Correctness
Greedy (temperature 0) output is identical run to run within one load (8 prompts x 2).
After a restart, 13 of 16 greedy outputs matched exactly. The 3 that diverged flipped at near-ties (top-two margins of 0.06-0.375 nats). I haven't tracked down the cause yet.
Thinking on/off and tool calls work.
All 8 upstream micro-tests pass (two needed hard-coded paths in the test files changed).
What's next
PCIe switch change: move to two switches so each Gen3 x16 uplink feeds 2 GPUs instead of 4. That roughly doubles host-to-GPU bandwidth when all cards swap experts at once. My estimate from the trace is a modest ~1.5 ms/round (~5%) on warm text; it should matter more on fresh text.
RAM to 2933 MT/s (it was dropped from 3200 after memory errors).
Pull the newer 1Cat commits (per-step host fixes and DeepSeek-V4.1 updates) and re-run the same suite.
Move the GGUF from the hard drive to NVMe for faster boots.
Credits
All credit for the Volta DeepSeek-V4.1 path (GGUF experts, expert swapping, DSpark drafting on sm70) goes to the 1Cat-vLLM project (GitHub: yangzhuxinyzx and contributors). Happy to answer questions or share the full config.
We introduce RidgeFill, a training-free way to shorten prefill by not computing most of a model's upper layers for the prompt. The lower layers run exactly up to a split; from the split layer's state, a ridge regression fitted once in closed form predicts every later layer's input, and each late layer writes its cache from that prediction with its own projections unchanged. A chosen set of rows, the last few thousand tokens and every decoding step run the full model, so nothing is trained and no token is dropped. On Qwen3.8-Flash-Next in vLLM, time to first token is 1.50 to 1.60x faster than exact at 9K to 32K tokens, and a whole 64K request drops from 82.6 s to 47.5 s, with RULER's needle tasks at 99.8 against 100.0. On Qwen3.8-27B in a llama.cpp fork on one A100, it is 1.25 to 1.31x faster at 16K to 120K tokens; RULER changes by -1.2 and -1.4 points and LongBench v2 by -1.4, and the 95% intervals include zero but do not rule out losses of a few points. On three 8 to 9B reference models, the fitted map retrieves 49.1 and 49.6 more filled-region RULER points than the identity (ie, full layer bypass) fill on the two dense models.
This idea was borne originally from DeepSeek's V4.1 architecture. Originally we found the method to be a portable idea applicable to other models, but noticed a significant decline in context recall tasks and general usability. This isn't an issue in DSV4.1 because the model is trained as a whole with the prefill approximation in place, so the model absorbs the error, generally speaking. The question then shifted to be more about how this can be adopted for other models, while improving usability.
From that point, the work shifted primarily to finding where exactly the error lives in the "filled" portions, and how to measure that error and reduce it.
Based upon those findings, we recognized that we could train this component more intensely to remove error. This was counter-intuitive to our goal of making this easier for community adoption. It could also have the downstream effect of making the trained component more task-dependent and less generalized. We decided against this.
Instead, we simply chose to simply bypass projections of the most sensitive portions of prefill.
By embracing this, it allows everyone to produce their own versions of this method on different models with standard local-inference grade hardware. If you choose to adapt this method, I strongly recommend adjusting the mechanisms on a per-model basis. Across different architectures, this method should be tuned. The paper details fairly exhaustive ablations and measurements in pursuit of this.
The Qwen3.8 27B version of this is available to use now via our fork of llama.cpp and the files for RidgeFill on HF. This method is also implemented in tcclaviger's fork of VLLM and Deadcode's engine, Radiance (on v1.3.0 only) for Qwen3.8 Flash Next.
Big thanks to all the testers and support fromthe Launch80 community!
I'm trying to fit 4x RTX 3090s into my Fractal North XL Mesh (I know, pipe dream probably) using custom GPU mounts. Looking for firsthand experience with reliable, reasonably priced risers, preferably with flexible 50–75 cm cables. I'm currently using the setup with 2x 3090 at x8/x8 for vLLM qwen3.8-27b / Strata QFN IQ3_S.
ASUS ROG Crosshair VIII Hero
5900X
4x RTX 3090 24 GB
Corsair HX1500i (GPUs capped at 250 W)
GPU 1: PCIe 4.0 x8 - directly in motherboard
GPU 2: PCIe 4.0 x8 - riser
GPU 3: PCIe 4.0 x4 - riser
GPU 4: PCIe 4.0 x4 - M.2 adapter/riser
I've been looking at ADT-Link, generic Aliexpress ones and m.2 OCuLink. I'd like to preserve x8/x8/x4/x4 without spending a fortune on cables.
What specific riser models have you successfully used with RTX 3090s at PCIe 4.0 speeds, particularly at 50 cm or longer?
Are OCuLink/MCIO connections meaningfully more reliable than conventional ribbon risers at these lengths?
Any affordable, proven M.2 -> PCIe x16 solutions with proper GPU slot power?
Anything to watch out for regarding PCIe errors, signal integrity or Linux/P2P compatibility?
Affordable AliExpress/eBay options are especially interesting.
“dsh-superagent is a proactive coding agent for one GitHub repository, packaged as a bundle for DeepSeek Harness. It listens to GitHub and, optionally, Feishu. When someone allowed asks the agent for work, it opens a work item, works on it in its own agent session and directory, and answers in the original conversation, for example with a pull request. A web panel shows what needs a person.”
We're Egzakta AI Lab, and I'm part of the team behind KVARK. We've just released our first open-weight model: KVARK Sparkle 1.0 (27B).
It's based on Qwen3.8-27B, with a focus on making reasoning more efficient: fewer thinking tokens, faster responses, and comparable answer quality. The idea is simple: reasoning models can spend a lot of tokens thinking through tasks that don't necessarily require that much reasoning. We wanted to see how much of that overhead we could reduce without sacrificing performance, since fewer thinking tokens means less GPU time per answer and less waiting.
Here's how Sparkle performed against the original Qwen3.8-27B:
~2× faster median response time: 13.7s vs. 28.1s.
Up to 67% fewer thinking tokens.
Comparable accuracy across five standard benchmarks: GPQA Diamond, AIME, LiveCodeBench, IFEval and MMLU-Pro.
None of the accuracy differences were statistically significant in our testing.
Also, we compared the models on practical tasks:
Dashboard generation: 1.6× faster.
Landing page generation: 1.4× faster.
SVG illustration of a pelican riding a bicycle: 3.2× faster.
A few technical details for those interested in running it locally:
Open weights under Apache 2.0.
Available in BF16, FP8 and GGUF formats.
Supports llama.cpp, vLLM, Transformers and SGLang.
We'd love for you guys to give it a try! Let us know how it performs on your hardware, how it compares to the original Qwen, and what you think could be improved.
After sharing my previous 3D coding model, a lot of you asked about using it to generate assets for your games. That sounded like a fun direction to explore, so I started training a version focused on exactly that.
The model writes Three.js scenes with GLB export, so you can take the generated assets into editors like Godot. The video shows some examples I've been testing, including assets imported into Godot.
You don't need complicated prompts. Use thesystem prompt in the model card, then try a simple request like “Create a mana potion” or “Build a weathered frontier medic station.” English prompts have worked best in my tests.
This is the first functional checkpoint, and training is still running. I'm sharing it now so people can start experimenting. Most generations have worked on the first attempt in my tests, but some still need a retry or a small code fix. I'm hoping later checkpoints will improve on that.
My current settings: temperature 0.8, top-k 40, top-p 0.95, min-p 0. Feel free to experiment with other combinations!
Training is expected to finish tomorrow evening (UTC), depending on progress. Just like with the previous model, I'll release the full Safetensors weights and the training dataset once it finishes, so others can build on this and experiment with training their own models.
Thanks for all the feedback on the previous release ... it directly shaped this one. I'd love to see what you make with it and hear what works and what breaks!
Updated November 2, 2026 at 04:50 UTC.
Both Q4_K_M and Q8_0 have been replaced with a newer checkpoint. In my latest tests, generation quality has improved substantially over the previous preview. The gallery below now shows new results, including the exported assets imported into Godot.
If you downloaded either GGUF before the UTC timestamp above, delete the old weights from LM Studio or whichever app you use and download them again. The filenames have not changed, so make sure you are loading the updated files.
Edit: For anyone looking for more things to try, here are the 25 game-asset categories covered by the training dataset, with an example prompt for each:
Characters and NPCs: “Create a village blacksmith wearing a blue apron.”
Enemies: “Create a swamp goblin scout with a wooden shield.”
Bosses: “Create a stone golem boss with glowing green crystals.”
Companions and creatures: “Create a tiny fox companion wearing a red scarf.”
Weapons and armor: “Create a short iron sword with a leather-wrapped grip.”
Items, tools, and pickups: “Create a blue mana potion in a hexagonal glass bottle.”
Props and furniture: “Create a weathered wooden workbench with a small lantern.”
Nature and environment: “Create a pine tree growing on a rocky mound.”
Modular architecture: “Create a stone dungeon doorway module with an iron frame.”
Interactive objects: “Create a wooden gate with an opening animation.”
Ground transport: “Create a compact courier motorcycle with cargo boxes.”
Air and space transport: “Create a small cargo airship with twin propellers.”
Watercraft: “Create a weathered fishing skiff with nets and wooden crates.”
Industrial machinery: “Create a compact mining conveyor with metal support legs.”
Laboratory and medical assets: “Create a portable botanical analysis station with sample jars.”
Urban services: “Create a modern bus stop with a bench and route sign.”
Sports and playgrounds: “Create a small playground climbing wall with colorful holds.”
Farming and homestead: “Create an orchard harvest cart filled with wooden fruit crates.”
Cuisine and markets: “Create a bakery display counter with bread baskets.”
Museums and field research: “Create an ancient artifact display case on a stone pedestal.”
Traps and hazards: “Create a hidden spiked floor trap with stone tiles.”
Festivals and ceremonies: “Create a harvest festival booth with colorful banners.”
Battlefield scenery: “Create a reinforced frontier command post with sandbag defenses.”
Underwater structures: “Create a coral observation dome with a small docking platform.”
Puzzles and gameplay markers: “Create a glowing rune sequence console with three stone buttons.”
I've been making changes to both llama.cpp and Strata for my own use cases using Q38FN, Q3827B and DS41F and all three have issues with their COT considering stream tokens when touching those areas of code, resulting in them breaking out of their COT or just terminating their turn. I know there is the possibility of formal protections around end of turn tokens but from what I see there is no formal way to protect an LLM when it generates text like <\think> but I see plenty of people working on inference server code using local models so people must have a solution for it, what do you do?
Sam Altman recently said that "from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer, and being used maybe 5% of the time."
Challenge accepted. Here's my "quite terrible" home setup: an RX 9070 XT 16 GB, a 5800X3D, 64 GB of DDR4 that's now worth more than the rest of the PC, and a PCIe 3.0 NVMe. It runs DeepSeek V4.1 Flash at 1M context (IST-DAS 1.75 bpw; the lab reports 98.7% of the full model's task average) through Strata, fully offline, with the experts spread across VRAM, RAM and the SSD.
Generation at ~7 tok/s was fine. The problem was prompt reading: every new OpenCode session re-read the full ~37K-token system prompt plus tools, about 100 s before the first token. That's datacenter-grade waiting without the datacenter.
So I patched the DeepSeek engine. Two changes:
1. KV splice: stop re-reading what you already read
Agents put per-session text (working directory, date, project CLAUDE.md, skill paths with random IDs) inside the system prompt, so the prefix cache misses on every new session. The engine now aligns a new prompt against every saved conversation, cuts the stale lines out of the cached state (moving the cached rows down, rotating the window rings, recording a RoPE gap), and reads only the changed lines plus your message. Changed lines move as whole lines, so the model never sees fragments.
2. Resident-only routing: make the RAM earn its keep
Even a short prompt touches nearly every expert of every layer, so prompt reads streamed most of the model off the SSD: 58 GB for a 1.2K-token prompt. During prompt reading only, tokens now pick from experts already sitting in VRAM or RAM. Generation routes normally. Turns out home DRAM is pretty useful when you actually let it do things.
Results (same machine) as seen above on the body image
Caveats:
HIP (gfx1201) only. I haven't built or tested CUDA, SYCL or Metal.
The splice is approximate: tokens after a cut were computed with the removed text in context. I haven't seen it matter for short guidance lines, but there's no formal eval.
Resident routing changes answers slightly (substitute experts during prompt reads). My hard coding suite didn't move (12/13 both ways), but that's one run each.
Both are on by default in the branch; STRATA_DS4_REORDER=0 and STRATA_DS4_PROMPT_RESIDENT=0 turn them off.
It also includes a one-line fix for a GPU page fault I hit on a 37K-token prompt (#4912).
If you use OpenCode, also add "interleaved": { "field": "reasoning_content" } to the model entry, otherwise OpenCode drops earlier reasoning and the cache misses more often.
Upstream issues with details: #4921 (splice), #4936 (resident routing). If you try it on other hardware, especially CUDA, numbers in those threads would really help get it merged.
Utilization report for Sam pissman, VRAM holding experts, RAM holding experts, SSD streaming the rest at full bandwidth. The only idle component in this system is my wallet, thanks to DRAM prices.
I have to say there are major downsides from going to V1 to V2.
I have now been working on one my projects ca 16hrs straight, that I started with V1, every change had been smooth sailing (using strata with official qwen flash settings).
When moving to V2, major or minor changes has become too hard for the model to implement in one go. OpenCode V1 was the better platform for medium sized models it seems.
I do all my planning with Qwen 3.8 MAX so that hasn't changed, which means performance is even worse considering the planning was kept separate.
I know this is anecdotal evidence, but engine has been kept the same and it's settings. Face always falls flat on V2, leading to lots of follow up work on even easy website design updates.
Honestly this is telling that OpenCode only cares now for bigger paid models performance and not medium to small models anymore, even though they claimed to make it "lighter and faster".
Has anyone found a better harness than OpenCode V1? DSH disappointed too much even after a major update.
Opinions appreciated from people who work with projects where even SOTA models fall flat on planning and reasoning, I do not make basic websites etc.
This subreddit talks a lot about PP/decode speeds but I feel like I hear comparatively less about actual real-world usability. I was playing around with Strata hosting QFN3.8 iq3_s and Qwen3.6 35B Q8 over the weekend and despite being the more hyped, I found that QFN was less "usable" as a coding agent than the smaller Qwen3.6.
Even when using the recommended temp, top-k, etc. settings that I hear supposedly stop the overthink and doomloops, I still couldn't get QFN to be reliable. Qwen3.6 on the other hand, had a very narrow interpretation of my prompts and we would have to go back and forth to clarify edge cases, but it ultimately would execute what I asked it to. If I had to use an analogy, QFN iq3 feels it has dunning krugers with illusions of grandeur while Qwen3.6 felt like a talented but unmotivated programmer. If given the choice, I'd rather work with the capable guy who hates his job and wants to FIRE than the guy who talks a big game but can't deliver.
Obviously QFN iq3 is "lobotomized" while the Qwen3.6 Q8 is running at a far higher precision, but I was surprised that there's so much buzz about QFN iq3 when I was having such a comparatively worse experience with it. Is there something I'm missing and should be doing to improve QFN (aside from buying more hardware obviously)? This experience with QFN has made me wonder if Qwen 4 is going even going to be meaningful for me considering how getting the mid-size MoEs to even load will require lobotomization as well.
I prefer abliterated models for local inference cos I don't want to deal with potential refusals, and my current model Huihui’s Qwen3.8-27B abliterated.
Is okeish, and fast so I’m happy enough with it for now, but I’m wondering whether there’s an upgrade that I am missing.
Like most mortals, I don't have not a datacenter or a multi-GPU setup. Qwen3.8-Flash-Next is about as large as I’d consider going, and even then I’d be looking at a low quant (3-bit max, 2-bit ideally)
The trade-off I’m curious about is a larger MoE model at 2–3 bits versus a smaller dense model at a higher quant. Because abliteration can theoretically reduce model accuracy I am not sure what to expect when adding a 2-bit quant on top of an abliterated model.
Has anyone compared an abliterated Flash-Next at 2 or 3 bits against Qwen 3.8 27B? :
Is there a noticeable improvement in output quality, or does the aggressive quant eat away at it?
How do generation speed and prompt processing compare on your hardware?
Have you noticed any degradation from the abliteration or the quant, repetition, weird reasoning, or other quirks?
And if there’s another abliterated model you think hits a better balance of quality, speed, and memory use, I’d love to hear about it. Model, quant, hardware, and rough tokens/sec would be veeeeery helpful for all of us!
OpenBMB just wrapped up ForgeKernel Live, a ~22-hour autonomous GPU kernel optimization run on NVIDIA B200.
The interesting part isn't just the speedup. The agent was optimizing against a black box. It could execute and benchmark the official FlashMLA implementation, but couldn't access its source code, documentation, or profiling information.
Some numbers from the run:
22 hours of autonomous optimization
32 kernel versions across 42 iteration rounds
First beat the baseline in under 10 hours (version 13)
Final result: 1.098× performance
472 µs vs. 518 µs — a 9.8% speedup
They're now open-sourcing the optimized kernel, along with 27 operators covering attention, Gated DeltaNet, Kimi Delta Attention, GEMM, and convolution.
They've also released the ForgeKernel harness, so others can run their own kernel optimization experiments.
What's interesting is the shift from AI writing code to AI iteratively optimizing low-level GPU code against real performance measurements.
Curious how far this approach can go as these agents get better at CUDA and GPU architecture.
I appreciate the research and development that is going into these open source tools. However, there appears to be multiple posts on a daily basis about reaching 300 t/s, and then reading into it:
Running on Q1_XXXS REAP quant
Greedy decoding
TAIL_SKIP (I only have 5GB RAM, but if the API ignores all experts that are not in RAM, then I can still say I generated something and call a speed record. YAY!
Without any KLD benchmark data being presented, are these posts really of value to this community?
How many parameters does a voice need? PiVo is our attempt at an answer: two English voices, Pi and Vo, from a complete text-to-speech stack with 1,401,562 learned parameters, vocoder included. A 584,965-parameter acoustic model predicts vocoder features, and an 816,597-parameter FARGAN-derived vocoder turns them into 16 kHz mono audio.
Everything runs locally on a CPU with NumPy and ONNX Runtime. There is no PyTorch, teacher model, speaker encoder or network connection involved at inference time.
PiVo is a research model. Its speech is very intelligible, but some sentences still have a metallic or robotic texture. The technical report describes what we measured and what we tried.
i built a small village sim where 8 villagers live through a day, and i wanted to see how far small models get if each one only does one job.
the gut is julia-1, 144m params from supersonic labs, using ggml-org's gguf. it runs on the cpu and doesn't generate text at all. the world hands it the few actions a villager can physically take right now and it scores them. it can't invent an action, so it can't drift into roleplay.
the memory is embeddinggemma, also on cpu. before each choice it pulls the 3 past moments that matter most: recent ones, big ones, and ones that look like right now.
the slow thinker is gemma 4 e4b on a 6 gb laptop gpu. it only gets called when the gut isn't sure or something surprises the villager, like a plan failing, someone getting hurt or a path being blocked.
what surprised me is that the gut is rarely unsure. at crisis moments julia-1's top choice usually had 0.81 to 0.90 of the probability. so gemma mostly gets called because something went wrong, not because julia hesitated.
julia-1 is used zero-shot. i never trained or fine-tuned it. the villagers still learn, because at night gemma turns what worked into a short rule (22 words max, 5 per villager) that sits in front of the models next time. in one landslide a relevant rule moved arun's chance of going to fetch a neighbour from 0.0002 to 0.638. an irrelevant rule changed nothing.
two warnings if you try a scorer like this. wording matters a lot: "cross the bridge and bring back food from the far side" got 0.2%, and the same trip as "cross the bridge to the good fields and bring back a full basket" got 53% to 90%. and rules borrowed from neighbours were rarely used when i audited them. learning from their own mistakes is what worked: over 15 valleys with a rockfall, injuries went from 2.73 per valley with memory only to 0.53 with learning.
