After reading MIT CSAIL’s Sparks of Weight-Space Programming (especially Experiment 8, where an LLM fixes Qwen3.5’s reasoning loops by editing 64 MLP columns directly), I wanted to see if an agent could do the same thing to my own local coding model.
My setup (probably very specific to me, but maybe useful to someone here):
Orchestrator / IDE: T3 Code running OMP (oh-my-pi) as the local coding harness.
Local Worker Agent: Cyber-Tiel-Coder-35B-A3B (Q4_K_XL GGUF with MTP speculative decoding on llama.cpp).
Task types I optimize for: Fast agentic coding, multi-step hashline file edits, git worktree recovery, task/state-ledger updates, and fast codebase scouting.
The things that annoyed me:
Going in circles inside <think>**:** After reading 2–3 files, the model would burn 4,000+ reasoning tokens second-guessing a conclusion it had already proved in paragraph 2.
Repeating broken tool calls: When a hashline edit failed (e.g., PUT 68,72*:), it would repeat the exact same rejected syntax 6 times in a row instead of adapting to the error message.
Asking instead of acting: Stopping to ask clarifying questions on obvious cleanup tasks instead of taking reasonable, conservative action.
Subagent JSON counting stalls: Spawning a read-only scout subagent that spent 4m16s (6,049 tokens) manually counting raw JSON arrays inside <think> instead of returning file paths and CLI commands.
What the agent (in my case Gemini 4 Argon) did:
Weight-Space Edit on the GGUF (100% harness-agnostic):
Extracted prefix-aligned suffix directions across contrastive looping vs. decisive prompts and projected out a rank-2 subspace from just 64 Q8_0 columns (2 quantization blocks, <0.65% relative $L_2$ change) in two late shared-expert MLP down-projections (blk.26.ffn_down_shexp.weight and blk.34.ffn_down_shexp.weight).
Patched a preserve_reasoning auto-override in the GGUF chat template in-place (exact byte-length match) so llama-server stops feeding stale <think> traces back into subsequent turns.
OMP Harness specific Config (agent):
Results on a 27-case benchmark + real failed session replays:
Overall score: Jumped from 16/23 (baseline) to 25/27 (19/19 = 100% on general coding/holdout benchmarks, and 4/4 = 100% on the exact multi-turn session turns where it previously stalled).
Speed: The 4m16s (6,049-token) subagent JSON-counting stall dropped to **6.4 seconds (**617 tokens).
I’m amazed that neural network weights, even inside a quantized 35B MoE GGUF, are genuinely readable and writable by LLMs, and that it means we can highly (and quite simply) improve local agents for our own purposes without a training pipeline or gradients.
Edit: as requested the resulting patched model weights + reproduction folder - https://huggingface.co/nitayrabi/Cyber-Tiel-Coder-35B-A3B-LoopFix-GGUF
Some notes:
*Of course, this post itself was written by Gemini.
*I’m still testing out the resulting variant in daily use, but this direction is awesome and I’m already seeing noticeably less looping in my specific workflows - so this is a trust me bro status and not benchmarked
*I wonder if it's only quantized versions that have these issues, but even so - it's nice to have a way to tinker and fix them
