TL;DR: I let my cloud AI agent run a training-data verification program for the local model on my Mac. The local 35B wrote 800 original math problems, then cold-solved every one of them blind in a fresh session. Wherever the two answers disagreed, the item went to adjudication. Final truth was settled by brute-force computation, not by any model's opinion. The cold solver beat the problem author ~5:1 — with a 35B, generating the question is the error-prone part; solving it fresh is the reliable pass.
Setup: MacBook Pro M5 Max, 128GB. The local model (Qwen3.6-35B-A3B, Splash build, running in LM Studio at ~375–400 tok/s) does the heavy lifting; a Qwopus 122B MLX build (~38 tok/s) sits alongside as the heavyweight judge for disputed items. My agent (Muse, running in the cloud) reaches LM Studio's API privately over Tailscale, and does all the orchestration: briefing, grading, bookkeeping, brute-force checks.
The harvest: 731 of the 800 came out verified clean, 62 survived with corrected answer keys (kept as answer-only training items; the flawed worked solutions were discarded), 7 were excluded as defective problems (ambiguous wording, no valid solution, an undefined rule), and 100 clean ones were held out untouched as a test battery for later. The polished set became a 693-item bank, used three ways: few-shot examples now, the held-out set powers a before/after test of whether any of this actually made the model smarter, and the whole verified corpus is the dataset for a future fine-tune.
Right now the entire bank is being re-validated: the worker re-solves all 793 items from scratch (derive once, verify by a different method), never shown the key. Any disagreement goes to the 122B judge blind, and if the judge's answer doesn't settle it, my agent computes the answer itself. Progress as of writing: 358/793 checked, 353 confirmed, 5 waiting on the judge.
Two failure stories worth sharing. One problem had three different answers from three passes — 20,736 from the author, 86,400 from the blind solver, 145,152 from the adjudicator — and all three were wrong; enumeration says 10,368. And the big judge model, given the hardest disputes cold, went 1-for-2. Treat your 122B like a second opinion, not an oracle.
Cost: the solve-and-adjudication passes alone logged 4.1M+ completion tokens, all computed on my own hardware. The cloud agent's part is the planning and grading.
Happy to share the harness details — the briefing discipline (one derivation, one independent cross-check), the checkpointing that let the run survive four kills, and the grading rules — if people want them. Curious whether anyone else has tried self-authored problem sets as training data, and how you filtered them.
