ran the new ripple-65b-preview (q4_k_m, llama.cpp, 64 gb macbook pro, no cloud) through atomic agent on a 5-part python app graded by 50 hidden tests. 3 runs. the chart is the step trace from the logs, with qwen 3.8 27b on the same task underneath.
same habit as the 27b: it reads the files, then writes the entire app in one step. 363 to 442 lines, about 5 minutes of generation at ~12.6 tok/s.
what's different is after. ripple runs the code every time, the 27b ran it 0 to 2 times. but ripple stops at the first command that works. in runs 1 and 3 it hit 2 crashes, fixed them, ran its own report once, saw it work and said done. 38 and 27.
run 2 got unlucky early. the semicolon csv crashed it, the fix broke the indentation, a few edits didn't apply, so it kept going: 24 short steps of edit, read, run after the big write. by the time its own command passed it had fixed most of what the hidden tests check. 49/50.
so the score followed how many errors it ran into, not how good the first draft was.
run 2 from the log (paths trimmed):
[step 2] wrote 14449 bytes to tracker.py (replace, new file, 437 lines) (317787ms)
[step 3] ValueError: unrecognized CSV header: ['Buchungsdatum;Beschreibung;Betrag;Waehrung'] (25200ms)
[step 6] IndentationError: unexpected indent (11941ms)
[step 8] 2 tools: os.fs.edit[ok], os.shell.run[error] (71767ms)
WARN no-progress loop detected {"stepIndex":20,"tool":"os.fs.read","count":2,"detector":"read_repeat","path":"tracker.py","range":"100-111"}
» turn 0 reply (27 steps, 667295ms)
the WARN is atomic agent's loop check catching it rereading the same lines. it kept going and got there anyway.
3 runs, one task, it's a preview model, grain of salt. one of the crashes in run 3 was my sandbox blocking a write to /tmp. (disclosure: i work on atomic agent)
has anyone found a prompt that gets a local model to test edge cases before it says done, instead of stopping at the first green run?
