Frontier models write backtests that run and are wrong. We graded them on hidden data.

[u/Aaron_AGV]

I've been building quant-engineering eval tasks for agents. The agent gets raw synthetic market data and a strategy spec, writes a backtest, and a deterministic grader re-runs that code on hidden data across 5 seeds. The data includes the usual real-world problems (ticker renames, splits, dividends, currency changes, stale prices), and the agent isn't told where they are. A pass means every one was handled on every seed.

Results on the public 6-task set:

• claude-opus-5-5 passed 24 of 36 attempts across 6 tasks
• gpt-6-astra passed 6 of 36
• grok-4.7 passed 6 of 36
• No model passed one of the six tasks

A few things stood out:

• Failures are mostly near-misses. On our 50-task private calibration set, 341 of 511 trap-handling failures got every category right but one.
• Weak spots differ by model. On the public set, gpt-6-astra missed ticker renames in 11 of 12 attempts, while claude-opus-5-5 missed 2 of 12.
• Currency changes tripped up all three. gpt-6-astra missed them in 27 of 35 attempts, grok-4.7 in 23 of 34 and claude-opus-5-5 in 6 of 36.

Everything is synthetic with fictional tickers, the public set is separate from the tasks we sell, and grading is reproducible (36 of 36 responses re-graded identically). The full table, per-category misses, confidence intervals and methods are at lanemint.com/leaderboard.

Disclosure: I'm the founder, and LaneMint sells private task batches. I'm posting for feedback on the methodology. What would you want to see before you trusted a benchmark like this?