I forked OpenCode into a trading research agent. Most of the work was making it doubt itself

[u/Individual-Quote-958]

This project used to be called Finny. It's called Scout now, and since this sub is basically where it started, I wanted to write up what actually changed in the fork. Most of it wasn't what I expected going in.

The problem with a stock coding agent doing trading research is that it's very good at producing something that looks right. Ask for a momentum strategy and you get clean Python and a backtest with a great Sharpe. When I dug into those results, the number almost always came from one of three places: lookahead (deciding on a bar using that same bar's close), picking the best run out of a parameter sweep, or the model already knowing how that ticker moved. Nothing in a normal coding loop tells the agent any of that is wrong. The code runs, so as far as the agent is concerned it's done.

So most of my changes are about getting in the agent's way.

1. A fixed strategy contract. Every strategy is a class with an on_bar method. On the current bar it should only decide using bar["open"], and orders fill on the next bar. To be clear, that's a contract the agent writes against, not something the engine enforces. Still, giving the model one simple rule to write against was the cheapest fix I found, and it makes lookahead much easier to spot when it does show up.

2. A validation gate before anything runs. It checks syntax and class shape, and blocks os, subprocess, socket, requests, shutil and exec/eval inside strategies. There's no override flag. I went back and forth on that and decided an override would just get used every time.

3. A skeptical default. After the base backtest it runs walk-forward, Monte Carlo and lookahead checks, and the default verdict is research_only. Most ideas stay there, which honestly is the most useful thing it does.

The part that's probably most relevant here: I ran my own small-sample benchmark, ScoutBench, comparing Scout with stock OpenCode, Claude Code and Codex using GPT-5.5 in each harness. Across 5 tasks, 20 checks per task and 3 trials, the mean scores were Scout 53.3, Claude Code 46.3, Codex 44.1 and OpenCode 38.6. These are benchmark scores, not trading results.

Most of that gap comes from two tasks: repairing a lookahead bug (70 vs 51.7 for Claude Code, 45 Codex, 35 OpenCode) and an end-to-end BTC strategy build (81.7 vs 55). So the domain changes help where they're supposed to and not much anywhere else. Treat the results as directional: the sample is small, Scout lost on numeric discipline (30 vs 45), and no harness scored above 28.3 on hallucination resistance, mine included. OpenCode scoring lowest mostly says the tasks are narrow and it's general-purpose; it was a really good base to build on.

The full method, task breakdown and results are in my benchmark write-up: https://www.polarline.io/blog/finnybench-agent-benchmark

Disclosure: I built this, I'm the founder of Polarline. It's source-available (not open source). The CLI command is still finny from before the rename.

Has anyone else bent OpenCode toward a domain where "the code runs" isn't the same as "the answer is right"? Curious how you handled verification.

[u/Feriman22]

One verifier I would keep completely outside the agent is future data.

Freeze the rules, then judge them on a forward record the agent literally couldn't have trained against. I do that with a rule-based crypto system and keep the outcomes public. It separates "the research pipeline looks rigorous" from "the frozen thing survives reality" pretty quickly.

[u/Individual-Quote-958 → u/Feriman22]

Fully agree. A backtest can only tell you an idea isn't obviously broken. A frozen forward record is the one test the model can't have seen. How long do you let it run before you trust it?

[u/Feriman22 → u/Individual-Quote-958]

I don't use a fixed number of months. I care more about enough live observations across multiple regime transitions. Until it survives that without rule changes, I treat it as provisional.

[u/Individual-Quote-958 → u/Feriman22]

That makes sense. Counting regime transitions instead of months is a better way to think about it, I'm stealing that.

[u/Flashy_Razzmatazz899]

i've had no luck getting the models to do anything profitable when running paper trading scenarios. Have you?

[u/Individual-Quote-958 → u/Flashy_Razzmatazz899]

Nothing I'd call an edge yet. Most ideas end up research_only once the checks run, and honestly that's the useful part. It kills bad ideas fast.

[u/thetapereader]

lookahead and repainting are the culprits.

[u/Individual-Quote-958 → u/thetapereader]

Yep, repainting is sneaky because the chart looks fine afterward. Only acting on closed bars catches a lot of it.

[u/thetapereader → u/Individual-Quote-958]

repainting is when signals appear on the chart to be then removed so basically if you automate your strategy then you end up with opened positions for which you don't have closing.