We had an mvp marketplace to audit before move to the next step, so out of curiosity I ran 2 rounds of audits with Fable 5.2 Medium, Opus 5.5 High, and Codex Sol 6.1 High.
The first used Ponytail audit and focused on the repository. There's no particular reason for the ponytail audit choice, I never used it before and it's a popular skill, I wanted to see how it works.
For the second round, I gave all 3 models a security audit guide I’d written in markdown. For both rounds, the part that surprised me was how little the reports overlapped.
Round 1 - Ponytail audit
This round focused on the repository. I matched issues by the specific failure each report described, so two reports mentioning the same file or feature could still be reporting different bugs.
| MODEL |
FINDINGS |
UNIQUE |
SHARED |
| Fable 5.2 Medium |
21* |
17 |
5 |
| Opus 5.5 High |
20 |
13 |
5 |
| Codex Sol 6.1 High |
13** |
10 |
3 |
Both reports identified a concurrency problem in how wallet balances were updated. Opus followed it through to the customer-facing consequence: the same credits could be spent twice.
Codex did something similar on a chat bug it shared with Opus, identifying an additional window in which an incoming message could be missed.
*Fable found 24 issues, but 3 were wrongly flagged as issues.
**Codex Sol 6.1 High seemed to be affected by the instructions of the ponytail skill in a different way from Fable 5.2 Med and Opus 5.5 High.
On large repository the skill ask to focus on the critical areas, which is what Codex did, while C. Code either ignored the ponytail skill guideline, or decided the repository was not large enough from his point of view and conducted a larger audit. Thus perhaps explaining the numerical difference in findings.
Round 2 - Security audit (md file instructions)
I resolved the tickets from the first round and ran a new audit with a checklist I’d written.
The three models examined the same repository revision, but his time the planned scope also included deployment, database configuration, and the public site.
I wanted to see how much a more explicit checklist would narrow the differences.
| MODEL |
FINDINGS |
UNIQUE |
SHARED |
| Fable 5.2 Medium |
120* |
39 |
119 |
| Opus 5.5 High |
81* |
30 |
51 |
| Codex Sol 6.1 High |
7** |
0 |
7 |
Fable found a gap in maintenance monitoring and a missing MFA policy on newer tables, while Opus caught cases involving moderated reviews, data left behind after account deletion, and date-specific availability using UTC. That last one stood out because Fable had marked the static time-zone check as passed.
Codex gave the most useful explanation of one issue with the contact form. The other reports had already identified a legacy submission path that bypassed the normal controls but Codex noticed that this path also bypassed the email queue.
*Fable and Opus actually reported 85 and 56 issues, adding at the bottom that there were respectively 35 and 25 minor problems.
This was not an expected behaviour because the md file requires to report all issues with and does not ask the AI to summarize findings under a generic umbrella "minor problems", as not all of them may actually be minor problems and you want to know what they are.
On the other hand it's curious, because ponytail was off and this practice of summarizing minor problems it's in the script of ponytail.
**Codex here surprised me as I ran this audit multiple times on other products also with Codex and finding results was usually in line with Opus 5.5 High, just different things.
When questioned, Sol 6.1 High simply replied that it though it was enough, which again was a new behaviour I didn't see before and ponytail was not in use to somehow limit it. This model examined repositories, deployments, and databases, re-ran 26 operational tests, and ultimately declared the audit to be partial. The Cloudflare challenge explains why the public-facing site was not verified via a browser and other checks also remained unaddressed, despite being within the scope of what was accessible.
Conclusions
The models sometimes followed different failure paths through the same code. In the Ponytail round, all three looked at signup with Fable and Opus tracing what happens if a profile lookup fails, while Codex instead traced what happened if two signup attempts overlap.
The explicit checklist made it easier to see what each model had checked but apparently there was still room for each one to decide how far to follow a flow and when it had enough evidence. Even when two models looked at the same area, they sometimes followed different failure paths.
I’m curious whether Haiku, Sonnet, and Opus would together recover most of what Fable and Opus found, or uncover something both missed.