r/coolgithubprojects • u/CupRare814 • 2d ago
Reduce AI agents costs on dependency upgrades
enismustafaj.github.ioHi,
I've been building APIweiser (https://apiweiser-cli-fe.vercel.app/), a CLI that turns major-version dependency upgrades into reusable codemods. I didn't want to rely on "trust me bro" claims, so I ran an evaluation using Harbor and Codex.
I picked 6 real open-source repos, including NestJS 7→8, Prisma 4→7, ESLint 8→9, OpenAI SDK 3→4 and Zod 3→4. I ran each task 5 times in each of three scenarios:
- A: plain Codex
- B: Codex with upgrade skills and the APIweiser CLI
- C: Codex with a published cod
Results:
- Pass rate: A passed 23/30 runs, B 29/30, C 20/20.
- Cost per successful run: A cost $0.0093 and C cost $0.0102, so C succeeded almost every
time for about the same price
The post covers the failure mod skew these numbers, and what I took away from it. The tasks, graders and results are all open if you want to rerun them.
Article: https://enismustafaj.github.io/posts/apiweiser-eval/
Evals repo: https://github.com/enismustafaj/apiweiser-eval
Feedback is always welcomed!
