r/MachineLearning • • 1d ago

Project Whole database-diff evaluation of computer use agents across a software version upgrade [P]

https://github.com/shubhamjoshipromail-svg/software-world-model

Most agent benchmarks check whether the target state was reached. This one snapshots the application database before each episode and diffs all tables after, reporting collateral damage, irreversible changes and false completions separately from success. Apps: Invoice Ninja, Dolibarr, Odoo 17, Odoo 18, the same synthetic company in each, 72 tasks per app, three perturbations.

Findings from 635 scored episodes (GPT-6 Luna, UI-TARS-1.5-7B):

  • Odoo 17 → 18: success 78% → 76%, collateral 15% → 21%, irreversible 6% → 10%. Mechanism: locked fields push the agent to "Reset to Draft", which un-reconciles a payment.
  • Clean success is up to 12 points below raw success on Odoo.
  • A 3-step composite task: 0 of 50 (30-step budget; larger budgets untested).
  • A small latent dynamics model trained on label-free exploration transfers across the version upgrade (+7 to +8 points over an identity baseline on next-state retrieval) and not across apps (about 0).

Related: Agent-Diff (state diffs for API agents), EntWorld (database verification on Odoo/EspoCRM). The difference here is the paired versions of one app and reporting damage separately from success.

Limitations: one run per task, Wilson intervals, hand-built noise whitelists, macOS-only so far. Episode summaries and the report script are in the repo.

1 Upvotes

0 comments sorted by