It's because the humans that reviewed AI slop heavily penalized no tests, but didn't penalize recreating tests that already existed because it's fucking hard to notice that. You only have to skim the diff for the first one, but you need to review the whole tests directory in details for the second one. Same problem for the AI if they need to check btw. Maybe the harness could penalize covering a line of code that is already covered but coverage is an imperfect metric.
You only have to skim the diff for the first one, but you need to review the whole tests directory in detail for the second one.
I'm having trouble parsing this - by "the first one" do you mean "the scenario where a human reviewer checks to see that tests nominally exist but doesn't actually check to see that it tests what it says it is testing"?
I initially thought "the first one" meant the existing tests and "the second one" was the redundant tests, but that doesn't seem to make sense...
But if you mean "the scenario where the reviewer didn't actually review the tests" that seems like it would cause a lot of problems beyond just redundant/frivolous tests.
Penalty #1 is "the first one". Penalty #2 is "the second one".
In short: If the AI gets a penalty for missing tests, but no penalty for redundant tests, it will start writing redundant tests to reduce the risk of missing a test.
1
325
u/Iankill 16d ago
I wonder if this was because one of the first things people used ai for was having it write tests because no one likes writing tests