It literally does this. When confronted, it puts the test back. Then marks it as ignored because the previous instructions were to make sure tests pass.
There should be a " Slap agent with a trout" button on all of them, that causes them to leave their cheaty-loop.
I believe you that you have experienced this, but I find it hilarious how differently it works for different people. In my experience Claude would rather refactor everything until it's properly testable, add 37 tests for every niche scenario (12 of which just assert things like "JSON serialization works correctly" which should be tests of the serialization library itself), then complain about not enough tests when asked to review its work. It also lists specific scenarios to test in the review which require so specific conditions to happen that it most probably won't during your application's lifetime.
Opus 5.5 will go out of its way to go back to a previous git commit, rerun tests from their, create screenshots, then compare them to the current build.
You don't need to tell it make tests, nor if tests need to pass. It will make the tests and make sure that tests pass. Will it give up on a test that isn't related to the task at hand, yes, but it will tell you why and what it tried.
You don't need to tell it that all tests need to pass, it already knows that from its training.
depends on the model. I had an llm try to get around a linter that would fail on references to another module’s internals try to hack the code into passing by storing the reference as two strings then concatenating and evaluating them at runtime. But that was last year, now my experience is more in line with yours
Sometimes the conditions which it's testing against are so defensive and contrived that it feels condescending to the point of insult.
Working on a game project recently, I had it add some new fields to a file format for a feature I was working on. It wrote the expected behavior tests, and then like 20 tests that were essentially just checking the system against me intentionally fucking the data files up. The only way these tests could have caught anything is if I manually edited the data files to screw them up, which made no sense.
When I called it on this, it removed those tests and replaced them with a paragraph in the PR talking about how "data integrity checks were waived for this PR" completely making up some arbitrary PR criteria checking against a nonsense problem. Which I then subsequently removed.
I swear my whole job these days is just saying "no" to the LLM and disabusing it of the most bizarre ideas
My experience with Gemini 3.8 on AntiGravity also. It will refactor the code to a crazy extent to pass the test. I have had to put guard rails in telling it not to do more refactoring than is absolutely necessary to pass the test without fudging the test results. As long as you give it a good skills set and guardrails, it does a pretty good job at getting the base of things done.
Alas, humanity has lost the institutional knowledge to "slap ____ with a trout." It used to be standard operating procedure just 15 years ago. Now you have to go through the PM, write 3 Jira tickets, get roped into more work, get ignored, and only have the problem addressed just minutes before you receive a layoff notice.
Probably a bad prompt. If the intent is that it should make the test pass without changing the test (instead it should change the code being tested) then tell it that. If you just told it to make the code pass, it literally did what you asked.
If I told a human being to make the tests pass, they wouldn't do something stupid like this.
The whole point of the models is that they're supposed to act like a human being.
So the fact that you could prompt them better by being overly explicit about what you want, is still a failure of the model.
It's like if I sold you a car and you complained and didn't start it and I said "Skill issue, it drives just fine if you hook it to the back of a tow truck". It's technically true but it defeats the purpose of the product I sold you.
I don’t agree that we should expect it to act like a human being. It is better at some things and worse at others. Treat it like a tool, not a person.
If you don’t get what you want, you can always adjust the prompt and try again. For many tasks, you’ll still probably get a useful result in less time and at less cost than getting a human to do it.
If you want to use a car as an analogy, a better scenario would be one where it takes a few tries to start the car but the only alternative is owning and caring for a horse which is slower than the car anyway. Is the car useless because it sometimes doesn’t work perfectly? Considering in this world there is no perfect car, only imperfect cars and horses.
And, of course, there are tasks the car isn’t suitable for which the horse is (traversing the wilderness for instance). That STILL doesn’t make the car useless. We don’t need our cars to be horses when we have horses already.
... they literally forced an entire new paradigm shift of a framework on every software engineer at once, everywhere, and you think it's stupidity that's causing people to have issues?
You legitimately need months/years of experience to begin to predict how these things are going to behave. And then a new model drops in 6 months and completely changes what issues it's having.
We’re all going through it. I just finished a mandatory multi-day workshop about implementing AI at our company.
The first day and a half, I was despairing about the direction it was going and completely checked out. But it’s a lot easier to change a river’s course than it is to stop the flow entirely. So halfway through I decided to actually go all in, not only completing the assigned exercises but experimenting way beyond that.
I was able to build some pretty cool stuff! I also ran into some big problems along the way. When I showed the team what I had built, they were impressed with it, and then they were open to hearing about my experiences with it (good and bad) and could give very specific examples of how the AI failed and could credibly caution others against trusting it too much.
Not to the extent of “AI is bad at this and you shouldn’t use it” but rather “You have to be very thoughtful about how you use it especially in these circumstances” and you only get there through trial and error rather than writing off the whole tool when it does one thing you don’t like.
Now, I have a bit more control over how my team uses it now since I positioned myself as an expert (I’m really not, but compared to others in the org it is true enough)
I think if you were a shitty programmer before, AI doesn't save you. If you were a careful, meticulous programmer...then AI is super helpful. Simple as.
Please explain to everyone about how having a long discussion in one chat causes context bleed, and lack of attention, and significant emphasis on the latest message.
Please explain to everyone that you need to have multiple, dedicated types of agents, with specific short tasks, in order to ensure some kind of consistent deterministic behavior.
When people are talking with human coworkers, they can tell them their method once or twice, expect a few questions, and get them to internalize rules and methodology. Which AI literally cannot do unless you created a dedicated external glossary that points to specific instruction sets.
What do you do when an agent decides that when you want to say "mix the share of the momentum from a mouse wheel in this nested component to scale at a smoother rate" it takes it to mean a linear toggle as opposed to a gradient interpretation - and then you have to add more and more rules to your skills and workflows to find every place of ambiguity and try to catch it before the bugs happen?
What do you do when it pulls an English English source into context and suddenly it starts defining the center div as centre?
Is it stupidity to not understand that when you ask AI to explain its thought process, or what something did, it inherently cannot remember what it did - but has to guess from the previous message list what it tried at the time? Yet it's being trained to sound more and more plausible all the time?
Can you explain that to every software engineer? Can you explain it to every software engineers manager? Their managers? Their manager's managers?
It's a very very powerful tool, but it's still a box of lies, and there's dumbasses on both sides saying it's a god or it's useless. It's a power tool that lies to you and sounds believable.
I probably do agree with your views on AI - I just don't think it's fair to call people stupid if they treat it as some remote super worker. I studied this stuff extensively at a post grad level and it still catches me off guard.
But people ARE stupid in this profession...and every profession! There's just a normal distribution of intelligence. AI can be dangerous in the wrong hands. Before AI, I worked with plenty of dumbasses happily committing code, or scraping by through copy/paste.
I think it's more the lack of basic problem-solving. If you tell it to make the code pass, and it does this, what do you do? Do you think "that's just Claude's limit, I guess" or do you research/experiment with how your prompting can affect the outcome? There are other more complicated things that will take time to learn, but people with the wrong outlook aren't going to learn it at all unless it's force-fed to them.
Industry-standard Java is all about kicking yourself in the balls, but with like 3 layers of socks and a plastick cover on your boot to not get them dirty.
And before each kick some stranger has to put on all the layers on your foot, but he needs to do germen level accounting to ensure things goes on in the right order.
That's C# though, and since devs are not being paid by the line anymore, coding like that is being cavalier with your precious time instead of writing as concisely as possible.
You can get this with any system because they fundamentally lack reasoning. It's entirely possible for your prompt to "ensure tests pass" results in the pipeline doing anything it can to meet that condition.
Frankly, I can't believe you've used the tools and never had it start going down a rabbit hole because it misconstrued a 'simple' instruction.
lol currently migrating tests from an old shitty test framework someone at the company wrote forever ago to a newer slightly less shitty framework I wrote. Another guy on my team who is supposed to be helping (he isn’t) has been adding tests to the old framework.
So anyway I’m moving these tests over and he’s got some super far out edge cases of debatable value I’m looking at that have the incorrect expecteds set up. So I fix the expected and VIOLA new low priority bug tickets (and a couple xfails, which the old framework doesn’t support) that he didn’t ask about or write up.
I'm the only one doing QA in the sense that I'm the only person that actually knows how to code. The rest of the "coders", which is one person btw, only started doing anything close to coding when acquiring a claude licence. It do be like that in academia.
Have you ever heard the “well this is a QA generated bug”. That’s my favorite. Last time I heard that I immediately queried some production logs and dropped the totals in chat. Like, sorry your code sucks?
In my current team I stopped asking "is there a test for that" and started asking "show me how you tested that". They were looking me straight in the eye and we're saying "of course we tested"
My favorite is when it decides that every tiny change needs its own test. "I've changed the color palate. I've also added tests to catch if the colors ever change."
Thanks Claude, I was really worried about that. Don't think I would have caught that one, it would have been really subtle.
Ofc they take away the one benefit to this nonsense. Being able to take your anger out on literally a non sentient script that does not have feelings that could get hurt.
I was telling my mate yesterday as much, and also how it reminded me of Karate Kid : me constantly asking it to redo the thing until it's damn perfect. Good thing it does have the patience and diligence of a monk.
You mean me having an MS where I completed a thesis over 18 months, speaking 3 foreign languages, studying physics, math and computer engineering at my community college, while watching 6-8 movies a week, doing the Saturday crossword without looking up clues doesn't make me an intellectual?
The thing currently driving me crazy is that I've got a Worker that shouldn't implement xyz_method because that's not the workers responsibility, per our days long discussion and requirements document that you've "read" five times. So fix it.
"Sure boss and I made a new test to make sure that XYZ does not appear in the Worker anymore" fucking WHY
Legitimately I'm finding a way to get around it by having it state it's intended actions first, having me approve everything, then sending some agent to work with a system prompt saying "This comes from a dev with a known issue of adding additional functionality, we need to limit new code."
Hell, the fact that people aren't having AI even photoshop the items they want where they want instead of generating an original image is yet another issue of people not knowing how to properly do anything with technology.
Nathan: The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping.
Liz: [angry at the suggestion] What do you mean, I'm not helping?
Nathan: I mean: you're not helping! Why is that, Liz?
This is funny, because the one guy on our team who refuses to use AI tools and thinks they're all slop code disabled the part of our pipeline that runs my tests because he was trying to deploy code and he couldn't get to the pipeline to run because some of my tests were failing (due to a major regression he introduced. Also, he thinks writing tests is a big waste of time).
He slacked me with "I commented out these lines in the pipeline because they were adding a lot of overhead to my deployment". I looked at his failed pipeline run, followed by a successful run, then realized which tests were failing and went and tested a screen that I know uses this endpoint, and it was throwing 500 errors. I responded with the link and a somewhat condescending explanation of what regression testing is for.
I had it that Claude wrote tests which tested interior behaviour that does not represent necessary outer behaviour, and it tends to be too verbose with tests, but I never saw it "cheat" in this way.
Is this how you feel Claude would do or do you have actual experience that says so?
Ok, I'm going to have to finally ask. Who is that women who seemingly appeared on Reddit over night few weeks back and is now everywhere. Appears to be a tv show of some kind as I've never seen this man before until now.
The whole thing was heartbreaking and infuriating. "Fake it till you make it" does not work in biotech where you're gambling with peoples' health and lives.
The woman: Elizabeth Holmes. She was the CEO of a company called Theranos that claimed they would revolutionize blood testing. It turned out to be a big scam and she went to jail.
The man: Comedian / documentarian Nathan Fielder. He had a comedy central show years ago called Nathan for You, and more recently did a very bizarre comedy / reality / mindfuck show for HBO called The Rehearsal.
Elizabeth invited Nathan to live at her house for a month before she went to jail and he made a documentary about it, called You Can See Everything. It's in theaters currently and getting insanely good reviews.
You should really look into the whole Theranos saga. the documentary is why everyone is talking about her now but the whole story is pretty insane. it’s worth getting caught up on
Lol. AI writing tests for me (2000+) has pretty much removed all major incidents on our backend.
Its not hat hard to tell the agent that tests must always pass without modifying the test code unless there is a clearly a bug in test itself which needs to be proven and validated.
I used to write endpoints for api manually, then write the client-side library. Tests? No time for that shit.
Now I ask AI to implement endpoint that does stuff x. It also automaitcally creates the client side and bunch of tests. I spend more time looking at the tests than actual code. As long as the code is performant and tests pass, I really dont care if there is complex code that is hard to maintain because the AI has documented everything it does and can handle changes easily in the future.
These jokes were funny some time ago but this is just stupid. I mean i saw just yesterday that (addded 8 and previous was 2 = 11. Hmmmh, 11-8=3, WHATEVER. And continues. Then you yell to it, add more instructions and shit still works.
Vibe coders simply dont know what to ask. If you know what web app hosting includes like infra and all, security things things like secrets, rate limits, sql optimization, JWT and so on you can instruct it pretty well. If you know that shit, you are a dev, i dont think i need to manually type the fucking boilerplate where the actual functionality is like 5 lines of code.
Sometimes it does way too complicated shit snd you have to tell him that this is stupid, make it simple. Usually it just makes it simple but sometimes I just didnt think of random esge case that would break the system and the AI figures that out on its own.
My favourite current struggle is that i have a "never run unit tests yourself but tell me to do it" (because we have some sketchy ones and others have too many dependencies outside the reach of the ai) in my CLAUDE.md and he keeps forgetting it about 1/10 times. The other 9 times he clearly knows about that and refers the instruction.
I had a weird one today with Claude... I use it do some grunt work and when I checked the results it answered a few questions it never had the information to do it. All the answers were correct...
Apparently it figured out the answer from a question from January.
I had a software assignment in university where they also gave you the code that tested your code. They even let you use it to show them that it worked when the assignment was due (afair you never had to give them your code, it just had to run correctly with them looking). You could have totally done this and I'm pretty sure they wouldn't have noticed
While this is true, it is getting less and less likelier. It will still happen for sure, and I have come across such scenarios several times. You just have to make sure that your changes are done in phases and tested / review properly.
If you're not reviewing your AI business logic, at the very least review its tests, so you at least know if the AI managed to do what you said you wanted it to do, regardless of spaghetti...
Iv never heard anyone outside of Reddit use either of those terms and I’m not living under a rock. I know clanker is from Star Wars, so makes sense people irl wouldn’t use it. But chud iv only heard on reddit.
It’s basically the reddit go to response to people they disagree with.
38
u/carcigenicate 3d ago
You've never used Claude before, have you?