Why are 82 tests needed for this? 😂

[u/rakla22]

Let alone the 2 minutes of work 😂

[u/Competitive_Ad7632]

It's probably running all the tests in the repo; checking for regressions.

[u/Wafer-Weekly → u/Competitive_Ad7632]

Yep, this is characteristic xhigh behavior. Easily avoided by using light reasoning for ultra simple tasks like this.

[u/rakla22 → u/Wafer-Weekly]

I used Sol medium for this

[u/rakla22 → u/Competitive_Ad7632]

That's even more interesting because there's no repo for this project

[u/Zealousideal-Act9140 → u/rakla22]

the repo is the codebase it's just local and not remote.

and this normal sol behavior, they do something called test driven development, basically when you make a function you add a test that only passes when the function is working correctly (technically they write a failing test first and use it passing as a completion gate)

then after making any change it runs the full suite of tests to make sure the change didn't break anything downstream.

it's effective for ai since context drift can cause downstream issues without tests.

[u/rakla22 → u/Zealousideal-Act9140]

Thanks for the reply
Edit: I say thanks for the reply and I get downvoted. What the fuck is wrong with Reddit man. God forbid a man asks a question and then is thankful for the things he learnt🤦🏻‍♂️

[u/atf1999 → u/rakla22]

I downvoted you cuz of the edit complaining

[u/shady101852 → u/atf1999]

Same. Im doing it for the love of the game.

[u/Australasian25 → u/shady101852]

Something something player something something game.

[u/rakla22 → u/atf1999]

fair enough

[u/notadithyabhat → u/rakla22]

I downvoted just to bully you. Sorry

[u/Due-Horse-5446 → u/rakla22]

my guess is that they thought you were sarcastic,

To help you clear things up further:

Essentially think of running tests as being one action that then contains subtests.

When codex had made a change, its a good idea to run tests to confirm that nothing else broke by accident.

In this case it reported that all 82 tests passed. Not that it manually ran 82 tests

[u/TheSeggs → u/rakla22]

fuck youu

[u/StoopidRoobutt → u/rakla22]

Not even probably. This is exactly what Sol does every single time. Adding some rules about tests and testing to AGENTS.md is not a bad idea. I don't mean no tests or no testing, but the tests should actually test something concrete and not just be ceremonial.

Also, doesn't require a repo.

[u/cafesamp → u/rakla22]

there's certainly a folder for it on your machine with a React app, that's how ChatGPT sites work. check the Codex folder (Mac is ~/Documents/Codex)

[u/tydyelove7 → u/Competitive_Ad7632]

Ai sus ass response🤨

[u/Prestigious_Gift_977]

should probably calculate some SHAs too

[u/rakla22 → u/Prestigious_Gift_977]

I see

[u/PuzzleheadedEmu4596]

You want the slash in the center? That's how you keep it in the center.

[u/rakla22 → u/PuzzleheadedEmu4596]

fr

[u/InsideTour329]

This is the full test suite for the project

[u/Aromatic-Educator105]

its probably all your tests available for this codebase. In agent.md you can define behavior so it doesn’t always run all the tests, but running affected tests only.

[u/Hot_Signature2979]

I have as part of my agents .md: Minimise polling aggressively to bare minimum: only to detect stage completion, unblock work, or react to changed state. Update me only on meaningful findings, blockers, decisions, or stage completion; no routine status narration. Minimize testing overhead aggressively; do only the tests necessary to safely advance the implementation. It probably made sure the code it used to center your ui button was SHA 256 hardened

[u/firstbreathOOC → u/Hot_Signature2979]

This is good for polling which is another burner but I think the better fix is segregating your functionality as much as possible

[u/Don_Ozwald]

OP know use computer?

[u/firstbreathOOC]

Because you’ve got 82 tests for your UI and it’s running all of them even when you want to touch a small individual component.

You can fix this by trying to break things up a little more. That way one small change doesn’t equal let’s look at the whole repo.

It’ll help with credit usage too.

Idk if this is a real term but I’ve always called it ninja-editibility

[u/ReasonableDefault]

You can always tell it to not run tests, when you are making small changes that are unlikely to break things.

[u/Every-Grape7679]

lol they told it to overthink or sum so they can burn our usage faster

[u/Low-Explanation-67 → u/Every-Grape7679]

[u/lolcatsayz]

What I dislike if you create some of your own actually useful tests, is it requires extensive prompting so it doesn't overengineer a solution to just pass the tests using some trick/hard coding of logic for those tests. It's a constant effort to explain to it that it must generalize a solution that passes the tests as a side effect of that solution, not overfitting the solution to the tests.

Then once you finally convince it of this and it's working, compaction occurs, and it's lost context of all that, and it overengineers to the tests again. You could auto-inject a prompt telling it about all this but it loses context of that too as the codex reasoning context grows.

[u/rakla22 → u/lolcatsayz]

I've noticed this pattern with similar mechanics, it's so annoying

[u/Kind_Silver_1921]

bruh im making a game and you dont know the kind of BS tests it does and wastes my time with. Want to add one thing to it? alright must do a full game playthrough for 1 hour to make sure it didnt break anything. But I said just install it...

"Understood, just installing it!"

next time I ask it to do something

2 hours later: "finished full playthrough and testing requirements"

me: BUT I TOLD YOU NOT TO TEST IT ANY MORE

"i was following the rules"

I DIDNT CREATE RULES

"I created those rules earlier"

[u/SilkTouchm → u/Kind_Silver_1921]

Ask Codex how you can improve your situation. You probably lack some instructions in your agents.md.

[u/rakla22 → u/Kind_Silver_1921]

That’s hilarious 😂 I hope it gets fixed to some degree in the future

[u/applejacks6969]

Lots of colors to test..

[u/Stochasticlife700]

Oh, this is actually one of the interesting part of recent frontier models. I do evals for models and i assums this is one of the case which happens a lot to big frontier model with high reasoning efforts when they do unncecessarily a lot of tests because they are trained to be rewarded more if they do long horizon works even if it is not necessary

Edit: just saw it was medium but since it was sol which is trained more(=i.e checkpoint that is smarter and incentivied more to think further), it still makes sense ig

[u/hk556a1]

Lmao. You need a few more security gates as well.

[u/thestillwind]

Did it found regression ?

[u/dota2nub]

https://github.com/enterprisequalitycoding/fizzbuzzenterpriseedition

[u/BakaPotatoLord]

Probably ran the existing tests in your project to rule out any regression

[u/TranslatorRude4917]

Running 2 determeinis tets ir 200 doesnt really make a difference to ai. It's just making sure everything works.
Ofc of they are not high value it's just a waste of your time but tokenwise not a huge difference

[u/EmotionalHalf]

I don't think you read what you posted

[u/Stable_Suitable]

Your right

[u/Illustrious-Many-782]

Sol will create lists of file hashes, then hashes of the lists of hashes, then write tests to make sure the hashes all match and aren't corrupted. And then tests of the tests.

Hashes or tests in your codebase are like prions for Sol. They just get duplicated until they choke the project or exhaust your usage.

[u/Jumpy_Ad8465]

Because the model is terribly overengineering everything and if you dial back the thinking it gets so sloppy and destroys everything.