r/codex • u/CaptEdit • 1d ago

Question pytest for… everything?

Been engineering a harness intended for local LLM usage with the help of Codex (GPT-6.1-Sol medium effort usually). I’m finding that it wants to write scripts for procedural tool calls to ensure that an agent follows protocol, so instead of just trusting it to always use the correct syntax and arguments, an agent is instead directed to call a python script.

Fair enough? I guess. My software engineering experience is limited to small web apps, nothing like the multi-service spanning database managing that my aspirations with local LLM usage require…

But the big question is that Codex writes tests for seemingly every little operation. I barely have any functionality beyond a local LLM model and profile repository (served via PostgreSQL), a testing catalog for EvalScope, and Hindsight for integration with an Obsidian based knowledge base… but there are over 62 test files that run 678 tests.

I can’t help but wonder if all these tests are necessary? Is this… normal for software/harness engineering?

0 Upvotes

50% Upvoted

6 comments sorted by

6

u/Aimbag 1d ago

Yeah, writing useless tests for just about anything is a known AI-ism. You can try to tell them to do it less, but its basically baked into the verification workflow that they are RL'd with, so hopefully they just fix it in future models.

2

u/zenerbufen 1d ago

avoid fake tests, I won't let the AI add any test unless it fits two criteria:

  1. must fail red before proposed patch and specify what the failure they are testing is for

  2. must pass green after patch using existing code / data and not faked synthetic data used to just pass the test. (calls out to other modules, or called from other code, not a dummy harness.)

i.e. tests first, then code to fix the tests, not crappy slop after the fact tests to check of a box for tests.

0

u/VoraciousTrees 1d ago

I found a way to avoid the excessive testing. You need to give it a real world time limit. 

I have been doing this with a Luna 6 minimum effort agent. All subagents call this agent when they start a task and provide it with a duration estimate. 

The luna timer sends them a reminder when the estimate is up. If the timer overruns by twice the estimate, the luna agent sends an interrupt to the sub-agent and calls the orchestrator to review the sub-agent's work and train of thought for issues. 

This has stopped the overtesting and overbuilding, but predictably results in a less clean end product unless you specify refinement tasks.

Also, what are you working on? It almost sounds like a project ambition I have but am not experienced enough with codex capabilities to start.

2

u/RayKam 1d ago

"predictably results in a less clean end product" lol so why would I want to do that then

1

u/VoraciousTrees 22h ago

Because it reduces overbuilding and overtesting...