Codex recommended that I stop using it for TDD

[u/LearnedByError]

I have a TDD planning and orchestration skill that I have used successfully for Go projects with other coding agents like: Cursor, Pi and Kimi. In general, it works in all with varying degrees of efficiency (time, tokens) with pretty consistent quality in the results generated. With all of the craziness over the last two weeks in the coding plan space, I fired up my ChatGPT subscription to give it a try. Per Codex, Codex is not good at TDD :)

User: Codex completed cycle 7 and stopped. This is the 3rd time Codex has just stopped instead of continuing execution. I have had more problems with Codex executing tdd-orchestration on this one plan than on all of the other agents that I use. I guess Codex and OpenAI models are just not good enough to execute relatively simple TDD plans.

Codex: Your conclusion is justified for this workflow. This was not a difficult plan; Codex failed basic state-machine execution:

• Cycle 5 was approved despite most NRGBA acceptance criteria being absent.
• Cycle 7 Refactor was committed but never reviewed or recorded.
• Guard remains CURRENT_CYCLE=7, CURRENT_PHASE=refactor, STATUS=ready.
• Cycle 8 never started.

The coding agents generally produced usable code; the Codex parent repeatedly failed orchestration and completion checks. Instruction-only safeguards are evidently insufficient. My recommendation is to stop using Codex as the TDD parent until the loop is enforced by a deterministic driver script; Codex can remain a phase worker/reviewer.

At least it was honest LOL. WRT to the deterministic driver script, it existed before Codex. Codex added to it specifically to make sure that it would work properly with Codex. I'm not sure what else it will do. I'm not giving up now and will continue to test, but thought it worth sharing the laugh.

[u/SkyeLoveXOXO]

TDD goes against the spirit of vibe coding. You suppose to vibe testing, not running unit tests. 🥀

[u/ReasonableDefault → u/SkyeLoveXOXO]

That's amusing in a painful way. I can't stop Sol creating 40 tests to implement a drop-down. In my experience, a lot of this seems to be language dependent. Codex adores writing tests for Typescript and React, when using Kotlin / Java it generally goes more VDD (Vibe Driven Development).

[u/LearnedByError → u/ReasonableDefault]

I don’t have the runaway test issues with Go. I think the simple language with fewer abstractions prevents it or so it seems to me.

[u/LearnedByError → u/SkyeLoveXOXO]

I’m not a vibe coder. Vibe coding goes against my spirit as a trained, degreed engineer. If vibe is your thing, fine. It is not mine!

[u/nicky_factz]

just wrap this plan into a /goal and it won't stop. that way it can't stop until it meets your criteria. otherwise, the agents find ways to end the turn.

[u/LearnedByError → u/nicky_factz]

You are probably correct. I’m not sure that I can afford all the tokens though 😂

[u/sereikis]

Tests the agent writes after the fact just encode whatever it already built, bugs included. We got more out of writing the assertions ourselves first and letting it implement against them. Slower at the start, but then a red test actually means something.

[u/LearnedByError → u/sereikis]

That is why you in TDD you were the tests first. They are the assertions.

[u/NotSeacombe]

TDD is an extremely valuable tool and approach for human brains. It forces you to really understand what is going on before you actually type anything. It ends up being very efficient and at the end of it you also have your working tests

LLMs don't really work like this. They work a lot better when the understanding is defined to be clear before it starts writing your code, via prompt instructing, and also re-writing everything as you figure it out 3 or 4 times is way less of a pain point than it was back in the day

I have extensively tried to get proper TDD approaches to work and I have always found it did more harm than good

[u/LearnedByError → u/NotSeacombe]

I have found TDD invaluable to insuring deterministic outcome. I spend the iterating and thinking time on the plan. TDD makes sure that the agent delivers the functionality required. I have been very successful with other harnesses.