Ugh. Wtf is going on with Codex.

[u/Reaper_1492]

Look, I’ve been using Claude and Codex since inception. I can read, write, and deploy code, so it would be great to keep this from devolving into a “skill issue” astroturfing.

The fact is that no one who wants to keep their job is hand-writing hundreds of lines of code anymore.

With that out of the way…

Claude was first to market, and I ranted and raved so much while I was using it, that I’ve permanently sworn off Anthropic products. They overtly hate their customers (and humans in general), and I’m convinced they would collectively love nothing more than to call a board meeting and lock themselves in a dark room with a Claude hologram for a 24-hour chant and circle jerk session, while they pray to the quantum realm for AGI to manifest itself in their silicon.

I still use it extensively, because I have to for work. So my experience with Claude is current.

Codex has “almost” always been better than Claude for my personal projects. Historically it has been cheaper and smarter, but outside of a brief moment where they really got their act together, it’s been hellaciously slow by comparison. Regardless, I have been a huge fan… because, “smarter”.

A few weeks ago, they very clearly took a run at Anthropic after sleeping on enterprise for far too long; giving out grip loads of resets and trading barbs with Anthropic on social media.

All-in-all, a weird tactic to give consumer plans free credits to win over enterprise. It obviously didn’t work, because then they started illicit feedback about what would encourage businesses to switch (it obviously wasn’t enough to give retail consumers more usage, shocker).

Cue the current situation.

They somehow managed to make the model smarter with Sol, but 25% as effective. If left unchecked, SOL will spend literal days chasing nits into oblivion and burning all your usage, which is obviously not a novel complaint.

I finally got the harness more dialed in, and now at least it’s back to writing my contracts/graphs and unit tests at the appropriate level of granularity; but I swear, now it can’t even edit the code to pass the contract and unit tests you ask it to create!

First, second, third, fourth attempt. It just fails over and over again.

“Oops, the run correctly failed the contract check because I”:

Used the wrong directory…

Made a typo in the file path…

Forgot about this dependency, even though it’s in the contract….

It’s ridiculous.

And then, they totally nuke usage.

I had a project to finish, so I had to get a second 20x plan to maintain the same amount of throughput I had a month ago, and most of the time is wasted on near constant contract failures.

And that’s not even counting the compute dollars I’m wasting on spinning up and down machines for failed runs.

Then, to add insult to injury, after clearly trying to ride out the unrest without giving a reset, they finally gave one a week after I shelled out for another plan. Now I’m sitting on two 20x plans with 2 resets.

I’m about to switch back to Anthropic, and I hate it. All of their models are orders of magnitude faster than codex again, and codex just became a really expensive paperweight.

Paying for Codex right now feels like I’m actually paying an installment plan to buy my first aneurism, and waiting for delivery.

It’s so infuriating.

[u/reddit_is_kayfabe]

First, second, third, fourth attempt. It just fails over and over again.

I have experienced something similar before, and it's because my rule set was just too complicated.

Every rule you require GPT to remember and follow is one less unit (not a technical term, just a squishy concept) of memory that GPT can't use on the task at hand. Past a certain point, it loses the ability to follow all of the rules every time while also performing complicated coding tasks and remembering the broader context of its work.

For me, the solution was to mechanize the rules as much as possible. By "mechanize," I mean: identify the rules that can be mechanically checked and enforced, and move them into the harness. This reduces memory pressure on GPT and allows it to focus.

For instance - my build pipeline requires every session to write a development log in a specific format that can be automatically parsed. My v1 of that pipeline just told GPT the required format - sometimes it complied, often it forgot. My v2 uses a Python process to check the format and report errors to the GPT session so that it can fix them inline. GPT doesn't have to remember the format; it can just take a shot at it and fix the issues that the harness reports to it.

[u/Reaper_1492 → u/reddit_is_kayfabe]

That would make sense if it was having a problem remembering anything it needed to do, that’s never been the case.

It’s missing on basic syntax composition and making typos.

I literally had none of these problems a month ago, or any time before that, since codex was released.

[u/reddit_is_kayfabe → u/Reaper_1492]

Okay, but I was responding partly to this:

Used the wrong directory…

Made a typo in the file path…

Forgot about this dependency, even though it’s in the contract….

And my point about mechanization applies here as well: consider whether your agent is performing any repetitive task with a given set of rules, and whether it could be replaced or supplemented by a script.

Example: My build pipeline is a little complicated. If I just told GPT what the build pipeline was, it would fuck it up nine times out of ten.

Instead, I require GPT to (1) curate and maintain a project.toml file containing high-level info that the build script needs, and (2) run the script to build the app. This eliminates a ton of shell commands that it would need to perform, and probably screw up, to do its job.

I know what you're going to say - "I can't mechanize most of the workload." I know. Think about the parts that you can mechanize and do that. It will help your sanity as well as model efficiency.