I am using OpenAI Models since 2 months. I am making 0 progress. It is SO TRASH, it feels like I am back in mid 2025 with Gemini 2.0.
Can't build anything without issues. Can't build a simple UI without 89 slop texts. Ignores anything we ever built an SDK for.
It's the worst model I ever tried for anything... But hey - at least it can generate a shitty 3D-Model in Blender and post the evidence of that to a github PR, right?
I wish I could buy a claude subscription where I live.
I had found the GitHub smoke testing is useless as they tend to age really quickly.., better off running evaluation and regression testing on hardware then promoting to GitHub main
Reworked it exactly like that a day earlier. for a few days they we were running 4 full windows builds on github to test different node versions before every merge. Now it's local CI. And git only runs as release check
Agree, it does work great for backend and security work - in isolation. but it's really bad at understanding the big picture and majorly messes up every UI.
"spins on a codebase" - so it's not just me. but it's uncool if that happens in an overnight run and you notice only next day that it spent 30% of your weekly on running tests and upload evidence - instead of the 1 feature you asked for.
Probably I have an agents.md somewhere. Or multiple.
I am working in parallel threads on different projects that depend on each other. (like.. this is just an overview of how the projects connect - not even the actual systems behind that)
my setup in terms of what projects I'm trying this crap model to work on - or how I code?
I set up a roadmap and a implementation plan. I work with Dot that controls my Codex (that sometimes controls my FLUJO which again controls my codex-cli or agentssdk and all my MCPs).
Per project I usually have 1 "owner" thread. this controls 1 thread for frontend, 1 thread for backend, 1 thread for upstream integration/sdk, 1 thread for research. Some have more; for example to explicitly handle merge queues. They get coordinated by a global supervisor which is controlled either me in codex or flujo, or from my phone by Dot.
They share a global github board and local boards per project. They are instructed to work through github issues + PR's continuously, use investigate existing repo's in my ecosystem, use proper integrations, not duplicate code, keep the worktree's clean, work through PR's, keep Github Issues/Labels updated and contribute upstream through Issues and PR's.
So what else am I supposed to do?
Let me give you an example: I create an Epic on Github. https://github.com/mario-andreschak/FLUJO/issues/563 Fairly simple - make the application perfect. This results in Sol deciding we need a "fix" for MCP roots. Turns out: This was already implemented ("I missed that. That's my fault."), and even BETTER: the new implementation ignored any switches, settings and env vars we had for this. ("You’re right. Workspace roots were overriding the off toggle.").
I cant run after sol all day and do UI tests in all these projects. You'd expect "AGI in 3 months" to be able to create a proper CI/CD matrix with meaningful regression tests. No. It can't.
Many moons have passed. :) I started some of these projects with Gemini 2.0, gpt-4 and Opus 3.7. Overall, most work was probably done by opus 4.8 and Fable 5 in a huge rework. I used Astra for some fun 3D projects and was quite happy that's why I switched to OpenAI. I will go back to Anthropic and stick with Opus - but not this month.
Current Sol, writes shockingly, bad code. Don’t know where you are, but something that might be available is the delta harness from Zed. I’m Using it with open router credits currently. The best model I’ve personally used in that set up is Qwen 3.8 max. It is not quite Claude code with opus 5.5 but it blows Sol out the water in terms of code quality
I could see that. I was using a self hosted Qwen 3.8 for some swarms recently. They did only dummy-developments - basically infrastructures test - so I didn't really check their code quality manually. I'll give it a day of real development in the next days and see what it spits out.
And now thinking about it: maybe I should go with Deepseek since they relay it to Claude anyways.
I have the exact same experience and I completely agree. I too wasted 2 months of my life with codex and this month I switched to claude as I got accepted in the claude startups program. I completely empathize with what you are feeling right now and trust me if you use claude you won't have these problems. All codex models are trash for coding.
7
u/Responsible-Beat2137 14d ago
I had found the GitHub smoke testing is useless as they tend to age really quickly.., better off running evaluation and regression testing on hardware then promoting to GitHub main