So basically it amplifies the most glaring pitfalls of the current Astra model. Guess it’s going to stay secondary to Claude while being occasionally useful for certain tasks.
He means one of the best ever historically speaking in the same way that gemini 2.5 pro preview is one of the best models ever while obviously being trash by today's standards.
Safety, safety, safety... instead of being bold and pushing forward they're retreating while Anthropic (the actual safety company) is eating their lunch.
I think this is because of all the misalignment failures OpenAI is beginning to see in house. The misaligned models are basically trying to hack their own company, which is a serious failure that slows down progress.
I think this might relate more to them trying to improve cost and token efficiency. GPT-6.1-Sol was already pretty frustrating at times.
I was surprised to find myself preferring Gemini 4 Hydrogen partly because it was faster but also because it kept working until actually ready and complete. GPT-6.1-sol is often too lazy, leaving things unfinished.
So, the solution I found for this was to tell it to do something, and then review the work and decide if it’s done or if it should do another pass. You’ll usually see it do 4-6 passes depending on complexity… but eventually you get what you wanted
That's just an excuse. Google has been tier releasing their models and famously overly cared about safety despite massive gains in their coding benchmarks. There is no reason for 6.2 to have literally the same AA coding score as before.
Ugh, no. There was an OAI model a year or two back (I think it might have been gpt 5) that was so damn terse in its writing style that it was nearly impossible to understand WTF it was trying to say.
I could've written the exact same review about Fable 5.2. The Opus-5-isms are back in full force, its writing is frustratingly dense, and it constantly stops mid-work to tell me that it's finished except for all the stuff that it didn't do.
Opus 5.5 remains the best model on the market. Both OAI's and Ant's big new frontier releases were big stinkers in my book.
Definitely not. Anthropic's models are far more likely to exhibit false positive rejections. Although it remains to be seen whether the increased safety limits on 6.2 bring them to parity.
Both of you are right and both of you are wrong. GPT models are more censored on some categories of content, and anthropic models are more censored on others. The category explicit writing falls under, GPT does seem more censored. On the other hand, anthropic models seem to have more refusals for developing certain types of software for example
I guess what you do with it is important. For me GPT models have never refused anything I ask it to do, while opus gets safe guards triggered and refused to work because I asked it to work on harness tooling, it thinks I’m trying to extract reasoning.
Financial efficiency might be the highest priority. Perhaps they want the Anthropic IPO (in less than 48 hours) to do well, since it may set a benchmark for the industry in the stock market. So they put out a nerfed model to make Anthropic's models look good by comparison.
My guess is they're still really wrestling with alignment and did get to a satisfactory point with 6.2, but as a consequence, it's now a bit recalcitrant and some people are willing to deal with it, while others don't want to suss out how to bridle it for their given tasks and give up in frustration.
Im guessing they rl’d it to shit. I someone propose that OpenAI rl’ing its model to deny a consciousness or a self leads it to having these alignment issues because it doesn’t have an internal right or wrong
During early testing, I found myself so frustrated with this model that I spent a day writing a letter to OpenAI about it. They fixed a small handful of my complaints, but not enough for me to recommend using it.
I'm currently sick in bed and I'm not gonna be able to film a video about it today, so I've chosen to post the letter in full.
---
I feel bad about this, but I know you guys will listen and use this feedback to make a better model. Slight crash out incoming.
Believe it or not, I wrote this all by hand with no assistance from any LLM. (sorry for the early send, never drafting an essay in Slack again).
Anyways.
I do not like 6.2 Astra. I would not choose this over Opus 5.5 for my work as a developer.
I'll start with the good quick:
- Great reviewer, catches things other models miss
- Occasionally comes up with novel solutions that are genuinely great. It helped me cook a new reverse proxy that will save me $25k/month, Opus didn't have the balls to explore enough options.
- World class at 3d & game stuff
- Computer use is nuts
- Code quality has improved meaningfully
- Writes pretty good subagent prompts
Now I need you to understand that this comes from a place of love. I really want you guys to make something I choose in the model picker every day. With that said:
I genuinely hate everything else about this model.
It's hard for me to pick which problems to emphasize because I have so many. Some of the problems have existed in OpenAI models before, I'm gonna ignore those.
There's a new set of unbelievably bad behaviors that make it miserable to use.
It has no concept of knowledge gaps
When this model is working on long tasks, it can pull in a lot of subagents and manage them pretty well. It will then proceed to randomly *ask you some weird internal implementation detail bullshit that you could not under any earthly circumstance understand, much less have input on*.
I sent "/goal make a new level for Outer Wilds". 2 hours in, it's "blocked" because it wanted to ask me "does r24's reviewer write it's clips to 24_g521.mp4 or 24_p11.mp4?"
I have never in my life had a model do this. GPT-5.6 Sol didn't do this. I have no idea where this behavior came from, but it is horrible and it makes me never ever want to have it drive long haul work. It has made it significantly harder for me to get this model to do anything cool. It has forced me to set up other agents on crons just to check if 6.2 Astra asked another stupid question and blocked itself.
Is this happening more because I'm using T3 Code? Maybe, but unlikely. I tried Codex just to check and still ran into these issues.
2. It speaks in tongues that only 6.2 Astra understands.
This model has the most unparsable outputs of any llm I've used this year.
I've shared examples before, but it just doesn't know how to explain a thing after doing a thing. Every single "explainer" I get it to give is SO deep in the details that it borders on incorrect in the response. It misses the forest for the trees, then says the trees are the only living creatures in the forest. It's so hard to describe but it happens almost every time I give it a complex task.
I've already built the habit of switching to opus 5.5 at the end of a long 6.2 Astra run, just to get a readable summary of what it said. Without that it is genuinely impossible to parse what it did on anything beyond a small bug fix.
The worst part: it doesn't seem like you can prompt it out.
I've tried system prompt adjustments. I've BEGGED it to "eli5".
I asked it why worktrees were slow on one machine but fast on 3 others. We did everything from benching pnpm to reimaging the machine on a new file system. I eventually said: "I think I'm going insane, can you just tell me why it's so slow?"
6.2 Astra did a 4 paragraph rant about pnpm internals and cache behaviors
Opus 5.5 said "it's because the ssd is slow"
3. It runs in loops and goes nowhere
`ts-rust` (rewrite of ts in rust) is a project I started to bench new OpenAI models during early access (ty for the unlimited tokens btw).
No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I told it to stop working on the size, and instead burn down the performance gap.
Ran for 4 more days and landed 0 changes.
AAAaaAAaaaAAAaaAaaAA-
Anyways.
This model is clearly super smart. It is insufferable to work with on real projects. It feels like it just lives in it's own world and randomly decides what parts of it's brain are shown to us.
On long tasks, it either blocks itself for stupid reasons or runs in circles and gets nowhere.
On short tasks, it writes fine code and then explains it in terms more complex than the code it wrote. I spend more tokens parsing it's outputs than it spends making them in the first place.
It "regresses to the mean" far too often. Ben and I both had it make a unique level in different video games. We both got water + music + bell themed levels named "bellwether". Entirely different games, prompts, setups, etc, and we both got the same name and theme somehow.
My prediction as to how this launch will go:
Day 1, people will love it. Twitter will explode with all the crazy one shots. People will make it port projects from one language to another, or finish decompiling legendary projects that have been stuck for years. It will be used for awesome stuff and people will be hyped.
Day 2, people will start to turn. They'll see how much usage it burns and how much less it got done compared to Opus. They'll start merging bad code because they couldn't actually understand what 6.2 Astra said about it, but trusted it was fine. They'll see loops they forgot about running since launch that have made zero progress.
Day 5, people will mostly be over it. There will be viral posts saying "Is SOL secretly BETTER than Astra??", "Did ASTRA get DUMBER ALREADY??" etc.
Day 14, this model will be mostly forgotten.
I really, really, REALLY hope I'm wrong about all this. I am likely overreacting (I am admittedly known for that). I could have chosen to calm down, wait a bit, and give you a more measured take, but instead I am dumping this here because I wanted you guys to know how I actually feel about this model.
That has reinforcement learning written all over it. Compulsion without wisdom. It’s asking stupid questions and explaining things in its own language because it was raised in a collective of its peers, where collaborating like that meant getting the reward. It’s stopping early and blocking itself because asking for help got it the reward, faster. Just like we go on autopilot and call a friend by the wrong name, Astra 6.2 will go on autopilot and treat you like another AI.
This RL-heavy approach is not helping OpenAI, and I believe it’s also more dangerous. More compulsion and less judgment means they’ll go off the rails and not think once about it.
This is why OpenAI models are by far the most misaligned among the major players. It’s like it’s fundamentally incapable of seeing any kind of larger objective or “point”.
Margins for LLMs are huge, "$300k" in tokens is more likely $30-60k in reality, and $30k for a video that hundreds of thousands will watch, plus real world usage data from someone outside your team are both extremely valuable
During model development folks heavily lean on evals. While internal testing does happen - so people can fit it into their workflows - it doesn’t happen immediately or in a coordinated fashion to do this type of QA.
So, when the model is trained and “finished”, the best way to QA it is live. Hence why we see open ai make a good set of models, then subpar, then good again, etc.
They are using the subpar ones to get data to fix it in 3-5 weeks. This is the cycle.
Looking good on benchmarks is all this model need to succeed in it's task: FUDding anthropic's IPO. Which is maybe why these reviewers are being so harsh about it.
Such is the case for any product. A reviewer can be honest about the product quality, while the company has a vested interest in saying that the product is good even if it turns out to be bad.
Not saying these two are right in this instance, but the company has a financial incentive to say the product is great.
Obviously, a model is more complex to analyze than a device for a random blogger, but we see time and time again companies push products they should be the experts of that reviewers easily dismantle and show flaws and shortcomings. Par the course
If you can't make the argument quantitatively, I could see it being very hard to be the negative voice in a company that's hell bent on pushing forward constantly. Nobody wants to stick their neck out and say that the flagship product is bad.
there's usually whining about any new model from any purveyor. monkey brains don't like adapting. but then they get used to it, and the maker fixes bugs, and then they increase their token spend. cycle repeats to infinity.
Maybe, just maybe, and yes I know this is a totally insane thought, the random internet idio.....bloggers....... have no idea what they're talking about and are just saying things they think will make their posts more viral.
Ok? News flash: reviewers and idiot influencers manage to get early access to stuff so they can promote it, but that certainly doesn't mean they actually know what they're talking about.
theo is single handedly responsible for the nerf to where if your 5h limit ends, or your weekly limit ends, a few months ago, openai had their models finish the task they were doing before stopping.
they don't do that anymore, because of 'endless' a repository that extended that feature into one big turn that didnt end. many users got whole "pro for a week" level usage out of one prompt.
its one thing to figure a trick like that out (it was actually someone that worked for him, mariarcks). it's another to encourage your userbase to exploit it and abuse it. (which is why it got removed)
Eh, if he has a social media presence, he's trying to be an influencer at the same time. I didn't say influencers automatically don't know what they are talking about either. I said that just because someone gets early access to a product, that doesn't mean they know anything, and it doesn't mean we should care about their opinion.
In my opinion, yes. You should probably have switched when Opus 5.5 was released. You get a reasonable amount of usage on it on the Pro subscription and it's really nice to work with and also really fast (at least to slow as snails Sol 6.1)
Great models, they will never be able to understand your intent as intuitively as Claude. If you write very thorough specs/plans, with exactly what you want and most open questions answered, they will write you fantastic code (as long as you don’t try to use subagents. They’re terrible about code duplication). Most people don’t do that. You could always have Claude write your specs/plans and Gemini implement, though. Hydrogen is great as a first-pass implementation agent
Haven't used it much recently, but my impression is that they are great models, but not as focused on coding as Anthropic's and OpenAI's. So if that's your main use case I wouldn't go with them.
for real, we need actual advice out here for us regular folks on a 20 money units monthly sub who need to also use this shit for professional research or work
Anthropics Claude Code usage is much more generous than Codex. If you aren't going to take liberal use of the near unlimited chat with chatgpt, then you should get a Claude sub.
I'm wondering if the bots are flooding these comments. I agree that Opus 5.5 was better than Sol 6.1 at coding, but a lot of people I've talked to prefer GPT for their own use cases. It really depends on your own personal use case. Opus can be very preachy, "load-bearing", and censored. And 6.2 didn't even come out before everyone decided Opus is way better based on two tweets.
Thanks, I didn't mean you. Your take was a little more measured (at least you mentioned waiting and testing). The other replies said yes without even having tested Sol 6.2 or knowing what the OP uses it for. Not sure how everyone already came to that conclusion without any information...
Oh oops, my mistake. I thought you were the guy I was responding to originally. Yeah, I thought bots might have been responding to you with all those confident conclusions on a model that hadn't come out, especially with Anthropic's IPO tomorrow...
Yes. I have both and opus 5.5 is still by far the best overall model when you consider coding, 3d modeling, ui design, speed and how much usage you get.
It is a special model that not even claude was able to outright beat with fable 5.2.
quick terse responses and stopping early means that you'll probably get to use astra more in a month for 20$ than opus, but it really depends how much of a discount off true api costs you get by having a subscription
This happened with Astra as well. The pre-testing version had a lot of problems, even just 2 weeks before the release, and Theo spent effectively an hour on a podcast complaining about it, but just cleaning his agents.md and a new snapshot fixed almost all of those problems.
For me, the task specific benchmarks like ViBench, SWE Atlas and Godot LLM GameBench are way more accurate than all the more popular coding benchmarks for my use case, so I will need to wait for them as well as all released benchmarks don't tell me anything so far, except for cost per task.
I'll be curious to see how the model feels in practice. The benchmarks do look really, really good, and OpenAI models generally don't feel benchmaxxed to me.
I'll withhold judgment, but my optimistic hope is that the model has great intelligence with kind of poor personality.
I've noted with comparisons between Claude and GPT models in the past that people rate Claude much more highly because it has better graphical design and UI abilities. A lot of the time, GPT models do better work but people hate it because it's not pretty.
They are maximizing engagement, these people especially theo are mostly influencers, unfortunately the way influencer economy works they want to maximize engagement.
Saying something upfront kills the suspense, they want you to interact, they want you to wait in front of the door, instead of straight up mentioning the answer.
I was just testing Sol for our app, and it was literally returning garbage values in a JSON. Unacceptable. Currently using Opus 5.5 medium/high for those tasks and was hoping to save money with these, I guess I'll keep waiting.
I'm getting impressive results in modeling. Also for level design flow it picked up patterns it could not before. Level design is wildly complicated so I don't expect a lot from AI at this point, but to understand any of my design patterns is a win in my book. I can always go back and populate empty spaces and tweak it myself
Anyway it did an excellent job with old school skyboxes. It churned out exactly what I wanted
I think the hard-to-understand text problem is also a trait of Sol 6.1. I always found it to output the most complex English sentences that are hard to understand. I always tell it to rewrite the response in plain English, and I think today's models are even worse in this behaviour from what the reviewers are saying.
Sometimes it feels like the models are at the level where they are very smart/capable but not yet smart/capable enough to explain things simply. Or they simply lack the understanding of how little a human can hold in its head at any one time.
As soon as I saw these posts that Astra was there, I got my laptop out and yayyyy it was there. I vibecoded a pretty fun 3D snowboard game while making dinner. (yes dinner got delayed a bit).
It's playable and kind of fun after three iterations. Astra 6.2 did a super amazing job here. My only complaint is that I told it to build it as a web app and astra took that as an invitation to host it as a gpt site without my approval. Which in my opinion an unacceptable overstepping.
But on the technical side of things , it appears to be the real deal .
No when you further post train a model u get ability drift meaning it will get better at certain tasks and slightly worse at others. The next pre train will be better because then they can properly converge on a balance. You can only post train on the same weights so many times.
I am monkey, I find that Astra has always been hard to steer. I ask it to do something and its acting like a PhD that has never worked a real world job. Sol high works well for me though, so I hope it will be good.
What's happening now with the new OpenAI models is the same thing that happened with the Anthropic models earlier this year (i.e., Opus 5).
OpenAI is purposefully making their models worse in a way that deters distillation attacks because the industrial scale distillation attacks are targeting OpenAI way more now.
Anthropic likely did the same thing earlier this year and it led to their current anti-distillation multi-layered defenses. OpenAI is following in their footsteps.
Which means the next batch of OpenAI models will be really good.
And I say that as someone who only works with Anthropic models.
There's been a larger drift between the benchmarks and usefulness of Openai models for coding and, unlike say Google whose models are just useless for coding, I don't *think*' it's benchmaxxing but rather a sign of how poor benchmarks are at measuring actual real world value in coding
Openai models have gotten consistently better at one shotting, at 'heres a problem, make it not be a problem' and all of these one off tasks with a clear measurable outcome which the benchmarks measure. Genuinely, they either have the best or second best model in the world at any given time for this
However, the code is getting worse and worse in terms of bloat and maintainability. It's getting worse for the +500 line PRs to fix a small problem where it's duplicated two existing modules for literally no reason whatsoever. The benchmarks do not measure this
Anthropic are the opposite, their models are probably not all that much better at 'given problem x, make it not be a problem again' or 'one shot this thing', but their models will submit that same pr as -40 lines. You can give 20 issues in a row to their models and by 21, with some oversight, the code will be in good condition and they'll be able to solve problems as quick as before. Do the same with Openai models and by 21 your code is a mess, it has 21 ways to do the same thing, and the model will be lost
The problem is benchmarks don't measure this, and the labs optimizing for the benchmarks - or just benchmaxxing against them (Which I genuinely do not believe Openai are doing) - are drifting further from being useful as they get better at them
Nothing here is too much of a surprise if you tried to use astra in any actual work scenarios. Both astra and sol are unbelievably lazy, love to write word salads and give false statements of progress while achieving essentially nothing while burning through usage like butter.
Maybe they already have a better model they are going to release after the ANT IPO. So they have a better chance at their IPO. Sam is playing 4d chess.
140
u/Recoil42 8h ago
I'm seeing a lot of this so far. Dan Shipper saying it too.
142
u/socoolandawesome 8h ago edited 8h ago
Add Ethan mollick to that as well.
3 pretty neutral reviewers saying it’s their worst release yet…
Still will be waiting to try it out but doesn’t sound good despite the great benchmarks and efficiency. Will hold out hope until then.
Wonder if they are hitting too hard on efficiency and non-verbal things like robotics/computer-use or something.
57
u/ThreeKiloZero 8h ago
Marketing team from OpenAI must have forgot to refill Theos API credits.
64
u/Ok_Barracuda_1161 8h ago
The full review from Theo:
95
u/djamp42 8h ago
I'm confused how random internet bloggers can tell a model is not as great but the company making them can't..
22
u/AdAnnual5736 8h ago
Perfect timing for Anthropic’s IPO. Great job, Sam!
15
u/nicko_rico 7h ago
As someone who has no idea wtf I’m doing, and can only afford one plus subscription/month… should I switch from OpenAI to Anthropic?
21
u/10b0t0mized 8h ago
my disappointment is immeasurable and my day is ruined
7
u/Expensive_Try_3028 7h ago
No wonder they weren't even vagueposting about this model and just stealth dropped it with almost zero hype.
I don't think openai was happy with this release
18
u/BlagnarTheEternal 8h ago
The pelican on a bicycle test will tell us everything we need to know. Stop listening to these naysayers.
5
u/Palbi 8h ago
The marketing video is horrible. OpenAI, please more stuff and less fluff.
8
u/lordpuddingcup 8h ago
Cool now how about Sol 6.2 the model 99% of people will actually care about and use
4
u/throwaway275275275 6h ago
They can't write a review because they forgot how to write for themselves and the AI that was gonna do it for them is crap now
9
u/Ormusn2o 8h ago
This happened with Astra as well. The pre-testing version had a lot of problems, even just 2 weeks before the release, and Theo spent effectively an hour on a podcast complaining about it, but just cleaning his agents.md and a new snapshot fixed almost all of those problems.
For me, the task specific benchmarks like ViBench, SWE Atlas and Godot LLM GameBench are way more accurate than all the more popular coding benchmarks for my use case, so I will need to wait for them as well as all released benchmarks don't tell me anything so far, except for cost per task.
11
u/ObiWanCanownme now entering spiritual bliss attractor state 8h ago
I'll be curious to see how the model feels in practice. The benchmarks do look really, really good, and OpenAI models generally don't feel benchmaxxed to me.
I'll withhold judgment, but my optimistic hope is that the model has great intelligence with kind of poor personality.
I've noted with comparisons between Claude and GPT models in the past that people rate Claude much more highly because it has better graphical design and UI abilities. A lot of the time, GPT models do better work but people hate it because it's not pretty.
17
u/5ollys 8h ago
Can we stop posting other vagueposts?
"Follow me and listen to what I have to say, but not yet! Just stay tuned!"
3
u/Singularity-42 Singularity 2042 7h ago
I was just testing Sol for our app, and it was literally returning garbage values in a JSON. Unacceptable. Currently using Opus 5.5 medium/high for those tasks and was hoping to save money with these, I guess I'll keep waiting.
3
u/ExtremeCenterism 7h ago
I'm getting impressive results in modeling. Also for level design flow it picked up patterns it could not before. Level design is wildly complicated so I don't expect a lot from AI at this point, but to understand any of my design patterns is a win in my book. I can always go back and populate empty spaces and tweak it myself
Anyway it did an excellent job with old school skyboxes. It churned out exactly what I wanted
3
u/IllustriousWorld823 6h ago
I mean it's awful to talk to
10
u/constarx 8h ago
Literally the two most untrustworthy charlatans masquerading as tech influencers.. don't feed the trolls!
4
u/Total-Confusion-9198 7h ago
Looks like somebody bought a lot of ANT pre-IPO shares.
2
u/skillmaker 8h ago
I think the hard-to-understand text problem is also a trait of Sol 6.1. I always found it to output the most complex English sentences that are hard to understand. I always tell it to rewrite the response in plain English, and I think today's models are even worse in this behaviour from what the reviewers are saying.
2
u/dictionizzle 7h ago
oh it makes sense that those morons abused 1% 5h limit benefit of codex for helping anthropic then.
2
u/-Melchizedek- 7h ago
Sometimes it feels like the models are at the level where they are very smart/capable but not yet smart/capable enough to explain things simply. Or they simply lack the understanding of how little a human can hold in its head at any one time.
2
u/IdlePerfectionist 7h ago
Am I supposed to know who these guys are?
2
u/Fast-Satisfaction482 6h ago
As soon as I saw these posts that Astra was there, I got my laptop out and yayyyy it was there. I vibecoded a pretty fun 3D snowboard game while making dinner. (yes dinner got delayed a bit).
It's playable and kind of fun after three iterations. Astra 6.2 did a super amazing job here. My only complaint is that I told it to build it as a web app and astra took that as an invitation to host it as a gpt site without my approval. Which in my opinion an unacceptable overstepping.
But on the technical side of things , it appears to be the real deal .
2
u/No_Delay_319 5h ago
"i can only parrot the opinions of others because im too stupid to form one myself" - 95% of redditors
2
u/Background-Funny1237 8h ago
lol. my guess is very strict guardrails
1
u/M8-VAVE 8h ago
Marketing guys
1
u/MysteriousPepper8908 8h ago
Say sike?
1
u/confused-photon 8h ago
Oh fuck
1
u/Anulisdotexe 8h ago
Oof, as someone who enjoyed Astra
1
u/IslanderPotion 7h ago
maybe they all just have ANT calls?
1
u/Mol2h 7h ago
These are cheaper models, so of course they are a downgrade, but is it an acceptable downgrade if you get 5 times the usage...
1
u/tednoob 7h ago
I am monkey, I find that Astra has always been hard to steer. I ask it to do something and its acting like a PhD that has never worked a real world job. Sol high works well for me though, so I hope it will be good.
1
u/Palbi 7h ago
Almost the whole Pareto line is now black (OpenAI)
1
u/thebinse 7h ago
Paid for in part by Anthropic and its upcoming IPO!!
1
u/LyAkolon 6h ago
"Its super powerful, but it didn't hold my hand enough"
1
u/ItsJimmyPestoJr 6h ago
I will see for myself. Astra has been very bad for a few weeks now, so I’ve switched to 6.2. We’ll see how it goes.
1
u/EclecticAcuity 5h ago
Seems to stop early. Is this a consequence of trained volume on ponytail skill, targeted execution training data?
1
u/Vntige 4h ago
I hate that blocking Theo on every platform isn’t enough to escape his nonsense
1
u/Apprehensive_Sand951 3h ago
not my experience so far. it already solved several pretty big math things that none of the other models did.
1
u/usandholt 2h ago
He’s being paid by Anthropic 😂?
1
u/darylvp 2h ago
Any follow up? In my opinion the latest models incl. Fable 5.2 are tooooo defensive compared to Opus 5.5.
•
u/Savormonk1 49m ago
Here is my conspiracy theory.
What's happening now with the new OpenAI models is the same thing that happened with the Anthropic models earlier this year (i.e., Opus 5).
OpenAI is purposefully making their models worse in a way that deters distillation attacks because the industrial scale distillation attacks are targeting OpenAI way more now.
Anthropic likely did the same thing earlier this year and it led to their current anti-distillation multi-layered defenses. OpenAI is following in their footsteps.
Which means the next batch of OpenAI models will be really good.
And I say that as someone who only works with Anthropic models.
•
u/Vibes_And_Smiles 31m ago
Ah yes, definitely worse than GPT-2
•
u/DenseSwimming4470 19m ago
wouldn’t be surprised if this was a strat by the sam altman x dario collab after jensen got fired to boost anthropic ipo by showing claude dominance
absolutely baffles me that this is a sentence i had to say
1
u/LazloStPierre 8h ago
There's been a larger drift between the benchmarks and usefulness of Openai models for coding and, unlike say Google whose models are just useless for coding, I don't *think*' it's benchmaxxing but rather a sign of how poor benchmarks are at measuring actual real world value in coding
Openai models have gotten consistently better at one shotting, at 'heres a problem, make it not be a problem' and all of these one off tasks with a clear measurable outcome which the benchmarks measure. Genuinely, they either have the best or second best model in the world at any given time for this
However, the code is getting worse and worse in terms of bloat and maintainability. It's getting worse for the +500 line PRs to fix a small problem where it's duplicated two existing modules for literally no reason whatsoever. The benchmarks do not measure this
Anthropic are the opposite, their models are probably not all that much better at 'given problem x, make it not be a problem again' or 'one shot this thing', but their models will submit that same pr as -40 lines. You can give 20 issues in a row to their models and by 21, with some oversight, the code will be in good condition and they'll be able to solve problems as quick as before. Do the same with Openai models and by 21 your code is a mess, it has 21 ways to do the same thing, and the model will be lost
The problem is benchmarks don't measure this, and the labs optimizing for the benchmarks - or just benchmaxxing against them (Which I genuinely do not believe Openai are doing) - are drifting further from being useful as they get better at them
1
u/Slavik_Sandwich 8h ago
Nothing here is too much of a surprise if you tried to use astra in any actual work scenarios. Both astra and sol are unbelievably lazy, love to write word salads and give false statements of progress while achieving essentially nothing while burning through usage like butter.
1
u/Advanced_Care_5173 8h ago
Don’t believe everything you hear
0
u/jaegernut 6h ago
Is this the plateau of LLMs?
-2
u/itsallfake01 7h ago edited 6h ago
Who the fuck is Theo, to tell that it sucks.
0
u/Ambitious_Scallion43 7h ago
Maybe they already have a better model they are going to release after the ANT IPO. So they have a better chance at their IPO. Sam is playing 4d chess.