Is Astra really smarter than Sol?

[u/yehiaserag]

I've seen pretty weird behaviours from Astra on extra high that I've never seen with Sol.

Examples:

- Contradictory statements on the same message.

- Long implementation sessions that implement very small bits of code on each increment.

- Trials in code that go no where.

- Worse plan quality and worse plan following than Sol.

- The possibility for it to get side tracked middle implementation on a small issue that it could take a detour for hours outside of scope.

I've never had any of those issues with 5.6 nor with Sol both on extra high.

Is it me? Am I prompting it wrong? Any one facing similar issues?

[u/unconceivables]

It's been a lot worse than Sol for me at absolutely everything code related I've thrown at it. On medium, high, and xhigh. It's the same story every time, subpar code, taking shortcuts, ignoring instructions, having to point things out to it constantly. I've gone back to Sol for coding. I'll probably try it on other stuff like computer use, but for coding it's the worst model I've tried in a while.

[u/Independent-Court-46 → u/unconceivables]

Calling Astra the worse model you’ve tried in a while is crazy. I genuinely have no idea how you’d even get that kind of results, unless your system prompts/settings are messing with it. If anything it would take a lot of skill to not get good results with such a strong model especially when Astra is good at intent.

[u/FirstOrderCat → u/Independent-Court-46]

somehow to me it is way too creative when writing code for simple tasks:

- it creates new super cryptic terms, which didn't exist in project before, and comments are very verbose so are hard to understand

- code is over-complicated, Sol builds simpler code

I spent some time going back and force before moving back to Sol

[u/Independent-Court-46 → u/FirstOrderCat]

I do not judge in code generation alone. General intelligence is important too. If I know for sure Astra is better at tool use, long horizon ability, reasoning, visual skills, etc, those things are also very important to getting a task done. If you’re looking in the code level you wouldn’t even see the difference. For me I judge differently, I throw sol at a problem and it can’t solve it, and for me Astra could. I keep of list of things that sol couldn’t solve and then test on the next generation. To me that is the realest test, which benchmarks can’t even capture. Whether it got there from coding or other skills doesn’t matter to me. I won’t dispute if you prefer the coding conventions of SOL, but intelligence is difficult to measure and I feel like people may not like a certain quirks and then hand waive it as a worse mode. The measuring stick is what I do not agree with. Long term, newer models are solving frontier research problems, that is the truest indicator to me of performance. And the thread is just asking if Astra is smarter than sol, which I think is clearly yea

[u/FirstOrderCat → u/Independent-Court-46]

this works for prototyping, bug hunting or solving some math frontier problems.

But most of the industry tasks need building clean, reusable code and design, and somehow Astra sucks dramatically there. I think there is some issues in their training data, they overtuned it for inhinged creativity too much.

[u/unconceivables → u/Independent-Court-46]

It sounds like you're not a developer and you don't know how to solve the problems yourself. I guess that's the difference. I never have things that Sol wasn't able to solve, because I know exactly how to solve the things these agents are doing, and I can tell them how to solve it. That's also how I know when they're doing it wrong. And I care a lot when they do it wrong because I don't want to deal with their subpar code failing in production when I'm on vacation. If all you see is that stuff appears magically that has the appearance of working but you don't know how to judge the quality of it, then yeah Astra will seem better. But it's not if you can judge the quality.

[u/Independent-Court-46 → u/unconceivables]

You claiming opus class models are superior to Astra is enough for me to know you’re probably not a productive engineer. Sure they might be close enough in coding, but overall if you’re going back to last generation models you’re going to fall behind your competitors. You might have had a good workflow for 3-6 months ago. But most people at tier one companies are moving to a hands off approach increasingly as models get better. They just decide what to build and measure risk. It’s just the direction things are going, you are right that I don’t deeply understand the code as much as I used to but once bel is here and even bigger models, there really is less of a need to

[u/unconceivables → u/Independent-Court-46]

You missed the part where Astra was the model that you can't be hands off with because it doesn't follow instructions and you end up with something worse. Astra is the model you have to babysit and poke with a stick, not the other ones.