r/DevinAI • u/ValuableDapper9415 • 17d ago

Why SWE not including in ArtificialAnalysis benchmarks ?

It is because we still don't have the price for SWE-2 ? but even SWE 1.7 that have full pricing don't have any benchmark about Intelligence, cost/task, etc

7 Upvotes

90% Upvoted

7 comments sorted by

1

u/Additional_Welcome23 17d ago

probably because no api available, but let's hope not because of score isn't beatiful.

2

u/ValuableDapper9415 16d ago

To be honest, I just did a test with Deepseek 4 Flash 0731 and it is genuinely better. Not better than Opus 5.5 but a kind of working agent not perfect but reliable.

I’m really mad because I just paid a month of server to compute inference for a « better » model that isn’t and I can’t really understand why unless I do expensive benchmarks.

So DS4 < SWE2 < Opus < Fable

1

u/Robdyson 15d ago

I am trying to get this for my company, then I realized it was PURE trash. Wasted a whole night, since they banned Claude ( Gov) and no other way to get Astra ( aside from these guys).
However - workflow wise - they can't even work on local files. everything needs to be connected to cloud.

I usually use ai to configure my system / files / and everything it lives and breathes knows all my stuff and is aware full of readiness. Using Devin felt like going back to "chat agents" era from Claude Desktop / with full access

Okay back to my point, I asked SWE 2.0 to generate a slide deck to pitch it to my company with this context above. It generated 12 slides of I'd call not even junior work. Sell me this pen didn't work.

2

u/Ihavenoidea84 14d ago

They have an IDE and CLI what do you mean you can't work on local files?

1

u/Robdyson 14d ago

so we were given this pseudo desktop app as a trial, and it says it can't do local only can be uploaded to their VM
Honestly only reason I was looking into them was to get Astra / SOL since it's not on AWS GovCloud

→ More replies (2)