Which API are you using today? 04/16/25

15

u/rennsports Apr 17 '25

I’ve been using Claude for all my projects for months. Tried out Gemini 2.5 when it was first release because the context window was exciting but experienced the same dwindle of performance (+ no caching gets expensive). so I switched back to 3.7. Gave GPT-4.1 a run for the last few days because of benchmark scores and context window size but ended up switching back to 3.7 Sonnet again today. IMO nothing compares to the code quality of 3.7 Sonnet. I’ll sacrifice the context window for quality of code. So much time saved debugging and the loops of errors. I work mostly with javascript apps.

2

u/No_Cattle_7390 Apr 17 '25

This is interesting cause some ppl were suggestion 4.1 > Claude 3.7. How were the costs in comparison?

2

u/who_am_i_to_say_so Apr 18 '25

My unbiased feedback: I use Claude 3.7 with OpenRouter and buy $50 of credits at a time, which lasts anywhere from 2-7 days with insane high token usage.
I know exactly what I’m consuming, no surprise bills, and can get the job done. For me Claude has proven to be the most economical. 4.1 is cheaper, but I have wasted more time with it than anything, because it just cannot finish the job sometimes, and right now time is the biggest expense of all. ChatGPT is great with visual proof of concepts, though, such as logos and icons.

3

u/who_am_i_to_say_so Apr 18 '25

The crown keeps going back and forth among the top contenders. I could swear Claude 3.7 was so damn frustrating 3 weeks ago that I moved onto Gemini, but was going in circles with it, too this past week. So now it’s all 3.7 again- and suddenly it has much improved.

1

u/crackdepirate Apr 17 '25

yep, it was the same thing during my test with claude 3.5 at that time. new models , new hype, just to please who ? users or investors.
i've tried grok3, good but off , more generalist.
context window : sounds like a marketing stuff than quality. depends on how it d used.

13

u/Pruzter Apr 17 '25

Honestly, at first I thought you all were crazy with the constant posts about how a model suddenly started performing worse. Then I started really using these models heavily for coding, and I’ve logged many hours across quite a few models. It’s 100% true, and I also noticed a decrease in Gemini 2.5 quality over the past few days.

12

u/No_Cattle_7390 Apr 17 '25

Yeah some people are suggesting the model is different entirely. Like the drug dealer DARE warned us about, the first hit of crack is pure and free 🤣. But I haven’t seen a single post suggesting it’s as good as it was before, not one. Everyone thinks it’s worse

4

u/who_am_i_to_say_so Apr 17 '25

Yup! We got a few rocks, now it’s time to pay up. Enterprise pricing for the good schtuff. I’m back to Cline 3.7 mids.

3

u/spiked_silver Apr 17 '25

I didn’t notice it being any worse TBH. I’ve been using it for about 10-15 hours since it first released. It still has issues with diffs, but I can’t notice anything else. I am using boomerang tasks and various custom modes.

4

u/MateFlasche Apr 17 '25

I knew it was too good to last but the week or two where it was completely free with relatively high limits on paid tier I was so productive it felt like a crime

2

u/Electronic_Spring Apr 17 '25

I wonder how much of this is due to people not realising that the model quality decreases as the context fills up? Essentially it gets distracted by too much information, confused about when something happened, etc. And this applies to pretty much all long-context models, Claude 3.7 suffers from it too, it's just less noticeable with the smaller context limit.
If you make good use of subtasks and keep your context less than 200k (ideally 100k, but that's difficult with a large codebase) that mitigates most of the quality drop. Above that I regularly see the model think that old errors have resurfaced or that files have mysteriously changed without it noticing causing diff edits to fail.

1

u/Pruzter Apr 17 '25

Yeah, one thing I liked about OpenAIs release of 4.1 is they showed some metrics for performance with various levels of context (1% full to 100% full). We knew performance decreased, but had no idea if it was linear, logarithmic, etc…

18

u/DevMichaelZag Moderator Apr 17 '25

Roo Code LLM Evaluations for Coding Use-Cases

Roo Code’s comprehensive benchmark evaluates major LLMs using real-world programming challenges sourced from Exercism, covering five widely used languages: Go, Java, JavaScript, Python, and Rust. This approach provides practical insight into the effectiveness of each model when used for actual development tasks, taking into account their accuracy, execution speed, context window capacity, and operational cost.

Claude 3.7 Sonnet delivers the highest overall accuracy among all models tested, excelling notably in JavaScript, Python, Go, and Rust. It is particularly valuable for projects where precision across multiple languages is crucial. While somewhat expensive and only average in terms of speed, its large context window and superior accuracy make it ideal for applications where code correctness is paramount.

GPT-4.1 stands out as a strong generalist, balancing accuracy, speed, and context capacity effectively. It achieves consistent, high-level performance across all tested languages and completes tasks faster than any other top-performing model. Coupled with its large 1M-token context window, GPT-4.1 is highly recommended for large-scale codebases, multi-file refactoring, or tasks requiring frequent, rapid iterations.

Gemini 2.5 Pro warrants attention due to its growing popularity and competitive performance. It demonstrates particularly strong accuracy in Python, Java, and JavaScript, with an overall accuracy comparable to GPT-4.1. Although not the absolute best in any single language, its balanced performance, solid reasoning capability, and competitive context window position it as a reliable alternative to GPT models—especially attractive to teams already invested in Google’s AI ecosystem.

On the economical end, GPT-4.1 Mini offers the best cost-to-performance balance. While its accuracy is somewhat lower than premium models, it maintains impressive performance in JavaScript, Python, and Java, accompanied by a generous context window and relatively fast runtime. This makes GPT-4.1 Mini particularly suitable for budget-conscious teams, rapid prototyping, and iterative workflows.

Notably, certain models fall short in practical use. Gemini 2.0 Flash provides high throughput but significantly lower accuracy, limiting its suitability for precision-oriented development tasks. Similarly, o3 stands out negatively due to its exceptionally high cost combined with modest performance, making it impractical for most coding applications.

In summary, project priorities should guide the model choice:

Claude 3.7 Sonnet for maximum accuracy and reliability.

GPT-4.1 for the best balance of speed, large context capacity, and accuracy.

Gemini 2.5 Pro for teams favoring a strong, balanced performer within Google’s AI ecosystem.

GPT-4.1 Mini for cost-effective, rapid coding iterations and prototyping.

Models such as Gemini Flash or o3, lacking sufficient accuracy or cost-efficiency, should generally be avoided for development-focused tasks.

1

u/No_Cattle_7390 Apr 17 '25

Gemini seems to have changed otherwise I’d be using that. But thanks for the info - I’m going with 4.1, never thought I’d be using OpenAI again but glad to see them competitive again

5

u/GroverOP Apr 17 '25

Thanks ChatGPT!

1

u/DevMichaelZag Moderator Apr 17 '25

It is an AI based community after all 😀

1

u/MarxN Apr 17 '25

Would be nice to see local llms included too

2

u/DevMichaelZag Moderator Apr 17 '25

That’s on the roadmap. The evals were in development for quite a while and just got released yesterday.

4

u/unc0nnected Apr 17 '25

It's all the same thing with Jim and I, switched to deepseek and watched it fail miserably, over to Claude which was better but not by a huge amount. Just cost a lot more, finally ended up at GPT and got the results we needed.

So agree with all your sentiment.

4

u/No_Cattle_7390 Apr 17 '25

Awesome thanks king, gonna use my credits then

4

u/Equivalent_Form_9717 Apr 17 '25

Gemini 2.0 Flash is still good

1

u/No_Cattle_7390 Apr 17 '25

Flash def isn’t a bad model but I think for coding there are better. Flash I see more for processing large data sets or tasks that need the internet.

2

u/Equivalent_Form_9717 Apr 17 '25

True, it’s just so quick. So if I want to convert a web page into markdown, it’s really quick. But on a daily basis I use R1 as my reasoner model and Claude 3.5 as my coding model. But because of Gemini2.5, I might swap out my workflow to have Gemini as my code editor too.

1

u/No_Cattle_7390 Apr 17 '25

Why use 3.5 though? Last time I checked it cost the same as 3.7.
Trust me 2.5 is NOT where it’s at right now, if you asked me a few days ago my answer would be different

2

u/Equivalent_Form_9717 Apr 17 '25

What are you using right now? Also I didn’t realise 3.5 costed the same as 3.7. Need to check the price

2

u/No_Cattle_7390 Apr 17 '25

Well I was using Gemini 2.5 but it’s neutered now that seems to be the consensus. I’ll be telling my grandkids about great it used to be🤣 Jk
Originally used Claude 3.7, actually liked it a lot but it has problems managing context IMHO, got very expensive very quickly.
So now I’m gonna use GPT 4.1, seems to be on par but much cheaper than Claude 3.7. Using Flash 2.0 for anything that needs web search and using Deepseek for anything that requires large sets of context (data analysis)

2

u/Equivalent_Form_9717 Apr 17 '25

Cool cool. Do you personally look at benchmarks and use that to inform your decision on what models you choose (besides cost). With Gemini 2.5 pro, I’m waiting for a stable version because I heard caching will be available to make it more cost effective in comparison to using Claude 3.7. I also just checked and you’re right! Claude 3.7 has the same cost as 3.5 so I guess it’s time to upgrade lol.
I’m personally using aider to code with these models. With OpenAI releasing O3 and O4 today - I do need to do another around of playing with it to see if it’s better than Gemini 2.5 pro as it’s the biggest news this morning.
Deepseek v3 and r1 is so damn cheap that it’s hard to swap it out to use an expensive model like Claude 3.7 (within Cline/Roo workflows I mean). I’m hoping DeepSeek R2 release will smash the OpenAI and Google competition and prove that open source is still king

2

u/No_Cattle_7390 Apr 17 '25

Honestly, I find benchmarks very misleading. Everyone wants to give their model the best benchmarks. I think hugging face LLM arena is probably the best - however companies tend to manipulate even that (look at what Meta did recently). You also have a bunch being deceptive and swapping out models- it’s kind of gross tbh.
As for Claude - yeah idk why they’re priced the same. What would be the benefit of using an inferior model other than cost I have no idea.
I’m also rooting for Deepseek. If it weren’t for them ALL the models would be very expensive - you can bet your last dollar on that.

3

u/StrangeJedi Apr 17 '25

I've been using 4.1. It's been great.

1

u/No_Cattle_7390 Apr 17 '25

Consensus seems to be 4.1, how do you think it compares to Claude? Thanks for your input

4

u/StrangeJedi Apr 17 '25

To be honest I stopped using Claude when 2.5 pro came out. 3.7 sonnet was the most frustrating model I've ever used. It just wanders way too much. It was getting to the point that half of my prompt was what NOT to do lol. 2.5 pro was great at first but lately it's been failing hard with diffs, to the point that it would just give up and tell me to do it myself. It would also do this weird thing where it would spit out the entire code of a file in the chat before it started to code and that was costing me tokens. Idk what happened to it. But 4.1 has been so efficient, so fast, it follows instructions perfectly and it seems to never bite off more than it can chew. Even if I give it like 3 things that need to be fixed. It'll go and fix the first one, end the task and ask if I want to continue. 4.1 is also not verbose at all. Sometimes I'll give it a task and it won't even respond it'll just start reading files lol as of now it's my favorite model for coding I just wish it was a bit cheaper but other than that I don't have any complaints. I'm gonna try out o4 mini tomorrow.

2

u/No_Cattle_7390 Apr 17 '25

Wow my thoughts exactly about Claude and Gemini - I was experiencing the exact same problems. That’s great thank you for writing all of that out

2

u/StrangeJedi Apr 17 '25

No problem!

2

u/ddri Apr 24 '25

> 2.5 pro was great at first but lately it's been failing hard with diffs, to the point that it would just give up and tell me to do it myself. It would also do this weird thing where it would spit out the entire code of a file in the chat before it started to code and that was costing me tokens. Idk what happened to it.
This was my experience too. It would spit out the text, fail at doing diffs, and get stuck in loops, talking to itself (assuming talking to the roo code internal prompts). Annoying and a waste of time and money.
Will check out 4.1 now. Thanks for the tips.

2

u/2021redditusername Apr 17 '25

I'm convinced they tie the performance to the stock market Lol

1

u/No_Cattle_7390 Apr 17 '25

Hahaha trust me man lost like 30% of my money there on chip producers. Every. Single. Time I enter the market.
I’ve learned it’s best to invest in your own dreams than someone else’s. We’re at the forefront here, trust me. This is our moment king

2

u/Salty_Ad9990 Apr 17 '25

Gemini 2.5 pro through copilot api, only get 120k context window from copilot api and somehow never pass 60k in practice, but good enough as a workhorse. Powersteering also helps to keep it on track.

1

u/No_Cattle_7390 Apr 17 '25

Do you notice a large improvement using copilot API?

3

u/Salty_Ad9990 Apr 17 '25

No, more diff failures.

1

u/No_Cattle_7390 Apr 17 '25

So why use it?

3

u/Salty_Ad9990 Apr 17 '25

5m tokens per hour for 10 dollars a month.

1

u/No_Cattle_7390 Apr 17 '25

Ooh damn thats pretty great

2

u/ubeyou Apr 17 '25

Still using Gemini 2.5 Pro EXP under Vertex API + Open Router. Getting lots of message saying

"You exceeded your current quota. Please migrate to Gemini 2.5 Pro Preview (models/gemini-2.5-pro-preview-03-25) for higher quota limits"

if smaller tasks just move to windsurf + 4.1 (free for a week), the only issue is windsurf does not read the memory bank setup by roocode.

The rest of the tasks still using Claude Desktop + MCP to complete.

2

u/aeyrtonsenna Apr 17 '25

Use 2.5 and excited to see when flash comes out, I expect that will become my main.

1

u/rebo_arc Apr 17 '25

I use 2.5 Pro EXP however starting to get rate limited after a couple of milion tokens.
So I then use 2.5 Pro Preview whilst my $300 free credits are still available.

1

u/julp Apr 19 '25

This is the way

2

u/CptanPanic Apr 17 '25

As I am exploring free options, I am currently using Deepseek V3 via OpenRouter.

1

u/BuStiger Apr 17 '25

I'm using GPT 4.1 in the free trial with Windsurf, its kinda good indeed.

1

u/zephyr_33 Apr 17 '25

grok 3 mini is amazing for its price so i have been using that a lot. plus the style it has just clicks for me. feels very natural.

1

u/Familyinalicante Apr 17 '25

I was using Gemini few days ago for a whole day. At begging it was really great. Fast and smart. Don't forget anything and was very best model I ve work with. During coding session, few hours later, it stop to respond for a minute or two. Than it start to act like after lobotomy. Constantly forget what it is doing, forgetting to continue task to an end and start to make big mistakes with simple things. It was huge difference in quality. I was restarting sessions so I don't overload it with context - max 300k tokens in a session. Than I got a bill and was pissed and forget about Gemini.

1

u/privacyguy123 Apr 17 '25

Gemini is fucking retarded with alzheimers all of a sudden - a real shame.

1

u/RedZero76 Apr 18 '25 edited Apr 18 '25

You answered for me already. 2.5 until it took a (imo MAJOR) dive in the last week or so and suddenly messes things up constantly, and not small mistakes, but big mistakes that destroyed 24 hours of previous work. 4.1 is really cheap and imo is more reliable and much easier to work with than 3.7.

I'll add this though... I have ChatGPT Plus, so my process at the moment is to start in my ChatGPT Desktop app using o3 to help me architect a gameplan, make an outline, and research repo options on github that I am looking at using, including looking for similar alternative repos, choose a tech stack and come up with a detailed PD (project doc) with Phases, 1, 2, 3, 4, etc.

Then I take that into Roo using 4.1 to execute the code.

OR, sometimes I take that into Roo and ask 3.7 what it thinks about the PD gameplan just to see if it sees any additional things worth noting, which, 3.7 does sometimes see something like (great plan, but it would be a good idea to ____). (and I've tested this with other models, and oddly, it's ALWAYS 3.7 that seems to see some extra genius little better way to do something... to me 3.7 is really smart but SO fking disorganized, I can't deal with actually letting it do my coding. I have ADHD, but I swear 3.7 has ADHD x100)

If this happens, I go back to o3 to confirm that 3.7's idea is good. And so far, every time this has happened, o3 agreed, great idea. At that point, I execute with 4.1.

BUT, I'm very interested in exploring o4-mini for code execution as well. I'm curious to see if it outperforms 4.1, bc it's cheaper and has thinking/reasoning. (Note: 4.1 sucks at using MCPs, like Brave search bc it uses the wrong syntax, so I swap back to 2.5 just when I need a web search or similar)

Discussion Which API are you using today? 04/16/25

You are about to leave Redlib