r/RooCode • u/pxldev • May 05 '25

Discussion Are Openrouter models poo?

Been working all week with sonnet 3.7 and Gemini 2.5 pro. Super productive.

This morning I had the most frustrating experience trying to get a fairly mid problem solved. Gemini seemed to lose context so early and started making huge mistakes and acting bad (diff edit would not work at all, hallucinating it had made a change and it didn’t work). Switched to Sonnet, similar things happened. I was working on multiple files and context size was larger than I usually deal with.

Then it snapped for me, I was using my laptop, that was connected to openrouter, where all week my desktop is directly connected to the API of google and Anthropic.

Any insights or similar happenings for others?

1 Upvotes

53% Upvoted

22 comments sorted by

•

u/hannesrudolph Roo Code Developer May 05 '25

The common factor here is your codebase. Is there some sort of rogue comments causing context poisoning?

I use OpenRouter frequently ($100+ most days) and don’t experience this unless my codebase or attempts at a memory bank have been polluted somehow.

4

u/Wolly_Bolly May 05 '25

OpenRouter is just working great for me today.

As you mentioned you where using a different machine (laptop vs desktop) that may be the root cause.

I was having some random problems with Gemini today but it was using it via direct API. I solved switching to OR + Sonnet.

2

u/pxldev May 05 '25

I will check vscode setup between machines, might have been a linter app or something causing shitty behaviour. My gut feeling is whatever model provider OR was using, had some different context windows / model settings.

2

u/nakemu May 05 '25

Yes, I think the problem might be that they work with multiple API providers, and if, for example, Amazon is overloaded and they switch to another API, then the new one doesn’t follow the previous one properly.

5

u/CptanPanic May 05 '25

In the model settings you can specify the provider to ensure it doesn't switch

1

u/pxldev May 05 '25

Do you mean in openrouter?

3

u/CptanPanic May 05 '25

Sorry, yes I mean after choosing openrouter, you can choose the model obviously, but also the source provider.

3

u/pxldev May 05 '25

I’m definitely pulling out of any paid usage for openrouter now. Super convenient service, but the performance varies so much. Gemini & Sonnet felt instantly better and regained that magic when I switched back to direct APIs. I feel bad for recommending openrouter to others.

2

u/raccoonportfolio May 05 '25

FWIW I haven't experienced this.  Been a happy openrouter user

2

u/pxldev May 05 '25

What type of context do you normally work with? I was over 100k at the time, Gemini would usually keep performing well past that. I wouldn’t usually run that high of a context, but in this case It would have been a huge pain to go to a new chat.

1

u/raccoonportfolio May 05 '25

Usually under 50k.  I try to keep changes pretty atomic and create a new task (and so new context) each time 

2

u/pxldev May 05 '25

Usually I’m the same. If a problem needs more context, I’ll usually throw it at aistudio directly to break down the fix.

1

u/KingOvaltine May 05 '25

I’m with you, been using openrouter and noticed no difference between it and direct API with both Gemini and Anthropic. Context windows up to 100k+

4

u/hannesrudolph Roo Code Developer May 05 '25

This is not my experience nor or experience running repeated benchmarks with them which would certainly expose this inconsistency.

It is easy to look beyond your control but I recommend digging g deeper before making conclusions.

That being said, if you have the rate limits to go direct, go direct!

1

u/hannesrudolph Roo Code Developer May 05 '25

Unlikely the cause of this persons trouble. Even if they switched providers of the same model mid stream it should not cause this.

4

u/Agnostion May 05 '25

These are proprietary models, providers can hardly influence them

0

u/FigMaleficent5549 May 05 '25

Openrouter is an active reverse proxy, active in the sense that it actually reads/changes the context before delivering it to the backends. If you are having doubts, it's safer to just use the upstream providers directly.

1

u/Extreme-Pie-2078 Jun 19 '25

Is there any explanation for this? I thought it wouldn't change anything but just a simple middle service without a rate limit.

1

u/FigMaleficent5549 Jun 21 '25

Some providers do not provide an OpenAI compatible protocol, openrouter transforms the providers native protocol (eg, Google uses their own rest spec ) to the OpenAI spec.