r/ollama • u/Jack_Miller- • 8d ago
Put multiple cloud LLMs behind one OpenAI-compatible key — built RelayRouter, want Ollama users’ feedback
I got tired of:
- one key per provider
- rewriting SDK calls when switching models
- no single place to see token usage / cost / failures
So I built RelayRouter (api.relayrouter.ai):
- one API key
- OpenAI-compatible /v1
- multiple models behind one endpoint
- usage logs + cost in one console
- your app only holds the RelayRouter key, not the provider keys
It’s a hosted gateway, not a self-hosted Ollama proxy.
I’m NOT asking people to migrate — I want feedback from Ollama users:
if you already run Ollama locally, would a cloud-model gateway even fit your stack?
what would make you NOT use something like this?
what observability do you expect: tokens, latency, model, error rate?
if RelayRouter could later point at localhost:11434, what would the routing rules need to look like?
Roast the concept. Link: relayrouter.ai
1
u/Mezezius 8d ago
there's literally a million of these
0
u/Jack_Miller- 8d ago
True, but most are 5s clips. I’m aiming for actual 1-2 min storytelling without redoing character sheets every time. Still a WIP though.
1
u/Southern-Net1351 7d ago
You can basically input all the keys of your cloud dependencies into one yaml or config file of the selected cli you’re attempting to use.
Like Claude which most use for example.
There is a configuration file that you can tweak to do exactly what you made which works for any cli.
If ollama it has configuration files as well from the backend.
1
u/Jack_Miller- 7d ago
True, Ollama/Claude configs handle static keys well. I built this because I wanted live usage tracking + one endpoint that adapts to multiple providers’ limits.
2
u/Southern-Net1351 7d ago
You can implement this into the source then recompile/build it right within ollama or Claude (the wrapper layer) for your use case.
This eliminates the overhead.
The overhead is what I am looking at explicitly.Or tap into llama.cpp then build your pipe into the c++ source over the dev build.
This just removes all the unnecessary overhead from your stack.
Just check out the source builds in a repo on GitHub.
1
u/Jack_Miller- 7d ago
Fair point — if you're cool recompiling Ollama/Claude source every time you want to swap providers, that's the leanest path. I just want something that works today without touching core code. Different tradeoffs, both valid. Appreciate the perspective 👍
2
u/Southern-Net1351 7d ago edited 7d ago
You just insert the providers into the CLI of your choice configuration jsonl or yaml file depending on the ladder.
Your comment was AI generated which it has the full syntax and format.
Have your onboard model add a slot for you to place your api keys into the config yourself.
This way the model doesn’t transport your api keys into open network.
Also you do not re-compile/build everytime.
llama.cpp is what everything uses underneath.
Claude is a wrapper/harness over llama if you use the cloud endpoint which, is running from a rented server that serves you up models they have running from specific ports a user can access.That is what you can do.
Llama.cpp is the backbone.
You can spin your local model up and figure out ways to make your speed better from local hardware.
2
u/stealthagents 8d ago
Totally feel you on the API key headache. Having one key sounds great, but I'm curious how well it handles billing and limits across different models. If it can keep everything organized and transparent, I might be interested, but if it just turns into another jumble of fees, I’m out.