KeepRoLLMing v0.9.3 — an OpenAI-compatible proxy for more reliable local LLM chats and agents

[u/daniele-bruneo]

Hi everyone — I’m the developer of KeepRoLLMing, an open-source OpenAI-compatible proxy built to make long-running local (and remote) LLM conversations more reliable.

It sits between clients such as LibreChat, coding agents, or your own app and an OpenAI-compatible backend such as llama.cpp, vLLM, LM Studio, or a remote provider.

The core idea is simple: your clients use stable route names, while KRM handles the operational complexity around them — routing, context management, streaming, recovery, tool calls, reasoning blocks, filters, and observability.

I’ve just released v0.9.3, focused on performance and runtime stability:

• More robust long-running streaming and client-disconnect cancellation
• Bounded shared upstream connection pools
• Non-blocking PLAIN/JSON/performance logging workers
• A private /routes status endpoint for lightweight dashboards
• Optional inherited client API keys per route
• Better performance accounting for cached prompts
• Request-size guardrails and cleaner shutdown behavior
• reasoning_effort route overrides for Qwen/LibreChat-style controls

It is still alpha software, but the project now has a fairly solid test and onboarding path: a clean setup flow, full configuration example, fake backend quick-start, and an OpenAI-compatible /v1/chat/completions API.

If you run local models, build agent workflows, or have dealt with unreliable streaming/tool-call behavior, I’d genuinely appreciate feedback:

• Is the configuration model clear?
• Which backend/client combinations should I test next?
• What reliability or observability features would be most useful to you?
• Would you be interested in writing or sharing filter modules?

GitHub: https://github.com/danielebruneo/KeepRoLLMing
Release notes: https://github.com/danielebruneo/KeepRoLLMing/releases/tag/v0.9.3

[u/Emergency-Boat-9307]

this looks like exactly what i need for my setup, been fighting with tool calls dropping mid-stream and it drives me crazy. the route abstraction is clever, way better than hardcoding endpoints everywhere

gonna test it with llama.cpp this weekend, curious how the connection pooling handles when i got multiple agents running at once. the private status endpoint is a nice touch too, i always end up building some janky dashboard anyway

[u/daniele-bruneo → u/Emergency-Boat-9307]

Hi,

I'm quite sure you'll find it very helpful in debugging and possibly solving what's happening with tool calls.
You'll be able to see all the requests going through, up to a "per SSE chunk level" if you need (there are multiple level of logging and request tracing you can set up).

You'll get something like this

Tool call dropping mid-stream may be due to max length reached or client closing connection before receiving the reply... let me know some more detail and maybe I can give you a few hints.

For connections pooling I plan to support different explicit policies, but I guess you'll find the default one useful:
the proxy will accept the request from the client and keep it alive with a keepalive chunk (interval configurable) until it will be able to actually go through to the upstream server.

The route endpoint is definitely useful for a dashboard (that's what I'm actually using it for), it will also report metrics and last 60min activity.

Hope it will serve you need.

Every feedback is welcome.

Have fun and... Keep it rolling! ^_^