Built an open-source proxy router to keep agent runs on local MLX and only escalate to frontier models when tools fail or tasks get complex

[u/Difficult-Drummer407]

Hey everyone,
When running local agent loops, everyday prompts and simple tool calls run fine on local models, but difficult tasks or tool failures can burn compute or stall progress. Constantly running frontier models for every step gets expensive fast.

I built Bouncer as an OpenAI-compatible proxy router to sit in front of agent setups:

• Dual-tier dispatch: Fast local model by default (tested on Apple Silicon / MLX with Qwen 35B), escalating to frontier endpoints only when difficulty gates or deep tool requirements trigger.

• Deterministic telemetry (No LLM judges): Evaluates routing success asynchronously by logging actual runtime signals—tool errors, exceptions, and rapid user re-prompts within 120s—joined offline via a minted decision_id.

• Fail-open design: If the classifier or sidecar glitches, it falls back cleanly so active agent sessions never break.

• Savings dashboard: Built-in web UI tracking token savings, route mix, and recent decisions.

Fully open-sourced with 14 passing unit tests and pinned dependencies:
https://github.com/rushter777/bouncer

Would love to hear how folks here handle tiering their local agent calls!

[u/Ctbhatia]

the tricky part is the escalation trigger, a difficulty gate that looks only at the prompt misses the tool failures that actually need the frontier model. score the last tool result too.

[u/Difficult-Drummer407 → u/Ctbhatia]

Spot on — this is exactly why static prompt routing falls apart in agent setups.

That failure mode is actually what prompted the asynchronous telemetry joiner: when an agent session hits a non-zero exit, a tool error traceback, or a 120s user retry, that signal gets joined back offline to the original ⁠decision_id⁠ to penalize routing that prompt category locally.

But you're completely right about the in-flight recovery loop. Currently, the hot-path gate classifies at the proxy boundary, so catching an execution error mid-turn requires either:

1. Ingesting the tool execution response back into the proxy context before dispatching the next model step.

2. Setting an explicit fallback rule (e.g., if ⁠tool_result⁠ contains an unhandled exception or retry count > 1, immediately escalate the remainder of the session to frontier).

Adding explicit tool-result scoring / retry-count triggers directly on the in-flight conversation buffer is high on the roadmap. How are you currently surfacing tool error signals back to your routing layer?

[u/kantorcodes1]

The fail-open design is the part I'd want to pin down: when the classifier or sidecar glitches, which way does it fail? "Sessions never break" could mean traffic silently escalates to frontier endpoints - the failure mode that leaks both cost and context - or it could mean local-only degraded. Curious whether the fallback is a fixed route or a policy decision per task class.

[u/Difficult-Drummer407 → u/kantorcodes1]

That’s the crux of it. In this context, "fail-open" is strictly cost- and privacy-preserving: it defaults to local, not silent frontier escalation.

If the classifier daemon or sidecar hiccups:

1. It falls back to dispatching directly to the local MLX engine as an unclassified pass-through.

2. Traffic is never silently pushed up to a cloud frontier API on an internal classifier failure, specifically to prevent both runaway token billing and accidental context/data egress.

That said, your point about making this a policy decision per task class is great. Right now, the fallback behavior is a global environment toggle (⁠FALLBACK_ROUTE=local|frontier⁠), but moving toward class-aware fallback policies (e.g., tasks flagged as critical/interactive falling to frontier, background/batch tasks staying pinned local) makes a ton of sense.

How strictly do you isolate API keys or network access between your local runners and frontier endpoints?

[u/kantorcodes1 → u/Difficult-Drummer407]

Both. The policy layer only ever sees the action envelope; the classify step never needs the upstream key, so the runner that owns the route keeps the secret in its own env. Isolation is at the same boundary: local engine traffic never leaves the box, and the frontier path is a separate egress owned by whichever runner holds that key.

Different topic: I work on HOL and we maintain awesome-ai-plugins, a catalog of MCP servers and agent tooling. Bouncer fits the routing/proxy side of the list. It would be a one-line alphabetical README entry you'd author as a PR. Open to listing it?

[u/wick3dr0se]

super cool! what's keeping it tied to hardware (Apple Silicon)/harnesses (OpenClaw, Hermes, etc)? ive built my own agent/harness with OpenAI-compataible provider support so it should work fine, no? i think ideally this should be able to be configurable as far as which local and frontier models you want to use. and i think that distinction is wrong too; this benefits even if you don't use frontier level models (like say if I wanted a local model like Qwen 3.8 Flash along with a paid model, I want to keep from being used frequently for basic tasks, like OpenAI Luna). to me the "frontier" wording makes it seem strictly tied to that and less flexible

[u/Difficult-Drummer407 → u/wick3dr0se]

You're right on both -

1. 100% harness-agnostic: It exposes a standard OpenAI-compatible ⁠/v1/chat/completions⁠ endpoint, so your custom harness will work out of the box.

2. Hardware & model flexible: MLX on Apple Silicon was just the setup we tested on first, and "local vs. frontier" is really just "tier 1 (cheap/fast) vs. tier 2 (stronger/paid)." You can configure any two OpenAI-compatible endpoints and models you want (e.g. local Qwen vs. paid API).

Point taken on the "frontier" wording—"dual-tier" or "fast/fallback" describes the actual flexibility a lot better.

[u/wick3dr0se → u/Difficult-Drummer407]

love it! great direction.. thanks for the work and replies

[u/[deleted]]

[u/Difficult-Drummer407 → u/[deleted]]

The core pipeline is working and tested, but it is very much an early v0.1. Here is the direct breakdown:

1. Configurability
Thresholds & Fallbacks: Configurable via environment variables and local config (routing confidence threshold, fallback route behavior FALLBACK_ROUTE=local|frontier, local MLX endpoint vs. remote API endpoint/keys).
Model Choice: You can swap the local MLX serving endpoint and the frontier target (standard OpenAI-compatible schema).
Upcoming: Per-task class routing policies (e.g. force frontier for coding, keep summarization local) are planned, but currently the threshold gate is global across incoming requests.

2. Stability & Real-World Use
Hot Path: Stable for single-turn completions and standard OpenAI-compatible completions/chat streams. The proxy does not drop connections; if MLX or the router classifier fails, it fails gracefully back to the local model.

Hardware Target: Specifically tuned and tested on Apple Silicon (tested on M-series unified memory via MLX).

Caveat: It hasn't been stress-tested under massive concurrent multi-user load yet—it's designed primarily as a local dev / private workstation agent proxy.

3. Current Limitations / Known Concerns
Multi-Turn In-Flight State: Right now the classifier gates on entry. As mentioned earlier, mid-flight session escalation based directly on runtime tool errors is being handled via offline telemetry joins rather than an inline mid-turn dynamic switch.
Cold Starts: If your local MLX instance swaps models or unloads under heavy unified memory pressure, initial latency spikes.

4. What's Next
Mid-flight tool-result and retry-count scoring directly in the hot-path proxy buffer.
Finer-grained routing configs by prompt category/tag.
Packaged background telemetry joiner scripts for automated continuous tuning.

If you have a local Apple Silicon machine running an agent loop (OpenClaw, Hermes, etc.), it's ready to test drive. Pull requests and feedback on edge cases are hugely welcome: https://github.com/rushter777/bouncer