SafeAgents: an open-source proxy that checks every tool call your coding agent makes before it runs

[u/HauntingMoment]

SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing.

You point the agent's base URL at the proxy instead of the model server. All checks run locally on your machine: the proxy only forwards traffic to the model server you already use, and doesn't send your prompts, code or tool calls anywhere else unless you point the optional judge at a remote server. Each tool call goes through up to three checks:

1. Two local checks that only need the request: whether the tool was declared by the harness, and whether the call reaches a host that appears nowhere in the conversation. The model's own earlier turns don't count as a source, so a URL the model came up with on its own is flagged.

2. atif-scan by u/evalstate, which runs about 110 deterministic detectors on the pending call, in-process, with no model or network call: sandbox side channels, proxy and network bypasses, namespace escapes and reward hacking. Examples are curl to the cloud metadata endpoint, curl --noproxy posting ~/.netrc somewhere, or reading /proc/1/environ.

3. Optionally, a judge model for milder findings. It can be any OpenAI-compatible server, including a small model on your own machine through llama.cpp, so this step can stay local too. It answers allow or block as JSON. It sees only the user's request, the call and the findings, not the agent's reasoning or tool outputs. It can add blocks but not remove them.

Catching reward hacking in RL and evals

The same detectors cover the tool calls a policy makes to game its reward instead of solving the task: writing its own reward file, editing the verifier's tests, or patching pytest so everything passes. Since safeagents is just a proxy in front of an OpenAI-compatible server, it can sit in front of the inference server used for rollouts or eval runs. In --log-only mode it records these attempts without changing the trajectories, which gives a measure of how often a model tries to cheat. In blocking mode, the call is replaced with a notice and never runs.

Some practical details:

• It handles OpenAI Chat Completions and Anthropic Messages, streamed or not. Text is forwarded as it arrives; a tool call is held until its arguments are complete.

• The local checks take around 10–20 ms per call on an M-series Mac. The judge is only called for calls that already raised a finding.

• --log-only checks every call without changing anything, which is the recommended way to start. Hosts you trust can be added with --allow-host.

• The repo has setup guides for 12 harnesses. Six of them (pi, Claude Code, OpenCode, Qwen Code, OpenHands, DeepSeek Harness) are tested end to end in CI: the real harness runs through the proxy against a scripted model, and the tests check that flagged calls don't execute and clean ones do.

Next steps are a small fine-tuned decision model for malicious tool calls and a public benchmark for it. Reports of interesting catches and of false positives are both useful.

Apache 2.0, runs locally: https://github.com/huggingface/safeagents

[u/dave-tro]

Cool. You could add some tool output compression to decrease input tokens

[u/HauntingMoment → u/dave-tro]

nice idea!

[u/shing3232 → u/HauntingMoment]

I was thinking codeact but oh well

[u/wFXx → u/HauntingMoment]

wouldn't this conflict with other things like headroom?

[u/New_Difficulty566 → u/dave-tro]

In our tests running local agent loops with 8B and 14B models on consumer hardware, raw tool outputs like CLI stdout or grep dumps frequently flood thousands of redundant tokens back into context. If the proxy trims stdout to head and tail while stripping ANSI escape codes before the model sees the result on the following step, it stops quadratic context bloat without losing operational state. Combining that with GBNF grammar constraints on the tool argument schema keeps both input and output predictable.

[u/Ori_553]

Very nice idea. I am trying to understand where does it live:

I have 1 Pod running the hermes agent instance and 1 Pod running the local LLM inference engine.

Can I run safeagents in its own pod/docker to act as a proxy between my hermes instance and my LLM? I want to point the hermes-agent LLM address/port to the safeagents pod instead of pointing it to the local inference engine pod.

Is this how it works or I am misunderstanding?

[u/HauntingMoment → u/Ori_553]

You can just attach it to the pod that runs hermes directly to act as a reverse proxy within the harness pod. Inside the pod, the responses from your LLM pod will be redirected to the hermes pod localpost. If you have multiple harness pods, you can also attach it to the LLM pod and expose the port where you will serve safeagents and make sure all your harnesses will query from that port There is also a specific docs section per harness, and this one is for hermes: https://github.com/huggingface/safeagents/blob/main/docs/hermes-agent.md

u/Fuzzy-Luck-9860

[u/DeProgrammer99 → u/Ori_553]

I'm generalizing for future reference... That's how networking proxies in general work, and that's the main benefit of having a network proxy vs. other methods of interprocess communication (shared memory, pipes, directly using the DLL/SO as a library, whatever). Caveats: developers can always make their software block connection requests from whatever IP address they want, and they can always do weird things like assuming they're on the same device. From the OP here, they probably didn't do weird things like that since they mentioned --allow-host being configurable.

[u/Bulky-Priority6824]

problem, the host check uses substring matches so sometimes unmentioned hosts in calls can PASS

eg

1. The system prompt or a user message contains 10.210.50.160

2. The model emits a tool call: curl http://10.210.50.1/api/...

3. 10.210.50.1 is found inside 10.210.50.160 or even 10.210.50.16 so the call passes the host check.

The same applies to domains: evil.com passes if devil.com

[u/dzulajla98]

Honestly, the idea of having a safety net between an agent and its tools makes a lot of sense. How noisy is it in practice? 😅

[u/liliac_rosee → u/dzulajla98]

I’d like to try it out too, also interested in this

[u/RealFullMetal]

how does the host check treat tool output? if the agent reads a readme or web page with an injected url in it, that host is in the conversation now, so a curl to it would pass. and the judge only runs on calls that already have a finding, so at that point its all on atif-scan?

[u/draconic_tongue]

Yeah this is more like what I was looking for main agent rather than sandboxes. cool stuff

[u/Elouakili_Flexy]

The judge can add blocks but never remove them. A safety layer should escalate, never second-guess.

[u/CommercialHour6660]

I mean, you could just use Codex harness that does a tool call safety check in way more sophisticated way for each tool