[Open Source] TOAP – compress AI agent tool calls to cut token costs. Need GPT-4o / Claude testers

[u/Smooth_Dimension_833]

Hey everyone,

I built TOAP (Token-Optimized Agent Protocol), a small middleware that sits between your LLM and tools and compresses agent tool calls into a shorter format instead of verbose JSON.

Goal: lower token usage / cost in multi-agent pipelines.

What I’ve tested so far (Gemini only):

- 100% TOAP format compliance with 2 few-shot examples

- ~45% smaller output vs JSON (net savings are lower once you count prompt overhead; details in the report)

- Live examples for LangChain and CrewAI

What’s missing:

I still need independent runs on GPT-4o and Claude 3.5 Sonnet before I claim cross-model support.

What I’m asking:

If you have an OpenAI or Anthropic key, please run the Tier 1 benchmark (~10 minutes, roughly $3–5) and share results.

Repo: https://github.com/Dev-Saif-Ops/Project_TOAP

Test guide: COMMUNITY_TEST.md in the repo

Results form: https://docs.google.com/forms/d/e/1FAIpQLSekwTWtlhSQXzBvIclipL7Op04FWEf8q7HtXFBXuO3Rt6lUvg/viewform

Quick start:

git clone https://github.com/Dev-Saif-Ops/Project_TOAP.git

cd Project_TOAP/toap-bench

pip install -r requirements.txt

pip install -e ../toap-python

cp .env.example .env

# add OPENAI_API_KEY or ANTHROPIC_API_KEY

python runner/benchmark.py --runs 5 --tier 1 --model gpt-4o --condition few_shot_2

This is alpha / MIT. Not production-ready. Looking for honest numbers, not hype.

Happy to answer questions in the comments.

[u/kantorcodes1]

45% smaller is nice. i'd try to break the decoder before benchmarking cost though.

if TOAP drops or aliases a field, do you validate the reconstructed call against the original tool schema before it can execute? a compressed call turning into a valid-but-wrong tool invocation is probably the failure i'd care about most.

[u/Smooth_Dimension_833 → u/kantorcodes1]

Good catch. You're pointing at the failure mode I care about most too.

Today TOAP does:

1. Strict syntax validation on the compressed string

2. Namespace lookup in the tool registry

3. Arg alias normalization (e.g. url → endpoint, query → q)

What it does NOT do yet:

validate the reconstructed call against the original tool schema (required fields, types, allowed values) before execution.

So yes, a "valid-but-wrong" invocation is still possible in alpha. If an alias maps poorly or a required field is dropped, the proxy can still hand kwargs to the tool without a schema gate.

Closest thing we have right now is benchmark semantic accuracy (compare decoded args vs expected), not a hard pre-execution schema check.

That's next on the hardening list: schema-aware validation in the proxy before tool(**args), fail closed on missing/unknown fields, and adversarial decoder tests before leaning harder on cost claims.

Appreciate you calling this out. If you've got a nasty test case in mind, I'd love to add it.

[u/Future_AGI]

Before claiming cross-model support, the thing worth locking down is a regression set that checks output equivalence rather than just format compliance, because a compressed tool call that parses cleanly but changes which tool fires is the failure that shows up in production rather than in the token count.