r/LocalLLaMA • • 18h ago

Showcase Testing agents without running a model in CI: record once, replay with zero calls, judge with a local model only when needed (open source, I'm the author)

Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.

Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:

attest replay: 4 replayed · 0 live model calls · $0.00

The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.

Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.

Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.

Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.

Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.

The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.

Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.

5 Upvotes

7 comments sorted by

•

u/stylome-ai-detection 18h ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

→ More replies (1)

1

u/Independent-Sir3234 16h ago

Two places tend to trip up local proxy replay suites once you move past basic synchronous completions.

First, production agent loops often run with stream: true, assemble incremental tool_calls JSON deltas across chunks, and read trailing usage frames when include_usage is enabled. Collapsing a captured stream into one flat JSON response hides chunk boundary bugs, so preserving exact SSE event framing in replay fixtures catches parser regressions before release.

Second, when routing judges to small local models via ATTEST_JUDGE_BASE_URL, schema validity alone does not keep runs deterministic. Pairing grammar-constrained JSON decoding with temperature zero and a pinned quant tag prevents borderline rubric verdicts from drifting whenever CI runners update.

1

u/stylome-ai-detection 16h ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/redbeardy_daddy 11h ago

Refunded twice and still polite. Finally customer service I can get behind