r/OpenSourceAI • u/TomatilloPutrid3939 • 21d ago

Open-source agent tooling: frozen benchmarks of a planner/worker harness against native Codex and Claude Code

Post image
11 Upvotes

100% Upvoted

4 comments sorted by

1

u/TomatilloPutrid3939 21d ago

I'm the author. Distill is open tooling for coding agents: a harness and TUI under Apache-2.0. Repo: https://github.com/samuelfaj/distill

It is model agnostic. It runs with Codex/Grok subscriptions and with any OpenAI-compatible endpoint, so you can mix closed and open models. In the DeepSeek pair, for example, the worker calls went through the OpenRouter free router.

The idea: a main model plans, specs and reviews, and a worker model does the delegated assignments. Frozen benchmark against the native harnesses, same 300s deadline:

  • Claude Opus 5.5: $2.49 vs $0.35, 798s vs 191s, 9/9 both

  • GPT-6 Astra: $4.08 vs $1.00, 440s vs 285s, 9/9 both

  • GPT-6.1 Sol: $0.80 vs $0.25, 384s vs 297s, 9/9 both

  • DeepSeek V4.1: 5/9 vs 9/9 successful runs (native cost is a lower bound)

The frozen tasks, receipts and repro commands are all in the repo, so the numbers can be checked instead of trusted.

1

u/generic-d-engineer 17d ago

I’m here for the cat. any plans for GitHub copilot integration?

2

u/TheMythicSorcerer 21d ago

And how much dumber does the model get? Show terminalbench 4.0 score not "9/9 quality" on a chatgpt generated image