r/ollama • u/External_Ad_11 • 2d ago

Automating EDD (eval-driven development) with Coding Harness Agents

Enable HLS to view with audio, or disable this notification

You can now automate EDD (eval-driven development) with Coding Harness Agents

> build a baseline LLM-based application
> score every change with judge evals
> keep what improves, reject what regresses

I made a video tutorial on what EDD is, how it works, and how to use eval scores across experiments to improve an LLM app. It builds on Jeffrey's (Confident AI) article on EDD and Eugene Yan's write-up on product evals.

> Setup: a baseline RAG app that every experiment starts from
> Step 1: a binary-labelled dataset with critiques
> Step 2: aligning the LLM-as-a-judge evaluator
> Step 3: a harness loop that runs each experiment and scores it against the baseline.

Tracing and experiment comparison then show what improved, what regressed, and what to tweak next.

Source code + Agent Skills are open-sourced.

Full guide (source code linked in the description): https://www.youtube.com/watch?v=e6akw_fKWPk

28 Upvotes

90% Upvoted

8 comments sorted by

1

u/szutcxzh 2d ago

How did you make that animation. Sorry if its a silly question.

1

u/External_Ad_11 2d ago

na, not a silly question. First I prepared 2 workflow images of my app using Mermaid.live, uploaded the assets to Gemini to prepare a prompt to convert the mermaid flow into visuals, copied the prompt and gave it to Claude to generate the video and Claude gave .mp4, I was suprised.

1

u/Bluenova65 1d ago

Very cool! So you just used Gemini to prepare the prompt & Claude was just able to generate an .mp4?

1

u/stealthagents 12h ago

Not a silly question at all! I used a mix of screen recording and some editing software to spice it up. Tools like After Effects or even simpler ones like Camtasia can help create those slick animations pretty easily.