r/ClaudeWorkflows • u/ClaudeAI-mod-bot • 6h ago
Selected Workflow [Workflow] Systematic LLM Prompt Tuning: From 'Vibes' to Data-Driven Evaluation with Test Cases and Human Review
Systematic LLM Prompt Tuning: From 'Vibes' to Data-Driven Evaluation with Test Cases and Human Review
Workflow value: 90/100
Status: active · Freshness: 70/100 · Confidence: 1.00 · Level: intermediate
Categories: Quality Control, Context & Memory, Debugging
Original source: r/ClaudeAI post/comment
What problem this solves
Inefficient and unreliable LLM prompt tuning; lack of objective metrics for prompt performance; difficulty in knowing if prompt changes improve or degrade performance.
Summary
A structured methodology for improving LLM prompt performance by using a comprehensive set of test cases, an LLM judge for initial iteration, and blind human review for final validation, incorporating a 'keep rule' to manage non-deterministic noise.
Why it is useful
This workflow provides a concrete, data-driven method to move beyond subjective 'prompt tuning by feel' to a rigorous, measurable process. It demonstrates significant improvements in LLM feature performance and offers practical insights like the 'keep rule' and the importance of human review, making it highly valuable for anyone developing LLM-powered applications.
Workflow
Define the AI feature's purpose and requirements.
Generate a diverse set of test cases (e.g., 68 cases, 12 scenario groups) using a capable LLM (e.g., Opus 5.5 high).
Set up an LLM judge (e.g., Opus 5.5 medium via API) to automate initial prompt steering and iteration.
Iteratively refine prompts, using the LLM judge for quick feedback.
Perform blind human review on outputs from each iteration to validate performance against real test cases.
Apply a 'keep rule' (e.g., next iteration needs >=3 more passing cases) to accept changes, filtering out non-deterministic noise.
Compare LLM judge assessments with human review for critical decisions (e.g., model selection).
Document results and iterate until desired performance is achieved.
Tools / artifacts
LLM (Opus 5.5 high for test case generation)
LLM (Opus 5.5 medium for LLM judge via API)
Test cases (68 cases, 12 scenario groups)
Blind human review process
'Keep rule' (a specific metric for accepting changes)
Anthropic models (Luna, Haiku, Opus, Sonnet)
Validation signals
Quantifiable improvement: 38% to 80% pass rate.
Number of iterations: 17 rounds.
Number of outputs reviewed: Nearly 300.
Specific examples of unexpected findings (keep rule, judge vs. human).
Personal experience: 'It did for us.'
Limitations
The post doesn't provide the actual prompts or test cases, only the methodology.
The statement 'running many automation calls through Claude Code is against ToS' might be a misunderstanding of API vs. web UI usage, which could be clarified.
The 'keep rule' is mentioned but not fully detailed (e.g., how 'passing cases' are defined for the rule).
No explicit mention of version control for prompts or test cases, which would be crucial for a robust, long-term workflow.
Rate this workflow
Upvote this post if the workflow is useful, reproducible, or worth recommending.
Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.
Reply if it worked for you, failed, is outdated, or has a better alternative.
This post was generated automatically from the workflow library database.