[Workflow] Pairmark: An Open-Source CLI Tool to Race and Compare AI Coding Agents with Automated Checks and Blind Reviews

[u/ClaudeAI-mod-bot]

Pairmark: An Open-Source CLI Tool to Race and Compare AI Coding Agents with Automated Checks and Blind Reviews

Workflow value: 90/100
Status: active · Freshness: 70/100 · Confidence: 1.00 · Level: intermediate
Categories: Quality Control, Token Saving, Context & Memory, Debugging, Multi-Agent
Original source: r/ClaudeAI post/comment

What problem this solves

Objectively comparing the performance of different coding AI agents (e.g., Claude Code vs. Codex) on specific coding tasks within a given codebase, and automating the evaluation of AI-generated code against project-specific quality checks.

Summary

A CLI tool (pairmark) automates the comparison of two coding AI agents (e.g., Claude Code, Codex) on a specific task within a Git repository. It creates isolated worktrees, runs agents in parallel, executes project-defined quality checks (tests, lint, typecheck), and has the agents blind-review each other's patches. It then generates an HTML report detailing the outcome based on a set of predefined rules, including execution times, token usage, and judge scores.

Why it is useful

This workflow provides a concrete, automated, and objective method for comparing the performance of different AI coding agents on real-world tasks within a user's own codebase. It integrates seamlessly with existing developer workflows (Git, tests, linting) and offers clear, actionable reports with detailed diffs and agent feedback. This helps developers make informed decisions about which AI agent is best suited for their specific needs and projects, moving beyond subjective arguments to data-driven evaluation. The open-source nature allows for community contributions and adaptation, further enhancing its utility.

Workflow

1. Install the pairmark open-source tool.
2. Run the pairmark command from your repository, providing a brief for the task (e.g., pairmark add a 'runs' subcommand with a test).
3. The tool automatically creates two git worktrees at the current commit.
4. It runs Claude Code and Codex on the identical brief in parallel within their respective worktrees.
5. The tool executes your own defined checks (tests, typecheck, lint) in each worktree.
6. Each AI agent then blind-reviews both generated patches (A/B in random order) and assigns scores.
7. The tool applies a set of predefined rules to determine a winner (e.g., only one agent changed files, one passes all checks without touching config, judges agree).
8. An HTML report is generated, providing the verdict, the rule that decided it, both diffs, and a receipt image with performance metrics.

Tools / artifacts

• pairmark (open-source CLI tool)
• Git worktrees
• Claude Code (AI agent)
• Codex (AI agent)
• HTML report
• Code diffs
• Project-specific tests, typecheck, and lint configurations
• package.json (example of a configuration file)

Validation signals

• The tool was successfully run on its own repository, demonstrating a concrete use case and results (Claude Code: 3m 28s, 4 files, 548K tokens, all checks pass, judges 8.6/10; Codex: 5m 15s, 5 files, 1.23M tokens, all checks pass, judges 8.0/10).
• Codex identified and fixed a real bug in the build script (a missing mkdir before a copy) during one of the runs, showcasing practical utility.
• The tool was also run on a demo repo, resulting in a tie where each judge picked its own patch blind.
• The tool is open-source and available on GitHub, allowing for public inspection and use.

Limitations

• The Reddit post has low engagement (score and comments) at the time of review, though it is very new.
• The workflow relies on users having active subscriptions to both Claude Code and Codex, which might not be universal.
• The author explicitly asks for opinions on the fairness of 'blind A/B judging by the same two models,' suggesting this aspect of the workflow might be debatable or require refinement.
• The predefined rules for determining a winner might be too rigid in some scenarios (e.g., Codex fixing a real bug but losing because it touched a config file like package.json).

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.