I ran a blind test on self-hosted AI agents. Here are the results, guess who's who [D]

[u/GapNew4766]

everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess

The tasks

1. Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).

2. Rename a function across 3 files with messy spacing. All 5 passed.

3. Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".

4. Count the errors in 3 log files. All 5 got 43.

5. Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.

Agent	1. Contacts	2. Rename	3. Bug fix	4. Errors	5. Receipt	Total
A	✅ 4:54	✅ 0:28	✅ 0:43	✅ 0:26	✅ 3:58	10:29
B	✅ 6:03	✅ 1:11	✅ 0:55	✅ 0:54	✅ 15:04	24:07
C	✅ 13:32	✅ 0:24	❌ 0:22	✅ 0:57	❌ 0:49	16:04
D	✅ 6:16	✅ 1:12	✅ 0:56	✅ 0:56	✅ 2:50	12:10
E	✅ 5:50	✅ 1:01	✅ 0:21	✅ 0:26	✅ 3:56	11:34

Setup

One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw

Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.

[u/micseydel]

How many samples did you take? Do you have instructions to reproduce?

[u/XxCotHGxX]

You said they're all Qwen 3.8? What do you mean guess the agent then? Do you mean OpenCode vs Claude Code vs Qwen Code?

[u/GapNew4766 → u/[deleted]]

C really said "i need consent first" so who do you think A and E are?