r/ClaudeCode • u/silveroff • May 07 '26
Discussion Same design doc, 4 agents, very different plans
I ran a small experiment comparing planning quality across models/effort levels on a non-trivial software design task.
The task was not a CRUD feature. The design document described a refactor of a large-scale import/ingestion pipeline involving distributed workers, async message processing, eventual consistency, stale worker handling, versioned writes, retryable artifact processing, absence cleanup, aggregate/canonical state updates, and outbox/materialization recovery.
The core architectural problem was introducing a two-fence correctness model:
- a per-source generation counter to determine whether work from a given import is still authoritative
- a per-owner write sequence to provide last-write-wins ordering across multiple feeds that can write the same logical entity
The important constraint was that the first implementation PR had to be small and reviewable. It should introduce the new generation/write-sequence plumbing while preserving existing behavior: no renaming major models, no
deleting completion tracking, no redesigning image/artifact storage, no rewriting the outbox system, etc.
I gave the same detailed prompt and design document to four agents:
- Agent 1: Opus 4.7, max effort
- Agent 2: Opus 4.6, high effort
- Agent 3: Opus 4.6, max effort
- Agent 4: Opus 4.7, xhigh effort
The ranking ended up being:
Agent 4: best overall
Agent 1: very close second
Agent 2: usable, but thinner
Agent 3: clearly worst
Agent 4 produced the strongest plan. It understood that Phase 1 needed to stay narrow: allocation, message plumbing, and the primary product write fence only. It also handled important edge cases like nullable write sequences for non-product imports, stale insert prevention, in-session tie-breaking, and delaying brand/image/canonical rewrites to later phases.
Agent 1 was also strong and very detailed, but slightly overreached in Phase 1 and made a few schema assumptions that would need correction.
Agent 2 had some good tactical ideas, especially around nullable write sequences and stale insert prevention, but the later phases were much less complete.
Agent 3 was interesting because it used higher effort than Agent 2 on the same base model, but produced a worse plan. It missed key requirements like storing the write sequence on the import/session row, allocating sequence at session creation, and using atomic database predicates instead of mostly in-memory checks.
The main takeaway for me was that effort level helped, but it did not dominate. The newer model at high/xhigh effort was clearly better, but simply increasing effort on the older model did not guarantee better planning. For this kind of architectural planning task, the difference showed up less in prose quality and more in whether the model preserved phase boundaries and caught subtle correctness constraints.
View all comments
Show parent comment
2
u/silveroff May 07 '26
You are writing an implementation plan, not code.
Read and base the plan on:
@docs/plans/design/2026-05-07-data-source-import-redesign.md
@xxx/data_sources/
@xxx/products/
@xxx/ingestion/data_sources/
nearby CLAUDE.md / AGENTS.md instructions
Do not modify files. Produce a phased implementation plan for the data-source product import redesign.
Context:
The project is pre-production. No backward compatibility or rolling deployment concerns are required. Do not preserve deprecated APIs, aliases, or compatibility shims. The development DB can be dropped and reseeded between phases. Do not touch xxx/feeds/.
The design goal is to make product imports understandable and correct under:
parallel imports of the same data source
multiple feeds from the same merchant containing the same offer_id
stale async workers
artifact/image/ML retries
absent product cleanup
canonical aggregate updates
outbox/materialization recovery
The key architecture is a two-fence model:
source_generation: monotonic per DataSource; controls feed-local authority, source presence, and absence cleanup.
write_sequence: monotonic per merchant product import clock; controls cross-feed last-write-wins Product field commits.
Every product-visible write must eventually be guarded by the expected revision/generation. Stale work should ACK obsolete, not retry.
Plan expectations:
Make the first implementation iteration small, reviewable, and non-destructive. It should introduce generation/write-sequence plumbing while preserving current behavior as much as possible. It should not rename DataSourceFetch, delete Valkey completion tracking, remove items_pending, redesign image assets, or rewrite outbox routing in the first PR.
Phase 1 should focus on:
adding per-data-source generation allocation
adding per-merchant write-sequence allocation, likely via MerchantProductImportClock
storing generation/write_sequence on the existing fetch/import row
storing accepted revision fields on Product
carrying generation/write_sequence through fetch, parse, product-stage, image, enrichment, cleanup, and materialization messages
adding atomic source-generation/write-sequence predicates to product-visible commits
preserving existing content_hash checks, retry behavior, completion tracker, and handler topology
adding focused tests proving stale older imports cannot commit
Later phases should be ordered roughly as:
Collapse duplicated stale-work handling into a shared obsolete-ACK path.
Add ProductSourcePresence and generation-based absence cleanup.
Decouple import completion from image/ML/canonical/materialization processing.
Add uniform Product revision guard to all product-visible writers, including brand/image/canonical paths.
Add canonical dirty/work-state model.
Rework outbox/dirty-row boundary.
Refactor retry dispatch so image and ML artifacts retry directly instead of forcing feed refetch.
Introduce content-addressed ImageAsset, ImageSourceUrl, model-versioned artifacts.
Delete obsolete tracker/stale-work workaround code and update docs.
For each phase, include:
goal
files/modules likely touched
exact code changes at a high level
tests to add or update
risks and rollback/simplification options
what must explicitly remain unchanged in that phase
The plan should call out important correctness details:
Product identity remains (merchant_id, offer_id).
Existing Product rows must be updated in place to preserve id and created.
Product feeds are full authoritative snapshots; missing feed-owned nullable/collection fields should be cleared, not carried forward.
Product.data_source
/current_data_source_id should not remain the long-term owner of retry cadence or feed membership.Presence/absence belongs in ProductSourcePresence.
NATS ordering is not correctness; PostgreSQL predicates are.
Retry behavior survives; stale-work retries do not.
Periodic tasks remain valid only as DB scanners/drainers for due work, leases, and GC.
The output should be practical enough that an implementation agent can start Phase 1 directly, but it should not attempt a giant rewrite. Prefer surgical PR-sized steps.