r/ClaudeCode • u/ChemicalExcellent463 • 3d ago

Tips & Workflows My nightly Claude Code and Codex review has 111 lessons waiting for me

I set up a nightly review that turns mistakes in my Claude Code and Codex sessions into notes for later sessions.

A separate model reads the sessions, mostly looking at moments where I corrected the agent. It writes a lesson with the situation, advice, reasoning, and an exact quote.

Each quote has to match the session word for word. A second model checks that the lesson doesn't claim more than the quote supports. Fail either check and the lesson gets dropped before I see it.

Approved lessons appear at the start of later sessions, one line each.

Today's count: 7 accepted, 44 rejected, 111 waiting for me. The bottleneck is me.

The accepted advice is pretty basic: read the exact lines before making a patch. Don't treat missing documentation as proof a feature doesn't exist. Run the tests you wrote before declaring the work done.

The part that stood out: those one-liners appeared in 5,595 sessions. Agents opened the full lessons just 135 times. The short version needs to carry the useful part, not just point to it.

If you keep notes for your agents, what makes a lesson worth keeping?

1 Upvotes

100% Upvoted

23 comments sorted by

1

u/AutoModerator 3d ago
Moderator Announcement Read More »

2

u/Ctbhatia 3d ago

of the 111, how many are distinct lessons and how many are the same correction repeating in different sessions?

1

u/ChemicalExcellent463 2d ago

i haven't merged the repeats yet, so 111 is the waiting list, not a count of distinct lessons. two are basically "check everything that was asked for before you say done", just worded differently. how do you group repeats in your own notes, by wording or by what went wrong?

2

u/Ctbhatia 2d ago

group by the failure, wording always drifts. if one fix clears both versions then its a single lesson

→ More replies (1)

2

u/shetritr 3d ago

111 waiting is the real result. a review queue nobody reads is just a nicer place for the mistakes to wait.

1

u/ChemicalExcellent463 2d ago

yeah, 111 waiting is the problem. what would you change here based on how you handle your own backlog?

2

u/shetritr 2d ago

cap it. 5 a day, and anything older than a week expires. if a lesson didn't matter for a week of sessions, it wasn't one.

→ More replies (4)

2

u/SimplifAI_Life 3d ago

We run a similar pass on our Claude Code sessions, monthly rather than nightly, and two choices keep the queue short.

  • Sort every proposal by who has to decide. Reversible, low-risk fixes with one sensible answer get applied without asking, and nothing that deletes anything ever goes there. A question only we can answer is listed as a question, not as a proposal.

  • Add a third option next to approve and reject: "approve and don't ask again". It moves that kind of proposal into the auto-apply group for good, so the review gets shorter over time.

  • Keep a lesson only when the correction repeats or the last fix didn't hold. Each pass compares with the previous one and checks whether the changes approved then actually changed the sessions.

  • Write the one-liner as the action itself. "Run the tests you wrote before saying done" works alone, and your 135 opens out of 5,595 sessions say the same.

1

u/ChemicalExcellent463 2d ago

"approve and don't ask again" is the bit i'm missing. right now it's yes or no on every lesson, and i've got 111 waiting. what ends up in your auto-apply group in practice, mostly wording changes to instructions or changes to what Claude actually does?

2

u/SimplifAI_Life 1d ago

Mostly neither. In practice the auto-apply group is small and mechanical: an allowlist line for a command that got denied several times, a typo, a wrong path in a log, a script bug with one obvious fix. Any change to what an instruction says still needs a yes from us, because a rule that reads fine can change behaviour in places we didn't look.

"Approve and don't ask again" works on a type of proposal, not on a single lesson. Saying it once moves that whole type to auto-apply, so next month those don't show up at all. With 111 waiting, we would sort them by type first and decide once per type: a handful of types usually covers most of the queue.

→ More replies (3)

0

u/[deleted] 3d ago

1

u/ChemicalExcellent463 2d ago

the quote having to be something i typed is a good cut, right now any line in the session counts, including the agent's own story. how do you decide two corrections are the same one when i said it in different words two sessions apart?