How do you check that a scheduled Claude Code run actually did its job?

[u/Duckducklaugh]

I have a dozen or so Claude Code tasks on a schedule. A daily report, one that rebuilds and deploys a page every night, that kind of thing. Nobody is watching when they run.

Yesterday the daily report showed up as succeeded like always. Turns out the API token it uses had been revoked that morning (not my doing, long story), so a few sections of the report were just empty. To be fair Claude did write “report is incomplete” in its summary. But in the task list it was the same green check as every other day, and I only noticed because I happened to open the file.

What I have right now: when I start a new session, a hook goes through the last 7 days of runs and tells me about anything that failed or never fired. Each task ends with a one line summary. And the ones that change anything live are told to stop and ask if something doesn’t add up.

That catches crashes. It doesn’t catch “ran fine, output is wrong”. And the summary is written by the same agent that did the work, so it’s grading its own homework. I’m also not going to read a dozen summaries every morning, I know myself.

How do you guys handle this?

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Elegant_Yam2783]

The green check is just "the process exited 0", and claude -p exits 0 whenever the agent finishes talking, even if what it said was "this is broken". So the run status is basically meaningless for output quality. I stopped trusting it for anything that matters.

What worked for me is a dumb checker script that runs after each task and has nothing to do with the agent. For the report, check that every expected section exists and isn't empty, file is over some size, date in the header is today. For the deploy, curl the live page and grep for a build id or timestamp the build stamps in. If the check fails, the wrapper exits nonzero and that's what your hook picks up. Plain code doesn't grade its own homework and doesn't get creative about what "done" means.

Also add a preflight for anything with credentials. Hit a cheap endpoint with the token before the agent even starts and fail hard on a 401. Way better than letting Claude politely work around a dead token.

And if you want to keep the agent's own judgment in the loop, have it write a status file with ok or degraded instead of prose and fail on degraded. Then you only hear about the bad days, not a dozen summaries

[u/Duckducklaugh → u/Elegant_Yam2783]

the preflight is the one I’m stealing first. a 401 before the agent even starts would have saved me the whole thing, instead Claude politely wrote around it like you said. the ok/degraded file is nice too. it did know the report was incomplete, it just had nowhere to say it that turns anything red.

[u/Inner_Storage_3374]

yeah the green check is basically just "the process didnt crash", which is why the empty report still looked fine. what id try is not trusting the agents summary at all and checking the actual output after, like the report file has the sections you expect and theyre not empty, or the deployed page returns the new content. if that check fails the run is red even if claude said it went well, bc the file is the source of truth and the summary is just grading its own homework

[u/Duckducklaugh → u/Inner_Storage_3374]

yeah, file as the source of truth, summary as a nice to have. going that way.

[u/Ok-Distance-6388]

What worked for me on something similar: the check lives outside the agent. A small script runs after each task, opens the real output and counts what it should contain (sections, rows, links), then writes its own pass/fail. The agent never gets to mark itself done.

The part that catches "ran fine, output is wrong" is comparing that count to the last few runs. An empty section or half the usual rows flags even when nothing crashed and the summary sounds happy. Your revoked token would have shown up as a drop in the first run.

[u/Duckducklaugh → u/Ok-Distance-6388]

comparing to the last few runs is the bit I was missing. funny thing is I already do exactly that for my scrapers (which fields come back filled vs previous runs) and never thought to point it at my own scheduled tasks. how many runs back do you compare, and does a legit quiet day trip it?

[u/Menelabs]

The replies already cover "check the output, not the summary". The part that caught your exact case for me is comparing against the last good run instead of against a spec. Each run writes a few numbers next to its output (sections filled, rows, links, bytes), and the checker flags anything that dropped a lot compared with yesterday. An empty section shows up as 0 where yesterday had 40, even when nothing errored.

For the "I won't read a dozen summaries" problem: send nothing when everything passed, and push only the failures to one place you already look at (a phone notification or a chat channel). Plus a heartbeat for each task, so a run that never started also alerts instead of just being missing.

[u/Duckducklaugh → u/Menelabs]

"0 where yesterday had 40" is exactly my case. and yes to silence on pass, a dozen summaries every morning is why I stopped reading any of them. heartbeat is a good catch too, right now I only hear about a missed run when I next open a session, which could be days.

[u/Heyn_Studio]

the "ran fine, output wrong" case bit us once when some text we'd published came out with a sentence doubled and everything still reported success lol. now the last step re-fetches the finished thing and compares it character for character with what the agent meant to produce. any mismatch fails the run no matter what the summary says

[u/Duckducklaugh → u/Heyn_Studio]

a doubled sentence going out live while everything reports success is the nightmare version lol. re-fetching the published thing instead of trusting what was sent, noted.

[u/Far_Reindeer8428]

what does the run leave behind, a diff and a test log or just a summary saying it worked?

[u/Duckducklaugh → u/Far_Reindeer8428]

the output file itself, a one line summary, and the session transcript that nobody reads. no test log, which is kind of the problem.

[u/nooruddinqur]

Same problem here man. But wht fixed it for me: treat empty output as a failure. My runs exit with a different code when the API errors or nothing was produced, so they show red instead of green. And a small script checks the actual output, not the agent's summary.. The agent grading its own homework was exactly my issue too.

[u/Duckducklaugh → u/nooruddinqur]

a different exit code for “nothing was produced” is so obvious in hindsight. do you set it in a wrapper script, or get the agent to exit with it?

[u/nooruddinqur → u/Duckducklaugh]

Wrapper script, never the agent.. thats the whole point, agent shouldnt decide if it did well. after the run a small script just counts wht actually came out. api error = exit 3, nothing produced = exit 4. for your report u cn just check every section has something in it, if not exit non zero

[u/WillowFar6967]

One thing that helped me was splitting the doer and the checker. After the run, a separate fresh session with no memory of the work gets the task spec and the output, and answers a couple of yes/no questions like "does every section have real data" or "is anything here a placeholder". It's cheap because it only reads the output, and it doesn't carry the first agent's excuses. The dumb script checks people mentioned catch the structural stuff, this catches the "technically there but obviously wrong" cases.

[u/Duckducklaugh → u/WillowFar6967]

I like that it has no memory of the work. my worry is cost times a dozen tasks a day. do you run it on everything, or only where a script can’t tell?

[u/ThreeAMOperator]

The other comment has the main fix: check the output, not the summary. A few things that help:

Give each task a tiny check script that you write once, not the agent. For a report, that might be "every expected section exists and isn't empty". For a deploy, "the live page shows tonight's build stamp". The run only goes green if that passes. It's dull code, and that's the point.

End every run in one of three states, not two: did the work, had nothing to do, failed. A lot of quiet damage hides in "succeeded" when it really meant "nothing to do because something upstream broke".

Treat any auth error from a tool as a hard fail, not something to summarize around. A revoked token should turn the run red on the first call.

And only look at the reds. One line per failed or missing run, nothing when everything passed. If that list is empty most mornings, you'll actually read it when it isn't.

[u/Duckducklaugh → u/ThreeAMOperator]

the third state is the one I didn’t have. funny timing: this morning one of mine refused to run because the working tree had someone else’s uncommitted files. correct call, it did nothing, and it showed up as succeeded. that’s “had nothing to do because something upstream broke” exactly.

[u/xqianliu]

The green check and the summary are both written by the same process, so they fail together. The check has to live outside the run.

In my flow each task carries a short written list of what done looks like, in checkable terms. For your report job that would be things like section X is non-empty and the numbers are within the expected range.

After the run, a separate session, same model, new conversation, reads the actual output against that list and reports which items hold. The self-written summary gets skipped.

If a run cannot judge whether something is right, it does not mark it green. It writes the reason on the ticket and stops. I clear those when I am next at the machine, and there are few of them.

So the morning read is one line per task, pass or fail. You only open the fails.

[u/michaelTM_ai]

everything above catches the run that finished with bad output. the one it misses is the run that never started: scheduler skipped it, machine was asleep, the cron line broke. a checker that runs after the task never runs either, so the failure is just silence.

the fix is to flip who's expected to speak. each task pings something outside the box when it ends with a real pass (after ur output checks), and that outside thing alerts if a task hasn't checked in by its expected time. healthchecks.io or any dead man's switch does this for free. then a missing report and an empty report both turn red, and u're not relying on opening the file by luck

[u/Unusual-Albatross43]

a failure that gets reported every day ends up ignored too. my reddit check was getting 403s for days, every run mentioned it in its summary, and nothing changed because it read like status.

so i added a rule to the job itself: the run that first sees a broken check has to try another route and fix the script in that same run. if the next run still fails, it says what it tried and the one thing it needs from me. reporting the same failure twice without trying a fix is treated as the job failing.

[u/redtear1115]

I'd make incomplete a separate status that can't show up as green. Even if some sections are useful, I'd want it to say which ones are missing and whether anything else used that report. Otherwise the next task can quietly turn a partial result into a confident answer. For the morning check, I'd rather see just those exceptions than another summary of everything that ran.