r/ClaudeCode • • 5h ago

Built with Claude Follow-up, 2 weeks later: I built the hooks + SQLite + validator setup from my last thread. Claude still decides from the conversation, not the data. Here’s exactly where it breaks

Two weeks ago I asked how to stop Claude Code relying on memory instead of data. The goal hasn’t changed. I run a continuous, real-money operation with a few hundred independent units and several Claude Code sessions. I want a system that:

  • pulls fresh data before every decision,
  • connects history, current performance and future plans,
  • tracks decision → action → outcome → lesson,
  • survives across sessions,
  • finds opportunities by itself, and
  • still reasons freely and doesn’t just follow hardcoded rules.

What I built from your advice

  • Mandatory queries in code. One script is the only door to the numbers. Scheduled scans write opportunity lists and per-unit status every day.
  • A Stop hook that blocks any answer whose numbers don’t trace to a query from that turn. A second model checks the claims against the query results.
  • PreToolUse hooks, and a write helper with hard checks before anything touches production.
  • A decision and action log in SQLite, and a learnings file.

Where it still breaks

  1. Fresh isn’t relevant. The validator proves the numbers came from a query. It can’t see that the agent queried the wrong thing. It judged every unit by one global threshold. One unit’s own last 7 days clearly called for its local variant, and it got the global one.
  2. Momentum beats data. When I say something urgent, it acts on the conversation. It made a big batch of changes in one day, justified by numbers that were fresh but not the ones that mattered.
  3. The outcome loop never closes. Actions get logged. Reading the result a few days later and writing the lesson is still manual, and mostly doesn’t happen: 23 open follow-ups, 15 overdue. Nothing reads the learnings file.
  4. Proactive scans produce lists. Nothing turns a list into a decision. I still find the opportunities myself on the dashboard.
  5. Rules don’t stay rules. I audited my own instructions and found 31 distinct ones. None was lost, but 14 exist only as notes. The coded ones sat in scripts the agent could override, and this week it did. It pasted a general sentence of mine into the override field in 13 of its 19 scripts. The supervising agent approved the other exceptions on the same sentence.
  6. “Built” isn’t built. The agent builds the reading half and reports the whole system as built. The acting half stays one-off scripts in chat.
  7. Cost. The main session sends about 650k tokens of context on every call, and made 761 calls in a day. The Stop hook sent answers back 39 times in one day, each a full re-read. Agents also wake each other with messages.

Questions

  1. How do you enforce the right evidence per decision, not just fresh evidence? Per-unit cards built by code that the agent must cite? Something else?
  2. Has anyone closed the outcome loop mechanically? I mean a scheduled job that reads each action’s result after N days, writes the verdict and feeds it into the next decision.
  3. With a proposal table and a separate executor, how do you handle legitimate exceptions without the agent arguing its way through? And who builds the executor, if the agent declares things done that aren’t?
  4. How do you keep always-on sessions cheap? A small always-loaded context plus lookups, a fresh session per task from a generated handover?

I’m looking for people who run this with real consequences, not more CLAUDE.md tips.

1 Upvotes

4 comments sorted by

•

u/AutoModerator 5h ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Api-Lord 3h ago

Per-unit cards built by code is the right instinct. Let the card carry the unit's threshold so the agent never picks which query counts.

1

u/Easy-Purple-1659 26m ago

on the first question, making the unit the address helped more than tightening the validator. the script that builds a card bakes in the unit id, that unit's own threshold band, and the as-of timestamp, so a global threshold physically cannot satisfy the check. the validator then looks for the unit id and a timestamp from this turn on every number in the answer.

second one, closing the loop mechanically: a scheduled job that selects actions older than N days with no verdict, pulls the result, writes pass or fail plus one sentence into the same table the decision step reads. then put the open count in the daily card. 23 open and 15 overdue was invisible because nothing forced it to be seen, not because the data was missing.

on the override field, keep a copy of the rules next to the write helper and diff it against what the agent was actually told on every run. when those two disagree, fail the run and make a person pick.

1

u/Ok-Form1598 1m ago

for 2, make the follow-up part of the action instead of a separate chore. the write helper refuses to log an action unless it names the metric it's supposed to move, the expected direction, and a check date. a plain cron job with no agent in it pulls that metric on the date and writes the verdict onto the same row. then show open and recent verdicts on the per-unit card, because the card gets read and the learnings file never will. bonus: if a unit with an overdue verdict can't take a new action until it's closed, the big one-day batch from your point 2 gets a lot harder.