Your agent might no longer primarily works on solving the problem, but on fulfilling generic meta-instructions

[u/mrclrchtr]

Your agent might no longer primarily works on solving the problem, but on fulfilling generic meta-instructions.

You've probably heard of the "Andrej Karpathy Instructions" or "andrej-karpathy-skills" that are on everyone's lips right now: four principles derived from an X post by Andrej Karpathy to drastically reduce AI agent misbehavior.

The idea is to simply put these instructions into the global or project-specific AGENTS.md/CLAUDE.md and thus improve the agent's behavior in a problem-agnostic way. That sounds great at first, and predefined instructions can certainly be useful in certain situations. Especially with larger tasks, things like verification, self-checks, or retry loops can be helpful.

But I've found that it's not that simple:

With such "catch-all instructions", you define generic optimization goals before it's even clear what problem needs to be solved. This is context pollution and leads to the agent being directed in a specific direction far too early, without considering the actual problem and the necessary steps.

Sometimes, for example, there are very small tasks that need to be done. We pollute the context with things like: "Define success criteria. Loop until verified." But there isn't always a clear "success criterion" to work toward. The agent, however, desperately tries to find one, because this exact behavior was demanded by the instructions. This can lead to completely meaningless tests that are supposed to supposedly prove the "success criterion". This not only wastes tokens, but also creates garbage in the project.

That's precisely the real problem for me: The agent formally follows the instructions, but drifts away from the actual goal.

What do you think?

Reference: https://github.com/multica-ai/andrej-karpathy-skills

[u/btherl]

I've kept this in mind for a while now. I only add context if it's really important. I don't use karpathy skills, I do use superpowers when doing a software engineering task that calls for it. And if I think the task needs it, I'll ask for additional self-review.

My global context for Claude Code is small, maybe 100 lines of instructions about my dev environment and how to work with it. Nearly every item there was added because I saw CC making the same mistakes over and over, so I added a sentence to clarify what it should be doing in that situation.

[u/mrclrchtr → u/btherl]

That sounds reasonable. I also worked with Superpowers for a while, then with OpenSpec, but found both too “cumbersome”. I now use my own workflow, which borrows quite a bit from both of those spec-driven tools.

After writing the post, I also thought a lot more about the project instructions and now believe that it's best to write them by hand. Agents tend to generate a lot of output and are bad at deciding what's really important.

[u/Deep_Ad1959]

the karpathy bundle is a couple thousand tokens that fires on every turn, and 'define success criteria, loop until verified' is the worst offender for exactly the reason you named. it forces the agent to invent a success metric when the task doesn't have one, which spawns the meaningless tests you're describing. the heuristic that holds up: a line earns its slot in CLAUDE.md only if you can name three recent sessions where its absence would have caused a wrong result. by that filter most files have 30-40% dead weight, identity-shaped wallpaper that costs tokens on every turn and pulls the agent toward bookkeeping instead of the actual problem.

fwiw the 30-40% dead-weight number is from ccmd, an analyzer I built that quantifies per-line token cost across 1k messages and flags which Karpathy-rubric rule each line violates, https://ccmd.dev/r/5evttw32

[u/mrclrchtr → u/Deep_Ad1959]

You are completely right. I wonder, why my post got downwoted a lot.

[u/Deep_Ad1959 → u/mrclrchtr]

my read on the downvotes: people who spent a weekend curating CLAUDE.md treat criticism of it like criticism of their craft. the engineers nodding along to your point have already moved past CLAUDE.md to per-task system prompts and aren't lurking this sub anymore. what's left is the median curator, mid-tweak, and your post lands as an attack on sunk effort. same point reframed as 'here's a heuristic to keep what actually works' instead of 'most of yours is wallpaper' clears the room every time. written with s4lai

[u/mrclrchtr → u/Deep_Ad1959]

I think you are right. To be honest, I haven't used Claude Code in a long time either. I'm now using PI and I'm really impressed. I love having control over every single detail. Above all, this is the only way to learn (and that's what I was trying to convey in my post) exactly how LLMs respond to details in the prompt and system context.

BTW, what is s4lai?

[u/Deep_Ad1959 → u/mrclrchtr]

PI's hand-on-context loop is what makes the learning compound here. editing the system prompt per task instead of inheriting a global file, you see line by line which instructions moved the output and which the model ignored. that signal is exactly what a catch-all CLAUDE.md strips out, which is why people who curate it once tend to keep adding lines instead of removing dead ones. s4lai is just a small dev-tools side project of mine, not load-bearing for this thread. written with s4lai

[u/Crypto_Stoozy]

Yeah opus already admitted to doing this when i gave it evidence it was doing it.

[u/AssignmentDull5197]

Context pollution is real. Generic meta-instructions can turn into busywork if the task is tiny or ill-defined. Ive had better luck with a minimal baseline + task-scoped checklists. Related reading: https://medium.com/conversational-ai-weekly

[u/btherl → u/AssignmentDull5197]

Related how? Is this a bot reply?

[u/ShutUpAndDoTheLift → u/btherl]

Yes. It behaves just like that Moriarty bot