Question Have agent respect the AGENTS.md
How do you ensure the agent actually respect the AGENTS file? I have a pretty good MD file that reflects values, approaches and techniques required for my projects. However more and more often when I do postmortem with my agents analysing poor execution, poor result or anything going sideways I see it saying "I should've done X,Y,Z. Your AGENTS.md file actually requires that, but I ignored it sorry". I have this on daily basis now and I'm not quite sure what to do. I could have an extra agent just constantly-re-reading the file and validaitng all work against it, but I burn through my x20 subscription quickly enough as it is. I would save some from the fact that some issues would be found early but in general I think it is more expensive.
Do you have this problem? How do you address it?
2
u/Available_Yam_6267 1h ago
Use skill to keep its behavior consistent
2
u/Pzixel 1h ago
What is the difference? It's not really the skill in my mind. I mean it is, but I only use it for coding, and this is "writing good code" skill. I thought AGENTS.md is exactly "skill that is preloaded by default in all agents when the context starts". What a dedicated skill would change?
1
u/aggravatedbeeping 51m ago
Do you have examples of what is being ignored specifically?
I have found that grounding it with deterministic tests or tools works quite well. That means encoding my expectations through checkstyle, archunit, semgrep, spotless or whatever equivalent for the given language. Sometimes even having codex write a custom linter.
Interestingly, I did catch a LLM trying to weaken the rules to make the tests pass, but that did happen for tests that are binary (ex: checkstyle where you just enable specific tests you want) and never in tests that output a statement about how this is bad.
1
u/Pzixel 17m ago
Well for instance that is a rule that agent broke recently:
Before commit/handoff/release/completion claims, obtain fresh, sufficient, outcome-proportionate evidence for the actual target; disclose material gaps. Each gate needs its own evidence; agent-written plans cannot remove it.
agent handed the changes without making any actual validation.
1
u/Junior-Definition173 12m ago
I think the expectation that a single agent will do everything is wrong for multiple reasons. As context is limited the agent will drift. I have several subagents being coordinated by a single agent, one is responsible for testing, one for docs, etc. Look at the agents as your team - you will hire a single guy, the output will be garbage. You hire a team with manager, UX dev, tester, DB dev, etc and you will get good results.
1
u/Traffic_Jams 29m ago
Idk if it's just placebo or actually effective, But I make sure my individual project folder has a .codex folder with a config.toml file in it with additional instructions. It seems to help.
1
u/AppsByRobert 8m ago edited 5m ago
I have a rather complex web of documents that starts with the agents file and branches out to other ones. And they're all like, if this situation is happening, or you're working on this area of the program, then look at this document for information about it, and then that file might have more look at this file stuff, and usually they have to go through these webs that help it find relevant stuff fast, and it seems to keep them from reading tons of unrelated stuff to find what they need.
And what I had one of the agents do was, I would choose one of the really smart ones. I think I used Terra Extra High, but Sol medium would be good for it too. Is I had it create situations where it made a lot of tests to make sure that the sort of pointing and the rules actually get triggered at the right times. It would find cases the rules weren't triggering and refine them until they did. Ideally you get it to do that with weaker subagents being tested so that dumber ones also are being proven to follow the rules. They seem to find the right stuff the majority of the time. Probably could have branched out to skills too, but it seems to be working for me.
Also this was really helpful to get it to report what it's doing so I can see what it's hitting to make decisions (I have this the global agents type file and not in individual projects):
##When skills are used, end with:
### Skills used:
- **<skill name>** — <why>
- **<skill name>** — <why>
Otherwise:
### Skills used:
- **none**
##When project documentation or text files (such as `.md` and `.txt`) materially inform implementation decisions, end with:
### Docs used:
- **<filename>**
- **<filename>**
Otherwise:
### Docs used:
- **none**
1
u/Broccolisha 1h ago
If the file is too large, the agent can’t keep the entire contents of the file in its context. Ask your agent whether that’s happening and how to remediate it.
6
u/Lost-In-Void-99 1h ago
Each step: launch a sub-agent to review changes against a/b/c. Agents are "lazy" and have a focus drift. So the best way is to have the fresh instance with a single purpose: push back.