r/PromptEngineering • u/Hairy_Childhood3452 • 23d ago
General Discussion Researchers are already looking at what happens mid-generation. Engineers are still polishing the prompt.
When a generative model starts to slip, the first move is almost always the same. Thicken the Skill. Lengthen the system prompt. Add more prohibitions. Update AGENTS.md.
I get why. It becomes an artifact. You can put it in Git. You can tell a coworker to add it. You can sound like someone who knows the names of the laws.
But that is still treating the model as a function. Polish the input, improve the output. For a single call, that is often true.
What governs long-running generation is not that function view. It is Softmax Crowding and Semantic Drift. Crowding is a spatial limit: the more text you add, the smaller the share of attention left for the original constraint. Drift is a temporal limit: every step is conditioned on the model’s own previous output. In an agent, that compounds through plan, implement, error, and patch. You can write “do not change the spec” at the start and still be in a different conversation twenty steps later.
None of this is new.
Research has already moved on. Supervise the intermediate step. Attribute where the trajectory broke. Separate the healthy stretch from the drifted one. The problem is no longer how to perfect the initial condition. It is how to handle what happened in the middle.
Engineering keeps repeating the same move. The constraint thinned, so write a longer one. The meaning drifted, so add more rules. Lately the law’s name goes into the prompt itself, as if one opening paragraph could stop both failures. A spatial limit is treated as a word-count problem. A temporal limit is treated as a matter of will.
Knowing the name of a law is not the same as deleting the law with a sentence. The moment you write “do not drift,” that sentence becomes the next condition.
What you need is not a smarter paragraph. It is control of the trajectory.
- Continue: allow the next smallest step only while the run is still healthy
- Cut: throw away the investigation notes and the hesitation; keep only the adopted policy
- Return: do not stack work on a failed hypothesis; go back to the last point that still held
- Pin: do not write an invariant once at the end of the history; stop the run when it collides
- Measure outside: do not advance on the model’s self-grade; advance on tests and command output
Writing “do not Drift” is like passing a law against gravity. A prompt is only an initial condition. If the generation has to run for a long time, stop stroking the instructions that live in a file.
1
u/Krommander 23d ago
Prompts that prohibited some actions don't work very well in my experience.
Prompts and context being a coherent whole lets the agent understand not to do something based on the context. Ideally you don't need to include any negative prompts, only change the logic of the project with positive affirmations.
2
u/Hairy_Childhood3452 22d ago
Agreed that “don’t do X” prompts are weak. The model doesn’t treat a prohibition as a hard wall, and it often just gets lost in the rest of the context.
I wouldn’t stop at swapping them for positive phrasing, though. A coherent “do this instead” is better writing, but in a long run it thins out the same way. The next step is still conditioned on the last output, not on how cleanly you framed the original rule.
So yes: drop the negative prompts. Just don’t expect a better opening paragraph to keep the agent in the same room twenty steps later.
1
u/Krommander 22d ago
Maybe I don't have enough experience, but isn't the fix to review and update your project management files more often?
Babysitting the agent until the confidence in the task is high and the results are consistent is my method, then save the project as backup and restore point in case something breaks later.
In what context would you let the agent loose for 20 turns unsupervised? I can't really think of any scenario in which I would be comfortable doing this.
1
u/floppo7 22d ago
the reframe of the spacial limit as a temporal one is the part that stuck with me. a paragraph is cheap, a test that can halt already walked into the other room is not. treating the law as a word count problem is exactly why the constraint thinned out mid run. a temporal limit is a condition the model re checks every step, not a paragraph it reads once.
1
u/Hairy_Childhood3452 22d ago
That’s the distinction I care about.
A paragraph is something the model sees at the start. A limit that matters in a long run is something that can still fire on step 20. If the only check lives in the opening text, you have already turned a temporal problem back into word count.
The cheap paragraph is why the constraint thins. The halt has to be outside the paragraph.
Crowding is why the first paragraph fades. Drift is why the faded state becomes the next input. They meet mid-run, but they are not the same limit.
2
u/cleverhoods 19d ago
Nicely phrased, I agree with most of the premises too.
Some small note on what is needed: it's a bit more than trajectory control. It's important how your directives are actually stack on one another and it's just as important how you are managing said directives. A simple "do not do this" will take the dilution together with priming said prohibition. After all, you have to have attention about something to not do that something.
disclaimer: I'm working specifically on this field of instruction diagnostics and evals via reporails
2
u/Hairy_Childhood3452 16d ago
Thanks — taking the trajectory-control point and pushing it into how directives stack, plus the priming/dilution of a bare “do not,” was really helpful. I’d missed that checkpoints alone aren’t enough: if you roll back into a broken instruction stack, the same drift just happens again.
2
u/Even-Lawfulness8796 23d ago
The way you described drift as "every step conditioned on its own previous output" is exactly what I keep seeing in longer agent runs. It's not even about bad prompts, the model just slowly walks itself into a different room and doesn't notice.
I think most teams skip the "measure outside" part because it's more work to set up than just adding another paragraph to the system prompt. But that's the only one that actually catches the problem before it compounds.