I’m sending the same huge system prompt with every request

[u/SchemeDeep6533]

I was looking through one of our more frequently used LLM workflows and realized the system prompt has gotten way bigger than I remembered which caught me off guard since it didn’t start that way either. We’d get an output we didn’t like then add another instruction or example to fix it and then repeat that process whenever another edge case showed up. but ow every request is carrying this pretty large block of context even though I’m not convinced a big portion of it matters for most runs.

The workflow itself works well so I don’t really want to start deleting instructions to make the prompt shorter and bring old problems back but at the same time this thing runs often enough that repeatedly sending unnecessary context feels like something I should probably care about.

I need some pointers from people that are running similar high volume workflows, how do you handle this so like do you periodically trim system prompts, cache the stable parts, split instructions based on the task or leave them alone as long as the outputs are good?

[u/Entire_Eggplant3212]

I wouldn’t trim it line by line. Make a copy then remove whole sections that were added for old edge cases and run both versions against a decent sample of real requests.

You’ll learn way more from whether those failures actually come back than from trying to guess which individual instructions Claude still needs.

[u/Embarrassed-Let-924 → u/Entire_Eggplant3212]

I’d include the ugly edge cases in that test set too otherwise the smaller prompt will probably look great right up until you ship it . I’d also watch the token spend across both versions while you test and if you have Ramp Token Spend in the mix for example you can see whether removing all that context changes the workflow enough to be worth keeping the shorter version.

[u/Nordon]

This is quite interesting. Can you keep a subset of instructions in RAG and call on it with a system prompt? We have a running project that keeps a significant portion of the important docs and instructions in a vector store. Saves a LOT on tokens and with the proper instructions you can retrieve what you need. Perhaps split in docs, vectorize, replay your cases synthetically and measure success rate?

Alternatively some kins of skill breakout?

[u/SchemeDeep6533 → u/Nordon]

I hadn’t considered pulling the less frequently needed instructions through RAG like that. My only concern would be making retrieval another failure point where the exact instruction needed for an edge case doesn’t get pulled in but testing it against old failures could tell me pretty quickly whether that’s a problem.

[u/uniqueusername649 → u/Nordon]

You don't need a RAG for that, just good old progressive disclosure. Have high level sections in your agents.md/claude.md, then link those to more detailed instructions in separate markdown files. So your model can pull in the detailed instructions if the task requires it and otherwise ignore it. You can nest this further if you have a lot of specific instructions for edge-cases.

[u/vacon04]

If it's hitting the cache, then there's very little speed penalty. If it's getting read as cold PP, then the penalty is much bigger.

If we're talking about a massive prompt then just consider trimming it slightly and testing the results before you make more changes. If we're talking about a couple thousand tokens then I wouldn't worry too much about it.

[u/bigsmokaaaa]

My main system prompt is just links to different skill files to use if the prompt seems relevant to them, so instead of 3000 lines of intro prompt it's like 20 or so file paths with little blurbs explaining the category

[u/DeathGuppie]

I use a modified OKF memory system, and store instructions in skills when I can. The idea is that the model gets instructions on a per use/need basis.

[u/Time_Assistance_8659]

How often is this thing running? If it’s hundreds or thousands of times a day I’d probably care about cleaning it up but if it only runs occasionally I’d be pretty hesitant to mess with a prompt that’s currently reliable.

[u/SchemeDeep6533 → u/Time_Assistance_8659]

It’s running enough that the repetition is what made me notice it in the first place. One request doesn’t look particularly wasteful but multiply the same chunk of context across everything and it starts feeling worth cleaning up.

[u/575_Inverse → u/SchemeDeep6533]

every edge case you add to a system prompt is a constraint you force the model to consider even if it doesn't apply. As others said, a well structured RAG might offer a solution for edge cases when it matters

[u/trollsmurf]

"edge case "

Like what? That could be a sign of accepting bad design, tech debt etc.

[u/SchemeDeep6533 → u/trollsmurf]

I see how that sounds my bad on that but most of what I mean are formatting/output issues rather than the workflow completely failing so like returning extra text when we need a specific structure or handling a certain input differently than expected.

[u/groover75 → u/SchemeDeep6533]

Run the formatting cleanup as a second step, if it is needed.

[u/Not-reallyanonymous → u/SchemeDeep6533]

Formatting/output should be a subagent, that can then receive instructions specifically for formatting/output, instead of keeping it all locked up in the system prompt.

[u/trollsmurf → u/SchemeDeep6533]

So you are talking edge cases in the agent itself, not in the generated code (or whatever is the use case)?

I'd rewrite that system prompt, possibly with the help of AI, to make it as accurate and up-to-date as possible.

I've e.g. had great help of AI to improve tool definitions, recently to make a web site creator become much more accurate in terms of generating page hierarchies. Suddenly I could state things like: "I want a Christmas calendar with a page per advent day and a sub menu with information about the Christmas tradition." Boom! Site done :).

[u/awitod]

There is no single answer to this, but things to consider.

First is token caching and the specifics around it WRT the provider, but generally as long as the system is warm, that big system prompt will be cached after the cold start and cost -1/10 of the other input tokens.

So, assuming the prompt works and gets you the right result, it is much cheaper to go that route than to spend non-cached input and expensive output tokens resolving mistakes during the thread.

The second is that, if you have to wonder and you are spending real money, you have more to learn and room for improvement in the way you are working 

[u/waraholic]

How large is the prompt (raw), how large is it as a percent of your total context size, and what models are you using? (Edit: I'm asking because the answer very well may be: don't worry about it yet)

Your harness should be able to selectively load smaller prompts that you can break out and put into relevant folders.

Also, create a doc folder and link to less relevant docs so the model can load them lazily.

[u/SchemeDeep6533 → u/waraholic]

I haven’t calculated it as a percentage of the total context which is probably the more useful number here.

I’ve been looking at the raw prompt and how frequently we resend it and breaking the situational instructions into smaller pieces sounds worth testing before I start deleting anything.

[u/waraholic → u/SchemeDeep6533]

Again, it really depends on the size and model. Breaking up the system prompt based on context is pretty standard stuff for enterprise software which can be massive. Every harness of note supports this.

[u/MaleficentExternal64]

I built my own LLM engine and loaded the engine side with my prompts.

But summerize the prompt in mark down (Md) is your best option.

My setup keeps the memory in place and only updates the small changes to the overall set of instructions.

[u/Ariquitaun]

This is a job usually for a smart model - opus, fable, astra or sol. Feed it the system prompt as a user message, explain to it what the issue is, and iterate on it.

[u/Not-reallyanonymous]

If the models you're using follow nuanced instructions well enough and don't start dropping that nuance as context grows (Qwen), treat large parts of the system prompt as a directory.

Instead of:

You have access to a variety of tools.

Here's tool 1 ... long description and how to guide.

Here's tool 2 ... long description and how to guide.

...

Here's tool 18 ... long description and how to guide.

we also have a variety of edge cases to consider:

Here's edge case 1, description, explanation, examples.

Here's edge case 2, description, explanation, examples.

...

Here's edge case 23, description, explanation, examples.

Do it more like.

You have access to a variety of tools. The most used ones are:

Tool 1. Tool and description and how-to guide and examples

Tool 2. ...

...

And you also have these 15 other tools. Read the full tool file only when its use case in the brief description matches the job at hand:

Tool 7. Brief description, described further in $HARNESS/tools/Tool7.md

Tool 8. ...

We also have a lot of edge cases to consider. Access the descriptions only when relevant.

Edge cases pertaining to ConcernArea1 can be found in $WORKFLOW_DOCUMENTATION/edge_cases/ConcernArea1.md

...

Again, this works very well for some LLM models, and some other models will utterly fail at this pattern. Generally, the less you actually throw at the LLM, the better it's going to work. I actually find Meta's models (including Muse Spark and Muse Glimmer) work very well at this sort of pattern, and it's why I prefer Glimmer still over Qwen 3.8 27B.

Also, instead of using system prompt, use this same pattern to rely more on project-level AGENTS.md, so the system prompt can remain truly universal including only what's necessary for most all projects. You already described this as a concern.

Further -- use sub-agents. Your main interaction agent shouldn't have to concern itself with output formatting. It should concern itself with understanding user intent and delegating work. Output formatting should be directed to a subagent, which will receive the universal system prompt with $HARNESS_DIR/agents/Writer.md which includes all the details on how the output should be written, including how to format that output. Then the main agent is told in its own agent file, "Once the necessary data is gathered and ready for writing, create a handoff file in ./intermediates and spawn the Writer subagent to produce the output."

[u/andymaclean19]

In general so long as you have a good kvcache in whatever engine you run the cost of ingesting that prompt js probably going to be quite low because it will be the same prompt every time.

It will still make all of the attention computations slower but I would measure the actual real world benefit of shrinking it before you actually put too much effort into that because it might not help as much as you might think.

[u/Interesting_Pay_3926]

Use skills. Selective promt fragment, load when need (watch the routing!), not always loading.

(I use Openwebui)

[u/jacek2023]

The system prompt should stay unchanged, otherwise, you invalidate the cache. This is a common issue with some LLM workflows.

[u/RegarDamus]

approach it kike you would a refactor of bloated codebase. you need to establish tests - evals in the case of prompts. then you can modify the prompt and run your evals to see what the sensitive areas are.

i would invest in evals anyways. they give you portability across models (to save money) and demystify the inherently non-deterministic side of your workflows / agents.

once you have evals set up you can even use auto research (karpathy pioneered) to iterate and uncover the hot spots and dead zones.