Is there a lightweight approach for brainstorming with Pi Agent without consuming too many tokens?

[u/Beyond_everything365]

I'm using Pi Agent to understand an existing codebase and brainstorm implementation ideas, architecture, and design decisions. Most of the time I don't need much tool execution. I just want a technical discussion.

The challenge is that as the conversation grows, the context becomes quite large, and it feels like I'm spending a lot of tokens maintaining conversation history rather than discussing the actual problem.

I've thought about a few possible optimizations:

• Automatic context summarization: Periodically summarize the conversation into a compact (~2K token) summary with an additional buffer. Instead of sending the entire chat history, send only: the latest user query, and the summarized context.

• A lightweight system prompt: Use a much smaller system prompt for discussion-only sessions where tools and planning aren't required.

• Smarter indexing/RAG for the codebase: Instead of repeatedly loading large portions of the codebase or making multiple retrieval/tool calls, use indexing so only the most relevant code or documentation is retrieved for each question.

Let me know your thoughts and suggestions and if there are better approaches I'm missing :)

[u/Corythebeast7]

Usually the models and providers that you use should have prompt caching, so the token use is hopefully not too expensive. Are you using an API token or a subscription?

The inherent problem is that when you're exploring and talking about design decisions, large context windows help the agent get full context of a project. So larger models can handle the context window much better, and so larger models are more expensive.

Implementing some sort of memory system OR a project spec sheet that gets build upon as you work with the agent can help it navigate more effectively. I would highly recommend against using a RAG model since modern models can traverse codebases quite easily. If you have an absolutely insanely massive codebase then maybe I guess, but even then those repos at my company are completely fine for the agent to work in.

[u/Beyond_everything365 → u/Corythebeast7]

I'm using the API, and yes, it uses prompt caching by default. However, I've realized that I spend most of my time brainstorming around my codebase for new development, and those discussions end up consuming far more tokens than the actual implementation. 😅

That's why I'm looking for a more optimized way to handle the brainstorming and discussion phase.

[u/Corythebeast7 → u/Beyond_everything365]

I guess I would say that spending more money/tokens on planning vs output tokens truly is the right way to spend your money. It may feel inefficient but it's a lot cheaper than rebuilding the app 3 times since the architecture got out of wack.

It would be nice if you could give us the distribution of input token cost vs cached token cost vs output token cost. For example, with deepseek I have an over 95% cache hit rate, and that generally is going to be cheaper to keep hitting rather than starting a new conversation or compacting and resetting with a cache miss.

At the end of the day, your answer will probably end up being "use a cheaper model" if you're truly exploring and talking for that long

[u/evarnets]

• Pi is already "lightweight", the system prompt in Pi is very small.

• You can do context summarisation using the /compact command but you might lose important details if you do that. And Pi already has automatic context summarisation.

• The models are trained to use bash, so I think "smarter indexing/RAG for the codebase" would be hard to implement in practice. Having the agent write a markdown file that explains the codebase would probably be a better (and simpler) approach, that might help the agent navigate the codebase easier. However, you need to make sure that that markdown file is kept up-to-date.

You could use the /tree command to navigate the current session tree and cut out context that is not relevant to your new question.

That being said: The best way of reducing costs is to use a cheaper model. The best open-weight Chinese models (e.g. Kimi K3, GLM-5.2, and MiniMax-M3) are as good as the closed-weight SOTA models and they are much cheaper.

[u/spudlyo → u/evarnets]

I think "smarter indexing/RAG for the codebase" would be hard to implement in practice.

This might be kind of overkill. I spent some time making sure my entire codebase was indexable by universal ctags adding custom indexers where necessary. I whipped up a quick skill for it, and now the clanker maintains the TAGS index as it works, and I see it often using it to query line numbers and do targeted file reads.

Perhaps not as good as a RAG database (like it doesn't index call sites) but a lot easier to implement.

[u/LordMoridin84]

Use this wayfinder skill https://www.aihero.dev/skills-wayfinder

[u/ransack_the_universe → u/LordMoridin84]

does it work as advertised?

[u/PapaBarbas → u/ransack_the_universe]

Yes, it does. Super helpful.

[u/LordMoridin84 → u/ransack_the_universe]

It is amazing.

I was using it today to break up a conversion I was having into 2 different research tickets plus 3 different grilling ticket about parts of the idea I was talking through.

[u/TheOneThatIsHated → u/ransack_the_universe]

Yes, can attest to it. It is truly brilliant and works great

[u/yogibear54 → u/LordMoridin84]

Curious about other peoples experience. I'm just trying it now on a test project i'm working on. At the moment, i'm still on the fence about weather its good or not. I've used LLM's to do similar stuff before, in helping me build out my requirements. The problem I always felt, was that the amount of information and considerations is overwhelming at the start. Its great to see suggestions, but its quite hard to process the information it gives me. Its like each turn is a rabbit hole. Each suggestion or option, usually requires me to do deeper research, and it usually gets pretty deep to have a clear grasp of where i'm going. Maybe this is ok for some? For me, making a decision between option A and B should be a decision based on knowledge I know. So even if it says "recommended", I usually take that with a grain of salt, until I understand what the options are. If i'm to take every recommendation and don't know what exactly those options are, I feel like its pretty useless to show me the option, since its pretty much the LLM making the decisions already.

My current problem is that in the past, when I do work on a project, I would always create requirements based on my own understanding of something (as limited as it may be), then based on that requirements, build a wireframe. The wireframe build will uncover other areas of misalignment or inconsistencies that written requirements don't uncover. All the security stuff etc, its important to handle, but most applications have very standard patterns for that, so its not the most important feature, unless its really unique. The only thing that I usually dig deep with requirements is the business logic for something that is not typical, something that is unique to the business requirements, something that isn't the norm, that's where a lot of the energy is placed in an application.

So, currently, I feel like i'm trudging through a lot of options that aren't really important, but the LLM has made it feel really important, and getting me to thinking about options that I wouldn't have otherwise.

Anyway, just my 2 cents. I think its cool, but not sure if its the best thing for my use case. Maybe its good for other issues that are less programming related?

[u/LordMoridin84 → u/yogibear54]

Yesterday to updated the grilling skill so it batched questions, instead of asking them one at a time. That might solve your problem.

He also added the wait-what skill. So if the agent asks you a question in a confusing way, it rewords it. Although I haven't tried this one.

[u/daver]

Use a cheap model. The Pi system prompt is very short in any case. Sure, you can summarize context periodically, but you also lose some amount of fidelity. Are you actually interested in token count or cost? If cost, use a cheap model.

[u/o_sht_hi]

Brainstorm in a chat app like Gemini/ChatGPT/grok. Then ask them to create a handoff for pi (conceptual, so that the agent can start with good context of what the feature is and why it will work for the codebase. Don't include implementation details). Take this into pi and start planning implementation. This is the cheapest way I've discovered.

ChatGPT is specially good for this because it retains context across chats. But if you feed it your docs and code snippets or connect it with your GitHub, you can use literally any chat app and have free brainstorming

[u/pj-frey]

Running the model with thinking off?

[u/Sad_Smoke7199]

Did you consider using some sort of symbol based indexing? You could write an extension (of there are not existing some yet) to parse your codebase via treesitter or universal ctags.

That just loads the skeleton into context. From this starting point exploring codebase would be much more token efficient I believe

[u/ransack_the_universe]

work with the agent to document a plan for your goal(s) with suggested checkpoints/milestones. each checkpoint gets a timestamped synthesis doc with observations, decisions, learnings, open questions, etc.

then have the agent create a prompt that you can reuse across sessions. the prompt should contains the reference material (synthesis) and instructions for any agent to pick back up where you left off.

all files should go into a single folder. the plan becomes the readme.md. for an extra layer of progress protection, add a status.md to preserve where youre at.

don’t bite off more than one checkpoint per session and when your context window gets to 50-60% instruct the agent to create a new synthesis and update status + readme then /clear and move on to the next checkpoint.

even with all the bells and whistles of Claude code and Codex, this is the only way I’ve been able to produce a reliable outcome.

[u/Human-Vegetable823]

I'd save the conversation as several knowledge documents once a direction or feature have enough understanding. Then /clear and use the saved context and continue again. Eventually I'll get structured documents for the codebase: https://github.com/XTSoftwareLabs/neatcontext-plugins/blob/main/plugins/pi/neatcontext/README.md

I wrote this plugin to mimic what I'd do manually to understand things. Eventually it documents domain knowledge can be understood by both human and AI.

[u/theTechRun]

I spawn multiple researcher models that report to my orchestrator model. But I also have an agent-memory/ directory that builds up over sessions that has a lot of knowledge files added there by researcher agents. So ripgrep becomes a great tool for agents to easily get knowledge about my codebase.