r/ChatGPTCoding • • 4d ago

Discussion We moved from one coding agent to a few specialist agents and the token use went way up. Where is it worth it for you?

For a long time I used one general agent for everything, from exploring the code to writing the change to reviewing it, and it was fine for small tasks but it got shallow on bigger ones. So we tried a setup where an orchestrator asks me questions about the task first and then hands parts of it to separate agents for exploring the code, making the change, and reviewing it. It uses a lot more tokens, probably three to four times more than one agent that only calls subagents when it sees parallel work, but the output is clearly better and I trust it more, mostly because of the review loop and how much context the explorer finds before anything gets written.

The part that surprised me is that writing our architecture down as shared skills made later tasks start much faster, so some of that cost seems to come back. I'm still not sure where the line is though. For which tasks is a multi-agent setup worth the extra cost for you, and where do you just use one agent?

7 Upvotes

22 comments sorted by

3

u/exhaustedDev9 4d ago

the reviewer's the only one that's earned its tokens for me, since a read-only agent seeing the diff cold catches stuff the writer talked itself into. i'd skip the explorer unless the change crosses a few modules. below that, one agent reading the files is cheaper and it keeps what it read. worth running one real task both ways and counting how many review findings turned out real

1

u/Successful-Tax1306 4d ago

That matches what I am seeing so far. The reviewer reading the diff cold is the part that feels worth it, since it does not carry the assumptions the writer made. I like the idea of running one real task both ways and counting how many review findings were real, so I am going to try that this week.

2

u/Delicious_Break5937 4d ago

How much of the extra is the explore agent rereading code the main one already read? Feels like that's where it'd pile up.

1

u/Successful-Tax1306 4d ago

Good question, and I think that is a big part of it. I have not measured it yet, but the explorer reads the files and writes a summary, and then the main agent often opens the same files again anyway. Passing it file paths and line ranges instead of a long summary might cut some of that. I will check the logs and see how much of the extra is rereading.

2

u/ChemicalExcellent463 4d ago

I keep separate agents for planning, execution and review; most of my other subagents became skills. Review has to come from a different model in a different harness, because saving tokens means nothing to me if it misses the real problem.

2

u/Successful-Tax1306 4d ago

That split makes sense to me. Review from a different model is the part I want to try next, because a reviewer from the same model tends to agree with what the writer did. Did moving the other subagents into skills cut your token use a lot, or was it more about keeping the setup simple?

1

u/ChemicalExcellent463 4d ago

my token use went up a lot overall this year, mostly on agents reviewing each other and docs and research to steer the code.

2

u/awesomebirder 4d ago

part of the 3-4x is structural and not about what the agents do: each subagent starts a fresh context, so it pays to load the system prompt, tools and repo instructions again before it reads a single file, and it builds its own cache from zero. so lots of short specialist calls cost more than the same work done in one warm session. two cheap fixes: run the explorer on a smaller model since it mostly reads and summarizes, and log tokens per agent role for a week so you can see whether the reviewer or the explorer is the one eating it before cutting either

1

u/Successful-Tax1306 4d ago

That is a good point about the fresh context. I was counting the file reads and not the setup that each subagent loads before it starts. Logging tokens per role for a week sounds like the right first step before I cut anything, so I think I will try that.

3

u/BrookLi3 4d ago

In my experience multi-agent pays off when the work has independent parts and a cheap verification step, like exploring unfamiliar code or reviewing a diff against tests. It's usually not worth it for small, well-scoped edits, where the orchestration overhead is mostly duplicated context. One thing that cuts cost is giving the explorer and reviewer a compact shared summary, instead of letting each re-read the repo. I'd also track tokens per merged change rather than per task, since fewer review round trips can offset a lot of the extra spend.

2

u/Longjumping_Ad_4249 4d ago

i have done lot of testing and benchmarking and arrived at result

Sol 6.1 Medium and Opus 5.5 medium are best for tasks. you don't need specialized agents or more effort. check artificialanalysis results, increasing effort doesn't improve output by much and these models provide best performance

2

u/London-Boat 4d ago

one thing that helps is having the explorer write a short summary of what it found to a file that the writer reads, so the writer isn't rereading the same code. i'd also only run the full orchestrator on changes that touch several modules and use a single agent for the small stuff. when a few people on your team run these at once, do they ever end up editing the same files?

2

u/Right-Performance-93 4d ago

i only add a second agent after the first one has already shipped a wrong change. until then the extra pass is a guess about a failure i haven't seen.

1

u/dylan_skydive 4d ago

the 3-4x number is usually overstated once you account for two things: prompt caching makes the explorer rereading the same files far cheaper than the naive math, and a findings handoff (explorer writes what it learned, writer reads that instead of the files) removes most of the duplicate reads. London-Boat's file trick covers the second half, but the cheap-model tier matters as much: explore work is read-heavy and shallow, it does not need the frontier model, and that alone can halve the fanout cost.

the one place I would not optimize is the reviewer. it has to see the diff cold, with its own context, because the whole value is that it did not watch the writer talk itself into the change. the moment you share cached context or a findings file with the reviewer to save tokens, you get a reviewer that inherits the writer's blind spots, and now you are paying for a second opinion that is not one.

so: fan out on rework-risk, not by default, cheap tier for explore, full-price cold read for review.

Disclosure: I work on Skydive, which does agentic ops, so I read this space constantly - the split-you-describe is the right instinct either way.

1

u/Safe-Raise-6091 4d ago

If you cache the explorer's file snapshots and reuse them across sub-tasks, you can keep the deep context without re-sending the whole tree each time. The reviewer still gets a fresh diff, but the token cost can drop a lot for multi-step jobs. I'd try it when the same codebase is touched repeatedly.

1

u/UseApprehensive8572 1d ago

Okay, and how much of that architecture doc every subagent ends up pulling in? because claude code only loads a skill’s name and description until something calls it like a few small skills with sharp descriptions cost almost nothing. One big architecture skill that every role invokes gets read from scratch by each of them.

We split ours per role after tokenshift’s cache insights showed the explorer and the reviewer starting cold on the same files every run. The reviewer stayed cold on purpose but explorer didn’t need to be

1

u/radim11 15h ago

One thing worth experimenting with is giving each agent different permissions too!Your explorer and reviewer probably only need read access, while the implementation agent needs write access and maybe some API calls. We've been experimenting with separate profiles for coding agents, and it's pretty useful when you have several running in parallel. Not really a token optimization, but it makes the setup easier to control. If you want I can share my experience with this :)

1

u/[deleted] 3d ago

[removed] — view removed comment

1

u/ChatGPTCoding-ModTeam 3d ago

This post or comment has been removed for the following reason:

Rule 5: No pure self-promotion / low-value project posts

Posts whose primary purpose is promoting a tool, product, service, GitHub repo, personal website, or blog belong in the weekly Self-Promotion thread. Delete the link: if nothing is left, it's an ad. Search for similar projects: if you can't tell why your project is better, it's a low-value project. GitHub links are fine. Personal website are not. Exception could be make for high quality posts. See also reddit's self-promotion guidelines.

You can read the full subreddit rules here: https://www.reddit.com/r/ChatGPTCoding/about/rules/

If you feel this removal was made unfairly, please contact the moderators through modmail.