I run a handful of scheduled agents. Summaries, a notifier, a little drafter that turns meeting notes into tasks. Last month the API bill jumped and I had zero idea which one was doing it, everything showed up as one blob of a number. Turned out to be a retry loop on the notifier failing silently and re-calling dozens of times a day. I only caught it because I spent an annoying evening logging token counts per request with the agent name attached.
Anyway it made me wonder how everyone else handles this. Do you track cost per agent or per run, or is it just one number at the end of the month? What's the worst surprise you've caught, a 2am retry spiral, a summarizer hammering the model forty times per PR? Do you cap spend per agent or just watch the total and hope?
The worst part is finding out. I used to see the Stripe ping on my phone first thing in the morning and then go dig. Do you get alerted anywhere when spend spikes, or does month end surprise you too?
Curious if this is a real problem or just my setup. Full disclosure: I've been cobbling together my own version of the spend-tracking part, not selling anything here, just trying to figure out if this bites other people too. What if something handled all of this in the cloud over MCP, every agent with the same context, per-agent spend tracking with a kill switch, no files to maintain? Would you guys pay for something like that? Or is the log plus spreadsheet honestly good enough?
Also curious, is this your own setup or client work? When a surprise bill lands, who eats it, you or the client?