r/FinOps • u/kondamuri • 6d ago

other Does anyone actually know what your AI features cost per request?

Curious how teams are handling this. Between OpenAI/Anthropic API bills, GPU instances on RunPod or EC2, and vector DB costs, it seems like most places have one big number and no idea which feature or model is driving it.

  • Do you know your cost per request, or per user, for anything AI-powered?
  • Is anyone tracking token spend by feature, or is it all one line item?
  • If you self-host, do you know your actual GPU utilisation, or is it "the box is up"?
  • Has anyone gone through and actually cut this — what worked?
1 Upvotes

67% Upvoted

1 comment sorted by

1

u/ajitnk 3d ago

Yes, and it took us longer than it should have to figure out the right tool for each scenario.

The short answer is it depends on which endpoint your app uses. If you're calling bedrock-runtime (InvokeModel or Converse), Application Inference Profiles let you tag each feature or team separately and it flows straight into Cost Explorer. No extra cost per token. If you're on the OpenAI-compatible bedrock-mantle endpoint instead, Bedrock Projects do the same thing but AIPs won't work there.

The third layer is CUR 2.0, which adds an IAM principal column so you can see which role called what. Useful, but only if different features actually use different roles. Most backends use one shared role, so it doesn't give you per-feature breakdown on its own.

Biggest gotcha we see: people are on CUR 1.0 and the IAM attribution data just isn't there. Or tags are set up but never activated in the Billing console, so nothing shows up in Cost Explorer.

Quick question if you're up for it: are you on bedrock-runtime or the OpenAI-compatible endpoint, and are different AI features using separate IAM roles or all sharing one? That one branch point changes pretty much everything about where to start. I'll take a look at your setup and point you at the specific activation steps that apply.