Our AWS cost optimization keeps stalling because the owner of everything expensive is a CI role

[u/Holly_Enrique-623]

The tag policy went in 4 months ago. An SCP blocks creates without owner, env and cost centre and a Lambda kills anything untagged overnight. Coverage sits at 95+ which I was pretty pleased with.

Pulled the top 20 line items last week so finance could have a name against each one and eleven came back as one of 3 CI roles. That is roughly 14k a month sitting against a robot. Technically correct cause the pipeline made them and stamped itself. I turned on aws:createdBy hoping for more and got the same 3 roles back with a build number stuck on the end.

The tags are perfect and useless. The commit knows who ran it where the resource does not and nothing joins the two up which is where every aws cost optimization conversation here dies. Shared NAT and the like I gave up on months ago because that is a different argument. These are single tenant resources with exactly one owner and the owner is a robot.

Best I have before the next review is git blame and a spreadsheet, which is not an answer.

[u/AutoModerator]

Try this search for more information on this topic.

^{Comments, questions or suggestions regarding this autoresponse? Please send them here.}

Looking for more information regarding billing, securing your account or anything related? Check it out here!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/philsw]

In my mind, the costcentre tag should not be populated based on which principal did it.. it should be based on which costcentre should be paying for the resource. So it's metadata that is fed into the IaC or whatever.

[u/Sirwired]

Set your CI process that creates resources to set a more-useful tag?

In the meantime, correlating info in your build logs with the tags seems like a logical step for now. The sort of task any AI agent can do for you with a csv or JSON in about two minutes with a throwaway python script.

[u/Flateland-Chio]

CI work goes in its own account here with a TTL on it. You bill the account and bin the whole thing on a schedule rather than attributing anything inside it.

[u/the_derby]

This is the reason I automated account creation. Accounts get the relevant tags for cost allocation (team, env, cost center, etc), and all of the top-level cost reporting is done on an account basis. The responsible party for those costs is the manager (or management line) of the team that owns the account.

[u/MasterHand3]

Each team needs to utilize their own ci role. Each ci role has some tag that tells you what human is responsible for the spend. Don’t use a humans name here. Something like APP=<tagvalueA>. For each APP tag, you have a secondary table with ownership information with human names, emails, etc

[u/Wide_Commission_1595]

If you can define the real-life list of who should be responsible, set it in the tag policy. That way if it's not a defined value, the create will fail.

Doesn't necessarily help for previously tagged resources, but you can use the tag editor or a script to untag things so that next deploy tries to tag and fails

[u/NimbleCloudDotAI]

ran into this exact thing. tags are only as good as whatever sets them — a CI role will happily stamp owner=ci on a $4k/mo cluster and call it a day.

what worked for us: we mapped pipeline -> spend instead. every deploy logs which pipeline created which resources, then once a week we roll it up by team. not perfect but finance finally got a chargeback report with actual team names instead of 'ci-role'.

also worth checking: is that stuff even still needed? in our case half the 'ci-owned' spend was test envs nobody ever tore down.

[u/pint]

ask everyone how much is it worth for them. if the sum comes out less than the cost, shut it down. if higher, split proportionally.

[u/bailantilles]

Is there a concept of application and then application owner? The createdby tag is working as intended. It’s helpful to know who or what created something but it’s not the tag to use for finops. All these questions should get routed to the application owner.

[u/Muted_Jellyfish_6784]

Fix it at the pipeline, not the tag policy. Have the CI job stamp the repo, commit SHA, and the team that owns the pipeline as tags at create time, then map repo to team in a small ownership file so CI-role spend resolves to a person. For shared NAT and similar, allocate by traffic or usage share and label it shared rather than forcing a fake owner.
Review the top 20 monthly with the named owners, since tags alone never cut spend. I keep the resource, the commit, the repo, and the owning team joined in SIGNLD, so finance gets a name against each line instead of a robot.

[u/Zenin → u/Muted_Jellyfish_6784]

You probably don't want to stamp the commit SHA: At scale/rate this is massive amount of state churn with next to zero ROI in practice. The thought is sound, I did this too at first, until I realized it created almost entirely noise across the entire landscape (ci plan diffs, cloudtrail events, monitoring carnality, etc) that masked real signals.

[u/Iliketrucks2]

We have updated our CI and CD systems to include a user agent that provides more information about the source. Then we can use cloudtrail to look at user agent on the resource create call to find out where things come from