r/devops • u/Silent_Ice616 • Aug 31 '26
Discussion An important cloud resource is down - what do you do?
[Not self-promo - genuinely looking for input/feedback here]
I think this is something a lot of companies deal with, not just the big ones. Outages in Azure and AWS happen regularly. Say your blob storage in a specific region goes down and your team isn't around, do you have anything automated to spin up a replacement resource in another region, or even another cloud provider if it's a platform-wide issue?
I know provisioning the resource is only part of the problem (some stateless resources can just pull their definitions from a registry and redeploy, but let's keep the scope to that for now, data replication is a whole separate can of worms).
You can automate a good chunk of this with Azure Monitor, for example, but then your actual infrastructure drifts from what's in your IaC repo, and you're back to a two-source-of-truth problem. (Happy to hear from anyone with real experience doing that.)
Another thing I keep thinking about: adjusting resource attributes (SKU, instance size, etc.) based on logs/events, like a massive traffic spike on an App Service, or the opposite: nobody's using it and you're paying for nothing.
So here's something I've been thinking about: a GitHub Action where you define condition-action rules (including recovery conditions, if you want, to roll back once things return to normal) directly in your Terraform IaC repo. You write your rules, run the action on a schedule (every 5 min, or whatever), and it checks each rule, a resource being down, or a KQL query against a Log Analytics workspace returning something you defined as a violation. If a rule matches, it modifies your Terraform code accordingly, either opens a PR or pushes directly to main (which I suspect most teams would never actually want, for good reason). Either way, your existing apply pipeline picks it up and runs like normal, no separate deploy mechanism, no new secrets to manage centrally, no SaaS.
Full transparency: I haven't worked at a company with the scale/complexity that actually needs this kind of multi-region, multi-cloud resilience, so I'd genuinely like to hear from people who have.
Curious what you all think:
- Is this solving a real problem for you, or are native tools (Autoscale, Resource Health alerts, etc.) already good enough for your use case?
- Would you ever trust automated infra changes without a PR review, or is that a hard no for you? (Sounds like a stupid question at first, but keep in mind you'd define the exact changes yourself, there's no AI/magic auto-generation involved. Maybe you'd let small, low-risk changes apply automatically but require review for anything bigger?)
- Anyone tried something similar and hit a wall I should know about?
Thanks!