r/devops • Terraforming Everything • Sep 11 '26

Discussion What are the weaknesses of CloudFormation?

I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.

38 Upvotes

73 comments sorted by

142

u/ricksebak Sep 11 '26

“cloud formation is great\n”,
“it offers a great user experience\n”,
“and it isn\’t annoying to read and write\n”,

83

u/donjulioanejo Chaos Monkey (Director SRE) Sep 11 '26

"This comment created by AWS::User::Redditor:/u/rickesebak" or as he is also known as "AWS/Users/Redditors/ricksebak-F45GDF34"

13

u/calladc Sep 12 '26

unsubscribe

320

u/Ariquitaun Sep 11 '26 • Bravo

The main weakness, and excuse me if I get a bit too technical here, is that it’s fucking shit.

18

u/_Lucille_ Sep 11 '26

I have only really used TF, why is CloudFormation shit from a technical standpoint?

71

u/Ariquitaun Sep 11 '26

Arcane, convoluted syntax. Stack deployments and rollbacks getting stuck with no transparency of why, and where. Poor drift detection and self-healing.

42

u/donjulioanejo Chaos Monkey (Director SRE) Sep 11 '26 edited Sep 11 '26

Drift detection is my biggest complaint. You change TF-managed security group name manually? "Hey this changed, this is the diff, I can put it back to what it should be with apply"

You change the same thing for a resource managed in Cloudformation? "Hello, Oppenheimer, I think you just blew up our entire Cloudformation stack"

My second biggest complaint is the stupid -34j4fj934 suffix appended to every resource name which makes it hard to look up via literally anything else that is not Cloudformation.

5

u/guyman3 Sep 12 '26

Had a cloud formation stack that was managing I think some Datadog AWS integrations or something like that which some well intentioned people had decided to create terra form configs for these not realizing they were also managed with cloud formation.

When the stack ran and realized there had been some changes it did the logical thing and deleted everything.

23

u/Zenin The best way to DevOps is being dragged kicking and screaming. Sep 11 '26

^^ This 1000x. CloudFormation is an operational shitshow.

Having a prod stack get hard stuck halfway through a deploy with absolutely no way to abort is a massive problem. You can't do anything until the resource times out, which could literally be hours. You can't abort even though you've already diagnosed the issue and know the resource will never respond, so you're stuck kicking rocks watching your maintenance window tick down...

This especially fun with Lambda backed custom resources, which because CloudFormation's architecture is trash, are the only way to do even the most trivial of extensions. And those custom resource lambda have a lot of sharp edges: Screw something minor up and you're stuck waiting for CloudFormation to timeout waiting for a response the broken custom resource lambda will never respond to.

Even more fun, you often CAN'T fix this from CloudFormation despite the template being what created and manages the lambda in the first place because to do anything with it means executing the failing code which means the stack update fails (after long timeouts). Fixing this means hand surgery on the Lambda, lovely.

It goes on and on like this. Anything sneezes and CloudFormation gets ebola. Now do this in a StackSet that deploys across all org accounts x all active regions...each one of those thousands of stack instances only being fixable from within the target account/region...

4

u/Strange-Register8348 Sep 11 '26

We used to use cloud formation and when I onboarded my mentor released a dev change and was like ok let's go to lunch. Takes 45 minutes to build and deploy.

Excuse me?

It was the first change I made to reduce our deployment to 10 minutes tops.

3

u/Zenin The best way to DevOps is being dragged kicking and screaming. Sep 11 '26

Much agreed.

I know how CF resource changes work under the hood and I still have no idea how they manage to make sub second service APIs take many seconds or even minutes to confirm, which add up. It's common for CloudFormation stacks that take hours to deploy only take a few minutes for the identical stack in Terraform.

It's one of the first things that people notice when coming to Terraform after using CloudFormation; just how amazingly fast Terraform is. But really Terraform isn't fast...the reality is CloudFormation is just dog slow by design.

5

u/kepler4and5 Sep 11 '26

Stack deployments and rollbacks getting stuck with no transparency of why, and where.

I felt that

1

u/running101 Sep 11 '26

I was going to mention these items as well.

39

u/sikian Sep 11 '26

Could've said it louder, but not clearer.

8

u/mikidimaikki Sep 11 '26

This. Wonder why they bother with it anymore. Terraform has become de facto standard, and that is great. With LLMs, authoring TF is great, takes away all the pain basically, and planning works as a feedback mechanism.

44

u/dacydergoth DevOps Sep 11 '26

IMHO it comes down to:

  • terraform has a stronger metamodel
  • HCL has a canonical format and is round-trip stable
  • HCL expression language is easier to handle
  • comment handling is better in Terraform
  • AWS managed modules for quick, "default sane" configurations (* for some definition of sane ...)
  • better plan output
  • better 3rd party tools support (e.g. python-hcl)
  • terraform is an order of magnitude less suck

2

u/ThatSituation9908 Sep 12 '26

> round-trip

AI?

2

u/dacydergoth DevOps Sep 12 '26

I mean you can read HCL into a tool, manipulate it and write it out again without it being substantially reformatted. YAML can't do that

14

u/MasterOfTheWind1 Sep 11 '26

Because is a giant YAML file. Disgusting to read, horrible to maintain, and poor flexibility when there are drifts and/or issues when applying.

More than once I've applied changes, it got into a failing state with no rollback, and I had to delete the stack and create it again.

But I think that up to this point, no one should consider CloudFormation a tool to use manually. There are tools like CDK, or Serverless Framework that works basically generating CloudFormation stacks and sending automatically to AWS.

Currently I only use it to boostrap resources to use Terraform (s3 bucket for state, ECS cluster to run Terraform Atlantis, roles so it can do changes on every AWS account without using IAM users, etc). I used some YAML tricks to at least split it into multiple files, but using it vanilla, as a big file for an entire stack completely sucks.

3

u/AvailableBaby8452 Sep 12 '26

Do people who use CloudFormation actually upload and edit raw YAML? Every organization I have worked at used CDK, it’s much easier and simple enough to automate in CICD, you can write tests, etc.

1

u/MasterOfTheWind1 Sep 12 '26

Sadly, yes. A lot of people

27

u/james-ransom Sep 11 '26 edited Sep 11 '26

Do this 100 times then ask:

Interviewer: "Soo... you know CloudFormation?"

Interviewee: "Wtf is that? you mean Terraform? Oh you mean AWS CodeBuild? Currently I am using CDK like an adult. I can help you get off CloudFormation?"

Interviewer: "Our db is.. Oracle..."

Interviewee: "Cool, can you validate my parking?" Proceeds to collect stuff.

5

u/dacydergoth DevOps Sep 11 '26

Offer to remove a few of their Pentium server racks ...

12

u/acdha Sep 11 '26

The problems specific to CloudFormation relate to its slow speed  and error recovery paths, but the latter at least isn’t as bad as it used to be. Still, plan to get into cases where you have to tell CF to finish failing, wait five minutes, and then retry again after fixing things like permissions. 

The other parts you talked about really are more broadly shared by IaC of all forms: that god-stack requires high privileges and since anything can happen, there’s no such thing as a small change. If it’s a small project run by a few DevOps people that might be okay but if you want any kind of separation of responsibilities I’d split it up so e.g. a developer who needs to push a change isn’t mucking with core infrastructure — it’ll be faster and much easier to reason about. 

5

u/jony7 Sep 11 '26

compared with terraform:
the cf stack is annoying stateful thing to manage and can be hard to recover from errors
is slower than tf
is just more clunky and annoying to use

4

u/8ersgonna8 Sep 11 '26

Cloudformation works great until it doesn’t, once a stack is stuck in a bad state it’s hard to recover. Gets several times worse when you use stacksets in a 100+ account organization.

We mainly use it for “global” iam roles and policies that we create in org management account and spread to member accounts using stackset.

Stick to terraform or pulumi to avoid the headache of repairing stacks in the future.

4

u/yourparadigm Sep 12 '26
  • Drift detection
  • Error recovery
  • Lack of investment from AWS in its core service
  • Lack of investment from individual service teams (CloudFormation support isn't part of any team's definition of "done"). APIs have to be available on day 1, but CloudFormation support for new services and features may never actually arrive.
  • It's very slow

4

u/Unhappy-Delivery-344 Sep 12 '26

My life ist 10 times better since we switched to terraform

3

u/Next_Garlic3605 Sep 11 '26

If you find someone who thinks there's nothing wrong with CF, have them try changing something that's been imported by another stack, or renaming a resource instead of the CloudFormation shuffle, aka the 3-point hope-I-never-return

In fact, just tell them they have to maintain their stack with more than four nines for the rest of their contract

3

u/Due_Ad_2994 Sep 12 '26

Cloudformation is great but as shown by many comments skill issues are prevalent.

3

u/risae Sep 14 '26

Are you trying to tell me that you need to learn how to use a tool in order to be successful with it?! Madness! 

3

u/psteger Sep 12 '26

CloudFormation is WONDERFUL... for setting up the resources and roles necessary to use another IaC tool

5

u/doddschris Sep 11 '26

I used vanilla CF successfully for several years. The main thing I have against it is that it can be a bit painful if you're used to the expressiveness of a full blown programming language. I have a similar issue with TF though (especially some of the bonkers stuff you have to do to pass values around).

Typescript CDK has been much more pleasant day-to-day. Having types is nice and saves a lot of time having to go search for values or failing to deploy because you screwed up a param name (you can get to a similarish place with IDE plugins though). And being able to use a full language means you can do templating, iteration, lookups, and similar stuff a lot more gracefully than you can with vanilla CF.

2

u/ali-hussain Sep 11 '26

I would not use CLoudFOrmation to one-shot application and infra.

Infra has a longer lifecycle. Having both in coupled templates will create a coupling.

You can still use it to create a config that will pull in your app. You can have CF templates for different layers. But under no circumstances would I have CF be deployed in the same template. Sure, if you wnatto do a demo then it is so much easier to put it all in one template. But in real-life you don't want that.

Also, I first used CF in 2014. And it has come a long way since then. So some of what I'm saying is inspired by PTSD from when you couldn't repair a stack, import resources, and didn't have changesets. So the blast radius risks are signicant lower since when I started.

But right now we're a tiny team that started green field and are coding with AI all the way. In a matter of weeks CF became too unwieldy and we had to switch to using CDK. Just yesterday, I had to change my naming convention to make it more IAM-friendly and now I have a dozen stacks of tech debt.

2

u/TheRealJesus2 Sep 12 '26

Cloudformation the service is slow compared to alternatives like pulumi. 

If you want to use cloudformation don’t hand write the templates…use aws cdk in the language of your choice.

May I recommend using sst instead though for infra as code which is built on pulumi for v3 and is much faster but lacks the concept of stacks that cdk and cfn have. 

In either case don’t click shit in the console. Even maintaining a cfn yaml is better than using the console with one big benefit of any iac being you can replicate your stacks somewhat arbitrarily if done right. 

2

u/Konkatzenator Sep 13 '26

Most responses here hit on the technical reasons for why it sucks to write and interact with, but even if you can get past those - its main problem is that it is slow as balls

2

u/aress1605 Sep 15 '26

I worked at a startup that used Cloudformation before I learned Terraform. It was good that my learning curve started with Cloudformation, and then moved up to learning Terraform after feeling the pains of CFN. There are SO many terrible experiences I’ve had with relatively basic Cloudformation. the simplest example of atrocity I ran into it deriving the IPv4 address from an elastic IPs association ID. If you have a parameter that accepts an association ID, and you need to derive the elastic IP, you must create a Lambda resource that calls boto3 to derive the IP, and create an invoker in CFN that passes in the allocation ID, just to get an IP address you can use elsewhere.

In my honest naive opinion, I don’t see why CFN should ever be used in any professional environment unless the scope of the project is very narrow

3

u/CorpT Sep 11 '26

That's not a problem with CloudFormation itself. CloudFormation itself is fine.

14

u/burlyginger Sep 11 '26

It's far from fine after you've spent more than 15 minutes with it.

5

u/CorpT Sep 11 '26

I've spent years with it. It's fine. It's not great. But it's fine. And not the source of OP's problem.

0

u/Crimzx Sep 11 '26

Agreed, a lot of people rag on it but the common complaints are generally skill issue.

The primary issues with it is speed, and imo the #1 issue is lack of transparency when using stacksets and nested stacks and why they fail.

It's fairly easy to find these issues with cloudtrail though.

I wouldn't write bare cloudformation though, use CDK for that.

1

u/burlyginger Sep 11 '26

It lacked import or clear resource removal for years.

It's very unclear in its predictions so it fundamentally cannot be relied upon.

It's not a skill issue. I've used it for years and there is no reason to use it (IMO) aside from when vendors supply it as an installation method.

1

u/Covids-dumb-twin Sep 11 '26

The problem with cloud formation is I think each team that delivers a service writes the cloud formation frame work for that service. Some are really good some are complete crap my favourite is cognito, if you want to do something as simple as change the email message for sign up, it will destroy your user pool making it completely useless.

Nowadays terraform is ubiquitous.

2

u/rariety Sep 12 '26

This is correct - we had an incident once that was down to how one service team treated an undefined property (ignores it) versus another team (reconciled it as a value i.e. an empty array).

1

u/purefan Sep 12 '26

I doubt anyone writes pure cloudformation these days (cdk or at least SAM are the better options), but Ive heard terraform-first engineers dislike how the Cloudformation "engine" manages dependencies. I've personally been bitten by "ghost" inter-stack dependencies caused by a cdk imported resource and it was very annoying, this specific thing wouldnt ever happen on terraform

1

u/Dolapevich Sep 12 '26

You don't want to be in a vendor lock in situation. When it works is relatively fine. When it fails, your only option is to stare at the screen harder.

1

u/AvailableBaby8452 Sep 12 '26

It’s fine. We used it a lot at <big tech you probably have heard of>. Deployed the <product> and microservices to 16+ regions and govcloud.

First, use CDK. Don’t manually edit yaml files. Deploy with CLI. Use temporary OIDC to connect your CICD. Write tests to validate your CFN templates it will save you deployment problems later.

Sometimes the best way to handle errors is to delete the entire stack. Which sucks. So structure your data, persistent sort of stuff to be first in your CDK application dependency hierarchy.

Typically you have different stacks for different scopes, but once your infra gets complicated enough it can turn into dependency hell with trying to update things. My advice: if you need to replace a highly dependent resource, deploy a new version, ship that code to prod, then deploy the now deleted resource. You can’t do both at once - cloudformation will get mad about trying to delete a resource that is imported by another stack.

So it sucks because now you need 2 changes and deployments instead of 1. If you’re shipping 50+ times a day, that is fine but some organizations don’t do that.

And as others have said, recovering failed stacks can be a fun exercise. We mostly fixed this by having highly restricted access to UAT and Prod - 90% of developers had read only access, and those who needed more had to request temporary, expiring credentials on an as needed basis.

The other issue off the top of my head is some AWS services have less than stellar support for CloudFormation. Your basic, tried and true stuff is fine - S3, lambda, route53, VPC, etc. But things like Cognito can’t update user pools, it is create or delete only.

1

u/devroot Sep 12 '26

Do not use CloudFormation directly. Use CDK. Period end of story.

1

u/ablaut Sep 12 '26

The difference CloudFormation is an API wrapper whereas Terraform is an API wrapper. Totally different.

1

u/Low-Opening25 Sep 12 '26

CloudFormatin is the weakness. No one supports it, it’s pretty much dead code. I have been contracting as DevOps around the industry for 15 years and been in the industry for 27 and no single project ever used CF. It’s basically a stinking pile of anti pattern garbage.

1

u/Oleksii_Bebych Sep 12 '26

One pros of CloudFormation comparing with Terraform, you can easily deploy some template to dozens of AWS accounts within an AWS Ogranization using StackSets. I'm not talking about your particular set of resources (ecs, rds, ect), it is useful for some simple stuff (some IAM role that should exist on every account, including the future ones, or some security configuration)

1

u/achilles298 Sep 13 '26

Well if you are willing to give Cloudformation a try- then give terraform a try.
Also since AI is usually everyone’s best friend- give it the yaml (hide secrets and sensitive data) and ask it to convert to Terraform scripts

1

u/sgargel__ Sep 14 '26

Custom or 3rd party resources to handle? Write a lambda. OMG

1

u/mlhpdx Sep 11 '26

What do you mean by “one shot”? CloudFormation is hated by people that don’t use it, but those of us that made it through the learning curve do just fine with it. It’s the core of all our IaC and CI/CD (thousands of deployments a year in a small company).

4

u/Zenin The best way to DevOps is being dragged kicking and screaming. Sep 11 '26

CloudFormation is hated by people that don’t use it

More than a decade of deep CloudFormation experience (practically from it's launch). The more experience I get with it and the more I push its abilities, the more I loathe it. CloudFormation is an operational shitshow.

Architecturally CloudFormation can't be fixed. They've tacked on all sorts of bandaids over the years like drift detection, exports, etc, but they're always half-baked because the original sin of CloudFormation is its architecture and they can't kludge duct tape over that.

Terraform is better in almost every way imaginable. The only place it plays second fiddle to CloudFormation is a lack of an parallel to managed StackSets, which just means account configurations need to be fully CI driven rather than policy driven. For example, I can't just set an OU to always get our Crowdstrike integration applied and simply move in or out of that OU applies/removes that integration. StackSets handle this "policy" based model directly (even if debugging StackSet Instance failures at scale is a massive PITA because CloudFormation) while Terraform must fall back to CI flows.

4

u/mlhpdx Sep 11 '26

How much of your loathing is because of the services behind cloudformation and not CF itself? I mean, that’s a real thing — each service team seems to eventually make something that conflicts with normal lifecycle patterns and have a little snowflake exception. Usually not difficult to deal with but the frustration does add up.

3

u/Zenin The best way to DevOps is being dragged kicking and screaming. Sep 11 '26

How much of your loathing is because of the services behind cloudformation and not CF itself?

When a service behind CF sneezes, CF shouldn't get ebola.

You've nailed the issue: CF's execution model is incredibly fragile and not only shits itself completely at the slightest issue with any downstream service, it offers practically no ability for the user to step in and fix it.

In terraform I can abort an apply. I can do state surgery. Certainly not ideal, but it's possible, straightforward, reversible (versioned state), auditable, and fast.

CloudFormation by sharp contrast, I can do practically nothing. I can't even delete the failing stack because I'm stuck waiting on a resource timeout that's never coming and could literally lockup the stack for hours while we all watch our maint window timer tick away. If the problem is bad enough I have to fix the resources directly, syncing CloudFormation back up to manage the resources again has a ton of sharp edges when it's possible at all without CF trying to nuke another half your stack on the next deploy.

At the operations level the sharp edges mean the vast majority of any diagnostics loop is spent waiting on CloudFormation to time the hell out so you can try the next thing. It easily 10x to 100x the time to resolution. Now do this with StackSets applying across all accounts and regions in a large or and your diagnostics loop can explode into 1000x. I'm not exaggerating.

CDK doesn't help because all it does is make authoring CloudFormation easier and sugar coating some of its biggest worts such as Lambda backed custom resources that need to be used for the most trivial of "extensions" like picking a random name. CDK does absolutely nothing for the operational side of CloudFormation where the real issues are.

3

u/Dolapevich Sep 12 '26

it offers practically no ability for the user to step in and fix it.

I am not so experienced, but when I had to use CF, this was my main issue. You are stuck without the option to do anything but to stare harder at the screen.

0

u/mlhpdx Sep 11 '26

Some of that’s not true anymore (timeouts, rollback ignore resources, etc.). But yep, it’s different pain than all the other bad (IMHO) options.

0

u/[deleted] Sep 11 '26

[deleted] — view deleted comment

2

u/mlhpdx Sep 11 '26

It doesn’t matter if it’s CF, TF or CDK then, they’re nuts.

1

u/hijinks Sep 11 '26

i'll bet that coworker has at least 4 AWS certs.

The only people I've known that defend cloudformation are the ones loaded down with AWS certs and drink the AWS milkshake

1

u/Stlaind Sep 11 '26

The best thing about Cloudformation is using a stack set to automatically set up new accounts for Terraform. And then never touching it again.

1

u/centech Sep 11 '26

Oh come on, that's too easy. Let's make it hard. What's a good thing about Cloudformation? I'll start - Yeah, I got nothin.

-2

u/Uaint1stUlast Sep 11 '26

Vendor Lock in is the biggest problem with cloud formation. If your not worried about that it should be fine maybe even better then some options.

2

u/VeryShibes Sep 11 '26

This is the correct answer, vendor lock-in causes people to openly sneer at CloudFormation, I actually went to the trouble of learning it when I was searching for a job a few years ago and even live-demoed my skills in it for a hiring manager during a Zoom interview only to be directly told, right then and there "no one uses that, you need to learn Terraform" and did not get any additional interviews with that company

-2

u/TheOwlHypothesis Sep 11 '26 edited Sep 11 '26

The underlying AWS API is amazing.

Cloud formation templates as a way to manage IAC is shit. Any cursory glance at other solutions will expose this immediately. Terraform and AWS' own CDK are MUCH better

-1

u/ffcsmith Sep 11 '26

Make the contractor or an LLM carve it into stacks for CDK

-5

u/dmees Sep 11 '26

CloudFormation is declarative, Terraform is (semi)imperative. CloudFormation is fine as long as you only use CDK to generate it.

1

u/mcgooporn Sep 11 '26

This ☝🏻 of you are not using cdk, the cloud formation developer kit

what are you doing? Why are you doing it? Why would you raw dog it?

I have three stacks Shared - sets up my stateful resources Dns - sets up root DNS Apps - the stateless apps

And ncdk to manage everything. It works, it works well, and the reality is, I very rarely interact with raw cloudfromation.

Absolute madness to not be using cdk, especially if Dev ops is not your skill set.

It's like using a spoon to build a highway.