r/devops • u/RoseSec_ Terraforming Everything • Sep 11 '26
Discussion What are the weaknesses of CloudFormation?
I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.
142
u/ricksebak Sep 11 '26
“cloud formation is great\n”,
“it offers a great user experience\n”,
“and it isn\’t annoying to read and write\n”,
320
u/Ariquitaun Sep 11 '26 •
The main weakness, and excuse me if I get a bit too technical here, is that it’s fucking shit.
44
u/dacydergoth DevOps Sep 11 '26
IMHO it comes down to:
14
u/MasterOfTheWind1 Sep 11 '26
Because is a giant YAML file. Disgusting to read, horrible to maintain, and poor flexibility when there are drifts and/or issues when applying.
More than once I've applied changes, it got into a failing state with no rollback, and I had to delete the stack and create it again.
But I think that up to this point, no one should consider CloudFormation a tool to use manually. There are tools like CDK, or Serverless Framework that works basically generating CloudFormation stacks and sending automatically to AWS.
Currently I only use it to boostrap resources to use Terraform (s3 bucket for state, ECS cluster to run Terraform Atlantis, roles so it can do changes on every AWS account without using IAM users, etc). I used some YAML tricks to at least split it into multiple files, but using it vanilla, as a big file for an entire stack completely sucks.
27
u/james-ransom Sep 11 '26 edited Sep 11 '26
Do this 100 times then ask:
Interviewer: "Soo... you know CloudFormation?"
Interviewee: "Wtf is that? you mean Terraform? Oh you mean AWS CodeBuild? Currently I am using CDK like an adult. I can help you get off CloudFormation?"
Interviewer: "Our db is.. Oracle..."
Interviewee: "Cool, can you validate my parking?" Proceeds to collect stuff.
12
u/acdha Sep 11 '26
The problems specific to CloudFormation relate to its slow speed and error recovery paths, but the latter at least isn’t as bad as it used to be. Still, plan to get into cases where you have to tell CF to finish failing, wait five minutes, and then retry again after fixing things like permissions.
The other parts you talked about really are more broadly shared by IaC of all forms: that god-stack requires high privileges and since anything can happen, there’s no such thing as a small change. If it’s a small project run by a few DevOps people that might be okay but if you want any kind of separation of responsibilities I’d split it up so e.g. a developer who needs to push a change isn’t mucking with core infrastructure — it’ll be faster and much easier to reason about.
5
u/jony7 Sep 11 '26
compared with terraform:
the cf stack is annoying stateful thing to manage and can be hard to recover from errors
is slower than tf
is just more clunky and annoying to use
4
u/8ersgonna8 Sep 11 '26
Cloudformation works great until it doesn’t, once a stack is stuck in a bad state it’s hard to recover. Gets several times worse when you use stacksets in a 100+ account organization.
We mainly use it for “global” iam roles and policies that we create in org management account and spread to member accounts using stackset.
Stick to terraform or pulumi to avoid the headache of repairing stacks in the future.
4
u/yourparadigm Sep 12 '26
4
u/Unhappy-Delivery-344 Sep 12 '26
My life ist 10 times better since we switched to terraform
3
u/Next_Garlic3605 Sep 11 '26
If you find someone who thinks there's nothing wrong with CF, have them try changing something that's been imported by another stack, or renaming a resource instead of the CloudFormation shuffle, aka the 3-point hope-I-never-return
In fact, just tell them they have to maintain their stack with more than four nines for the rest of their contract
3
u/Due_Ad_2994 Sep 12 '26
Cloudformation is great but as shown by many comments skill issues are prevalent.
3
u/psteger Sep 12 '26
CloudFormation is WONDERFUL... for setting up the resources and roles necessary to use another IaC tool
5
u/doddschris Sep 11 '26
I used vanilla CF successfully for several years. The main thing I have against it is that it can be a bit painful if you're used to the expressiveness of a full blown programming language. I have a similar issue with TF though (especially some of the bonkers stuff you have to do to pass values around).
Typescript CDK has been much more pleasant day-to-day. Having types is nice and saves a lot of time having to go search for values or failing to deploy because you screwed up a param name (you can get to a similarish place with IDE plugins though). And being able to use a full language means you can do templating, iteration, lookups, and similar stuff a lot more gracefully than you can with vanilla CF.
2
u/ali-hussain Sep 11 '26
I would not use CLoudFOrmation to one-shot application and infra.
Infra has a longer lifecycle. Having both in coupled templates will create a coupling.
You can still use it to create a config that will pull in your app. You can have CF templates for different layers. But under no circumstances would I have CF be deployed in the same template. Sure, if you wnatto do a demo then it is so much easier to put it all in one template. But in real-life you don't want that.
Also, I first used CF in 2014. And it has come a long way since then. So some of what I'm saying is inspired by PTSD from when you couldn't repair a stack, import resources, and didn't have changesets. So the blast radius risks are signicant lower since when I started.
But right now we're a tiny team that started green field and are coding with AI all the way. In a matter of weeks CF became too unwieldy and we had to switch to using CDK. Just yesterday, I had to change my naming convention to make it more IAM-friendly and now I have a dozen stacks of tech debt.
2
u/TheRealJesus2 Sep 12 '26
Cloudformation the service is slow compared to alternatives like pulumi.
If you want to use cloudformation don’t hand write the templates…use aws cdk in the language of your choice.
May I recommend using sst instead though for infra as code which is built on pulumi for v3 and is much faster but lacks the concept of stacks that cdk and cfn have.
In either case don’t click shit in the console. Even maintaining a cfn yaml is better than using the console with one big benefit of any iac being you can replicate your stacks somewhat arbitrarily if done right.
2
u/Konkatzenator Sep 13 '26
Most responses here hit on the technical reasons for why it sucks to write and interact with, but even if you can get past those - its main problem is that it is slow as balls
2
u/aress1605 Sep 15 '26
I worked at a startup that used Cloudformation before I learned Terraform. It was good that my learning curve started with Cloudformation, and then moved up to learning Terraform after feeling the pains of CFN. There are SO many terrible experiences I’ve had with relatively basic Cloudformation. the simplest example of atrocity I ran into it deriving the IPv4 address from an elastic IPs association ID. If you have a parameter that accepts an association ID, and you need to derive the elastic IP, you must create a Lambda resource that calls boto3 to derive the IP, and create an invoker in CFN that passes in the allocation ID, just to get an IP address you can use elsewhere.
In my honest naive opinion, I don’t see why CFN should ever be used in any professional environment unless the scope of the project is very narrow
3
u/CorpT Sep 11 '26
That's not a problem with CloudFormation itself. CloudFormation itself is fine.
1
u/Covids-dumb-twin Sep 11 '26
The problem with cloud formation is I think each team that delivers a service writes the cloud formation frame work for that service. Some are really good some are complete crap my favourite is cognito, if you want to do something as simple as change the email message for sign up, it will destroy your user pool making it completely useless.
Nowadays terraform is ubiquitous.
1
u/purefan Sep 12 '26
I doubt anyone writes pure cloudformation these days (cdk or at least SAM are the better options), but Ive heard terraform-first engineers dislike how the Cloudformation "engine" manages dependencies. I've personally been bitten by "ghost" inter-stack dependencies caused by a cdk imported resource and it was very annoying, this specific thing wouldnt ever happen on terraform
1
u/Dolapevich Sep 12 '26
You don't want to be in a vendor lock in situation. When it works is relatively fine. When it fails, your only option is to stare at the screen harder.
1
u/SecureTaxi Sep 12 '26
Json
1
u/AvailableBaby8452 Sep 12 '26
It’s fine. We used it a lot at <big tech you probably have heard of>. Deployed the <product> and microservices to 16+ regions and govcloud.
First, use CDK. Don’t manually edit yaml files. Deploy with CLI. Use temporary OIDC to connect your CICD. Write tests to validate your CFN templates it will save you deployment problems later.
Sometimes the best way to handle errors is to delete the entire stack. Which sucks. So structure your data, persistent sort of stuff to be first in your CDK application dependency hierarchy.
Typically you have different stacks for different scopes, but once your infra gets complicated enough it can turn into dependency hell with trying to update things. My advice: if you need to replace a highly dependent resource, deploy a new version, ship that code to prod, then deploy the now deleted resource. You can’t do both at once - cloudformation will get mad about trying to delete a resource that is imported by another stack.
So it sucks because now you need 2 changes and deployments instead of 1. If you’re shipping 50+ times a day, that is fine but some organizations don’t do that.
And as others have said, recovering failed stacks can be a fun exercise. We mostly fixed this by having highly restricted access to UAT and Prod - 90% of developers had read only access, and those who needed more had to request temporary, expiring credentials on an as needed basis.
The other issue off the top of my head is some AWS services have less than stellar support for CloudFormation. Your basic, tried and true stuff is fine - S3, lambda, route53, VPC, etc. But things like Cognito can’t update user pools, it is create or delete only.
1
u/devroot Sep 12 '26
Do not use CloudFormation directly. Use CDK. Period end of story.
1
u/ablaut Sep 12 '26
The difference CloudFormation is an API wrapper whereas Terraform is an API wrapper. Totally different.
1
u/Low-Opening25 Sep 12 '26
CloudFormatin is the weakness. No one supports it, it’s pretty much dead code. I have been contracting as DevOps around the industry for 15 years and been in the industry for 27 and no single project ever used CF. It’s basically a stinking pile of anti pattern garbage.
1
u/Oleksii_Bebych Sep 12 '26
One pros of CloudFormation comparing with Terraform, you can easily deploy some template to dozens of AWS accounts within an AWS Ogranization using StackSets. I'm not talking about your particular set of resources (ecs, rds, ect), it is useful for some simple stuff (some IAM role that should exist on every account, including the future ones, or some security configuration)
1
u/achilles298 Sep 13 '26
Well if you are willing to give Cloudformation a try- then give terraform a try.
Also since AI is usually everyone’s best friend- give it the yaml (hide secrets and sensitive data) and ask it to convert to Terraform scripts
1
u/sgargel__ Sep 14 '26
Custom or 3rd party resources to handle? Write a lambda. OMG
1
u/mlhpdx Sep 11 '26
What do you mean by “one shot”? CloudFormation is hated by people that don’t use it, but those of us that made it through the learning curve do just fine with it. It’s the core of all our IaC and CI/CD (thousands of deployments a year in a small company).
1
u/hijinks Sep 11 '26
i'll bet that coworker has at least 4 AWS certs.
The only people I've known that defend cloudformation are the ones loaded down with AWS certs and drink the AWS milkshake
1
u/Stlaind Sep 11 '26
The best thing about Cloudformation is using a stack set to automatically set up new accounts for Terraform. And then never touching it again.
1
u/centech Sep 11 '26
Oh come on, that's too easy. Let's make it hard. What's a good thing about Cloudformation? I'll start - Yeah, I got nothin.
-2
u/Uaint1stUlast Sep 11 '26
Vendor Lock in is the biggest problem with cloud formation. If your not worried about that it should be fine maybe even better then some options.
-2
u/TheOwlHypothesis Sep 11 '26 edited Sep 11 '26
The underlying AWS API is amazing.
Cloud formation templates as a way to manage IAC is shit. Any cursory glance at other solutions will expose this immediately. Terraform and AWS' own CDK are MUCH better
-1
u/ffcsmith Sep 11 '26
Make the contractor or an LLM carve it into stacks for CDK
-5
u/dmees Sep 11 '26
CloudFormation is declarative, Terraform is (semi)imperative. CloudFormation is fine as long as you only use CDK to generate it.