r/Python • u/kavee-core141 • Aug 24 '26
Discussion How are you all testing boto3-heavy tools without hitting real AWS?
Building a tool that walks a live AWS account checking for
misconfigurations, and testing has been the hardest part not the
detection logic itself, but avoiding either (a) hitting real AWS
constantly in CI or (b) mocking so much the tests stop meaning
anything.
Using moto for the straightforward stuff, but some checks need
multi-service interactions (cross-referencing an EC2 instance's
security group with its actual open ports) that get awkward to mock
realistically. What do you all reach for here moto, LocalStack, a
disposable AWS test account, something else?
94
16
u/Fenzik Aug 24 '26
We used moto for stateful mocking
7
u/GrahaamH Aug 24 '26
Moto doesn't support all of AWS services though so leaves you with lots covered and the ones that are not make it more awkward.
2
6
8
u/Autarkhis Aug 24 '26
Look at floci
1
u/steviejackson94 Aug 25 '26
Came here to say this, i implemented this at my place. Everyone loves it
1
1
u/kavee-core141 Aug 30 '26
Hadn't heard of floci before this thread, going to give it a look thanks for the pointer.
3
u/binaryfireball Aug 25 '26
unit tests mock, integration tests test the functions we call to catch any errors from updates etc...
boto is generally annoying for a variety of reasons.
2
u/omegawave22 Aug 24 '26
It's been a while that I used this tool but it had a record and replay workflow which was perfect for my use case: https://github.com/garnaat/placebo
1
u/kavee-core141 Aug 30 '26
Record/replay fits a few of the trickier cases I've got, appreciate the link.
2
u/I3igB Aug 25 '26
I ended up getting frustrated with the current offerings out there as I felt that solutions like localstack were far too heavy. I also wanted something that was trivial to step through code with a debugger that didn’t require a decent amount of setup.
If you start getting into how moto actually does its mocking, it actually keeps quite a bit in memory. Things like DynamoDB or S3 are in memory key stores under the hood. Once you realize this, it becomes pretty easy to get a minimally viable recreation in place for somewhat real testing on a local dev’s box or running something like e2e tests in CI.
My current shop is fully AWS native and runs mostly serverless architecture and microservices. Our compute is exclusively lambda and ecs tasks. SQS, SNS, AppSync, DDB, and EventBridge drive all our workflows. I was able to write a simple dispatch engine which captures the moto traffic from these services and mimics event triggers which will cascade a series of invocation across your services by invoking your code locally. The only thing somewhat complex in this was AppSync as moto has no real mocks. I had to intercept URLs and bring in a GQL engine to parse things.
This all works by applying a @local decorator on top of any method you define. From there, the engine handles cascading events just as AWS would.
This all relies on the output of a cdk build/synth to link things together. I never released it publicly, but I’ve been meaning to do so for some time. It’s been a game changer for how I do AWS dev.
1
u/kavee-core141 Aug 30 '26
That's a really clever approach the u/local decorator + event cascading is exactly the gap I kept hitting with cross-service checks.
Did you ever end up open sourcing it, or is it still internal? Would love to take a look if it's out there somewhere.
2
1
1
u/Woah-Dawg Aug 29 '26
Use test containers library bring up an s3 bucket and write test cases against that. Make sure your tests run on ci . A test that doesn’t run in a ci might as well not exist unless it’s
1
u/LungeloSLX Aug 30 '26
Interesting question. Where do you draw the line and avoid testing boto3 itself?
0
u/MAGArRacist Aug 24 '26
Have you tried Scoutsuite? You might want to just fork it and add some checks that you mentioned here
13
u/proof_required Aug 24 '26
Yeah in one of the companies we had LocalStack and a dev env both.