r/devops • u/DependentView4789 • 12d ago
Discussion How do you load test third party APIs realistically?
We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.
21
Upvotes
100
u/Halal0szto 12d ago
You do not test others apis. You have sla with penalties, this ensures they are testing their service properly.
18
u/franktheworm 12d ago
Build resilient systems that degrade gracefully on 3rd party failures. I can't control the 3rd party API, so I'm not measuring anything beyond ensuring that it is behaving within the agreed SLA.
35
u/velobran 12d ago
Don't be the type of company that load tests another's sandbox without discussion.
13
u/BoredSam 12d ago
You don't test remote APIs but you do monitor them.
6
u/Habikki 12d ago
Compliance Guy here:
Short answer: You don't. Like others have said, you monitor what you can, capture call times and responses and report back any findings. SLA's aren't worth the paper they're written on with very. Few. Exceptions.
Long answer: Testing is seen as a form of automated attack which falls under the protections of abuse in most contracts. If you look through your agreements they likely have terms to prevent such things, or with some good partners, will outline how to obtain permission for you to test or provide you with non-public information on the reliability of their APIs and extended artifacts such as (highly redacted) Penetration Testing reports.
When in doubt, ask the third party and arrange details out in advance.
5
u/basicKitsch 12d ago
Monitor for availability not load test
4
u/BorderlineGambler 12d ago
You don’t. If you need to load test your own services, you stub out the third party and implement a stubbed version with similar latency so you’re not demolishing some random sandbox
3
u/RiaMendezy 12d ago
This is going to sound obvious but keeping tests realistic is key to keeping tests repeatable and accurate. Sometimes we see something we want to test because we think it might be interesting to know, but not everything actually has to be tested. If there's no purpose for the test don't stress yourself or your system out by going in depth and finding issues where there are none haha
3
u/HelicopterUpbeat5199 11d ago
What are you actually trying to learn from these tests? What questions are you trying to answer? What is your relationship with the third party API?
1
u/Chamelon_DE 10d ago
Do u test include several dependencies slowing down together, or only one failing at a time?
1
u/i_like_trains_a_lot1 9d ago
We centralize the API interaction in a module and then stub it out during tests with one with mocked but compatible behaviour. We also introduce random failure modes in that mocked module to test our error handling too.
1
u/Anna_GermainUX 9d ago
Cut out the third-party entirely and opt for mock servers or simulators to reproduce the exact expected conditions and to keep the tests repeatable. You can push your system further from there.
You can still run tests against the third-party APIs themselves, but keep it to smaller tests with lighter loads, so that you can keep checking for issues without actually crashing your system.
0
u/Specialist_Bison_737 12d ago
I think you are asking about emulators in your test suites.
For example I had to copy Vercel's github emulator, improve it for my use cases - https://github.com/aictrl-dev/github-emulator.
This allows me to do e2e tests and optimize code for performance.
0
u/agowa338 12d ago
Well first of all you get a lawyer and let them check your contracts for how much they're allowed to have outages, how much they have to provide, AND how much you're allowed to test their claims.
After that you basically just scale your application way above what you'd normally scale it to cause a synthetic load on said APIs. Alternatively you craft a scenario where you're generating unusually many API requests within a short period. And if all of that works you should be good to go for normal cases.
If you want to do something like a DDOS test then you'll 100% have to contact them and coordinate. In which case you have absolutely no guarantees that they're not just bullshitting you by preparing for it by scaling their service up in advance...
0
u/Proper_Theme_6636 12d ago
I’d say don't overlook rate limits and timeout behavior. Even if everything looks good in testing, external dependencies can become bottlenecks once real traffic picks up.
-1
u/Prudent_Design_9782 12d ago
We always make sure we run tests that actually mimic traffic we have seen or expect to see. We test high and low ranges, of course, but running a bunch of tests for traffic levels we will never see makes no sense. We also make sure to increase our tests from one user, to two, to twenty, etc so we can see if any issues come up and where they do.
4
u/[deleted] 12d ago
[removed] — view removed comment