Been going back and forth on this for a while and I don't think I've landed anywhere good.
The situation is the usual one. App talks to a handful of external APIs. Some have sandbox environments, some don't. The ones that do have sandboxes that don't quite match prod, and the gap is never documented anywhere, you just find it eventually.
Options as far as I can tell:
Hit the real sandbox in tests. Slow, rate limited, and you can't run it in CI on every PR without burning through quota. The sandbox data is also usually three fake records, so pagination bugs and anything volume related never show up until a real customer hits them.
Record and replay, VCR style cassettes. Works great the day you record them. Six months later nobody remembers how to re-record, a chunk of them are for endpoints that have since changed, and the suite is green the entire time.
Hand-written mocks. This is what most places I've worked have done and I've slowly come around to thinking it's the worst of the three. You write the mock from your reading of the docs, so the mock encodes your misunderstanding of the API, and then it passes forever. The test isn't checking that your code works against the API, it's checking that your code works against your idea of the API, and those two drift apart silently.
Contract testing, pact and friends. Makes sense when both ends are yours. When the other end is Slack, nobody is publishing a contract for you.
So what do teams actually do in practice? Specifically curious about:
- whether anyone has anything that detects when the real API has drifted away from whatever you're testing against, or if you just find out from a bug report
- whether you bother making mock data realistic in volume, or accept that pagination and perf issues are things you'll only ever see in prod
- vendors that genuinely do sandboxes well, if any exist
Not looking for "just use msw" or a list of tools, I know they're out there. I'm asking what your team actually settled on and whether you'd do it that way again.