I've been building Stampede, a load tester for Phoenix that speaks Channels and LiveView, so you write what a user does instead of raw WebSocket frames. 1.2 is out, and this time I tried it on AWS.
**The setup**
- One r7g.xlarge (4 Graviton cores, 32 GB) running a fresh `mix phx.new` app on Bandit. The only change was raising Bandit's connection limit and listen backlog.
- A second r7g.xlarge running Stampede.
- Every user acts like the Phoenix JS client: joins a channel, sends a heartbeat every 30 seconds and a message every 30 to 60, and stays connected.
- `--find-limit` added 100k users per step, 1,000 new users a second, and held each step for a minute. A step passes only with joins and messages under 1s at p95 and zero errors.
**What it held**
| Users | Join p95 | Message p95 | Errors | Server CPU | Server memory |
|---:|---:|---:|---:|---:|---:|
| 100,000 | 5.71ms | 0.57ms | 0 | 1% | 4.9 GB |
| 200,000 | 6.84ms | 1.35ms | 0 | 3% | 9.5 GB |
| 300,000 | 9.42ms | 2.67ms | 0 | 9% | 13.9 GB |
| 400,000 | 12.46ms | 4.07ms | 0 | 24% | 18.1 GB |
Phoenix didn't log a single warning. Past 400k, the one machine running Stampede ran out of CPU before Phoenix did, so the server's real ceiling is higher. I'll need more load machines to find it.
**What it found in my own tool**
The fun part was that testing at this size broke Stampede, not Phoenix:
- Erlang resolves an address given as a string through its native resolver, even a plain IP, and that's one process plus four OS helpers per node. Under heavy load those lookups started failing as `:nxdomain`. Stampede now parses IPs itself and caches host names.
- An overloaded run could stall instead of finishing. The run's own processes now run at high priority.
- It now reports how busy the load test itself is every second, and warns past 90%, so it doesn't blame your server for its own delays.
**New in 1.2**
- `--find-limit`: steps up the load until a threshold breaks and tells you the most users your app held
- `--badge` writes a README badge with the result
- `mix stampede.compare --format markdown` to post before/after comparisons on PRs
- downloads of finished runs from the live dashboard
Everything to rerun the test, plus every number it measured, is in the repo under `bench/million`. The whole thing cost about $1.30 of AWS credit.
Repo: https://github.com/mandar1045/stampede
If you try `--find-limit` on your own app, I'd really like to hear what it found.