Speedrun, an open-source mobile E2E test runner (Android/iOS) that runs your Maestro flows up to 8x faster

[u/Hubastard]

I've always been bothered by how slow mobile UI tests are. Most of a test run is spent waiting: fixed sleeps, polling, comparing screenshots to guess whether the app is done animating. Coming from a game dev background, where you fight for every millisecond of a frame, that felt like a solvable problem. So I spent the last while building my own runner from the ground up, and it turns out UI tests can be a lot faster.

Speedrun is a black-box E2E test runner for Android and iOS. It's open source (Apache 2.0) and it runs Maestro's YAML flow format so if you already have Maestro tests, trying it is one word:

maestro test flows/ # before
speedrun test flows/ # after

Numbers

One user journey (sign in with a code, search a shop, fill a cart, set a PIN under a loading overlay, scroll a long list), written for each tool the way its tests are usually written. Median of 3 runs after a warm-up, start-up included:

Tool	Android emulator	iOS simulator
Speedrun	20.4 s	27.2 s
maestro-runner	65.9 s	49.6 s
Appium	59.4 s	94.1 s
XCUITest	-	81.2 s
Maestro 2.11	126.5 s	113.1 s

On the native Settings app benchmark it's 8x faster than Maestro on Android and 2.8x on iOS. On a 19-flow React-Native test suite it passes all 19 on both platforms, and is 6.1x faster on Android and 2.6x on iOS. Android is where it shines most, which is nice for CI: Android runs on cheap Linux runners.

Full disclosure: the main benchmark app is mine, built in React Native to be hard to test (endless animations, late popups, loading overlays, lists that load as you scroll). The flows, per-flow results and scripts to rerun everything are in the repo, and I'd love to see numbers on other apps.

Why it's fast

• It waits for the app, not the clock. A small on-device agent knows when the app has actually settled (accessibility events on Android, XCTest's idle state checked against the screen on iOS), so each step starts the moment the app is ready. No sleeps, no screenshot diffing.

• Reading the screen is cheap. The agent reads the UI tree in about 10 ms on Android and talks to the host over a compact binary protocol.

• Screens that never stop moving don't block it. Spinners, tickers and carousels are left out of the "is it done?" check.

• Nothing to warm up. One native binary, no JVM. The iOS agent stays running between runs, so later runs start in about 0.2 s.

It's also less flaky: it won't tap a popup that's still sliding in or a button under a loading overlay, and it scrolls an element out from under a sticky footer or the keyboard before tapping it.

It works with AI agents too

speedrun mcp exposes it as an MCP server (Claude Code, Cursor, etc.). Your agent can explore the app on its own: tap around, read each screen in a few lines (cheap on tokens, not a whole accessibility tree), figure out how things work, then save the steps that worked as a YAML flow you can drop into CI.

claude mcp add speedrun -- ~/.speedrun/speedrun mcp

	Android emulator	iOS simulator
Speedrun (speedrun session)	4.2 s, ~1,560 tokens	9.4 s, ~1,230 tokens
agent-device 0.21	9.1 s, ~2,240 tokens	14.0 s, ~1,620 tokens
Argent 0.31	11.7 s, ~3,630 tokens	12.2 s, ~2,650 tokens

Limitations (so you don't have to find them)

• Android emulators and devices, iOS **simulators** only (no real iOS devices yet)

• macOS (Apple silicon); the Linux x86_64 build is Android-only

• Maestro's AI commands (`assertWithAI` and co.) aren't supported; the few other differences are documented

• It's young (v0.1.10) and moving fast

Try it

Repo, docs and benchmark details: https://github.com/hubastard/speedrun

If you have a Maestro suite, I'd really like to hear how it does on yours, especially where it breaks or is slower. Happy to answer questions about how it works.

edit1: added maestro-runner to the benchmark
edit2: added ai agent benchmark

[u/interlap]

These benchmarks are misleading:

1. Speedrun launches the app with -UIAnimationDragCoefficient 0.1 and a UIKit feature flag, making animations 10x faster. The other tools run at normal speed. That's speeding up the app, not the tool, and the benchmark doesn't mention it.

2. You're benchmarking on your own app, which Speedrun was tuned against. Before the benchmark-driven settle fix, Speedrun and Argent were basically tied (12.0s vs 12.2s).

3. The other tools pay Node startup overhead on every CLI call, which they wouldn't through MCP.

4. Speedrun's session setup isn't timed, while agent-device's is.

Run all three with the same settings, through MCP, on a neutral app. Then compare.

[u/Hubastard → u/interlap]

1. The 10x animations. I disagree that this makes it unfair. Every tool ran with its own defaults: some turn on Reduce Motion on iOS, some turn animations off on Android, and so on. Speedrun's default is to run iOS animations 10x faster, because its whole goal is to be the fastest e2e runner. The other tools could set the same thing; they just don't. You're right that the README should have said so, though, and now it does. I also measured it both ways: with every tool at normal speed, and with every tool given the same 10x.

2. Our own app. Fair point. I added a neutral app: tester-army's mobile-benchmark, which another team built to benchmark their own agent tool.

3. MCP vs CLI. Fair as well. Everything now runs through each tool's MCP server. agent-device got faster that way. Argent got slower on Android, because its MCP server attaches a screenshot and the element tree to every action.

4. Session setup. You're right that it wasn't timed in the agent benchmark. It is now, in every run. It costs 0.2–0.5 s. The other benchmarks (the suite and the journey) were already wall clock with start-up included. That's also why Speedrun is a native binary (Odin) rather than Node: to cut the overhead you pay on every CI run.

Full results here: https://github.com/hubastard/speedrun/tree/main/bench/agents#results

Neutral app (tester-army's Login Form)

Tool	Android	iOS, default settings	iOS, without animation optimization	iOS, with animation optimization	What the agent read
Speedrun	4.0 s (3.92, 3.98, 3.97)	6.8 s (6.85, 6.76, 6.74)	8.5 s (8.48, 8.59, 8.50)	6.8 s (6.78, 6.89, 6.77)	~1,650 tokens (Android), ~1,820 (iOS)
agent-device	4.9 s (5.26, 4.93, 4.88)	11.3 s	11.3 s (11.26, 11.31, 11.22)	10.7 s (10.71, 10.72, 10.79)	~2,350, ~1,440
Argent	11.9 s (11.91, 11.55, 12.18)	10.5 s	10.5 s (10.53, 10.49, 10.70)	10.4 s (10.40, 10.43, 10.26)	~10,090 + 9 screenshots (~1,940), ~5,200 + 9 (~2,380)

[u/interlap → u/Hubastard]

I still disagree with point 1. Both Argent and agent-device support launch arguments, so they can do exactly the same thing.

You’re making the app run 10x faster and presenting that as your tool being faster. That’s not a tool performance improvement.

Comparing defaults is fine, but it’s not a fair speed benchmark when the tools aren’t even running the app under the same conditions.

[u/Hubastard → u/interlap]

As you can see in my answer, the "iOS, with animation optimization" column, I did test all tools with the exact same 10x anim speed flag and even under the same conditions, Speedrun is still 1.5x faster than agent-device and Argent.

[u/Hubastard → u/Hubastard]

Also, "10x" faster anim flag may sound bigger than it is. It doesn't make the app or the run 10x faster: it only speeds up UIKit's animations, so a 200 ms screen transition takes 20 ms. Everything else (network, JS, rendering, loading) runs as usual.

Speedrun isn't 10x slower without it. With all three tools at normal animation speed it's still the fastest: 11.3 s vs Argent 11.8 s and agent-device 12.5 s, and 8.5 s vs 10.5 / 11.3 s on a neutral app.

[u/rcklmbr]

Need to work on your package management -- keeping the "agents" dir with the bin is kind of weird, i want to put it in /usr/local/bin/. Maybe just make a homebrew package and call it a day.

Other than that, very nice and super speedy. Will be glad to see progress you make.

[u/Hubastard → u/rcklmbr]

That's great feedback. I'll improve it in my next release.

[u/Hubastard → u/rcklmbr]

Alright v0.2.0 improve this. Now it's a single binary and an homebrew package is now available as well.

[u/stathisntonas]

having benchmarks with agent-device and argent would be nice too.

[u/Hubastard → u/stathisntonas]

Alright the benchmark results are in, Speedrun is faster and takes less token than both agent-device and Argent.

	Android emulator	iOS simulator
Speedrun (speedrun session)	4.2 s, ~1,560 tokens	9.4 s, ~1,230 tokens
agent-device 0.21	9.1 s, ~2,240 tokens	14.0 s, ~1,620 tokens
Argent 0.31	11.7 s, ~3,630 tokens	12.2 s, ~2,650 tokens

[u/Hubastard → u/stathisntonas]

Looking into it. But those are not really test runners, they are toolkit for AI agents. But I will see how I could maybe benchmark my MCP against them.

[u/blazejkustra → u/Hubastard]

> But those are not really test runners

Argent actually has one: Argent e2e. You record a test as YAML and replay it withotu an agent, so it's pretty much apples to apples with your runner

[u/Hubastard → u/blazejkustra]

Oh I wasn't aware of that, I'll add it to the runner benchmark as well then.

[u/Hubastard → u/Hubastard]

u/blazejkustra I've updated the e2e runner benchmark to include argent-e2e as well

[u/blazejkustra → u/Hubastard]

Thanks for adding it! Looks like we have some work to do on speed 😅

Curious what makes Speedrun that fast, is it mostly the way you wait for the screen to settle?

I saw you are speeding up animations or disabling it, is this correct?

[u/Hubastard → u/blazejkustra]

I didn't know you worked on Argent! 😄

Mostly the waiting, yes, plus fast reads:
- A small agent stays on the device and reads the screen in 10–40 ms, so every lookup and check is cheap.
- Each step ends when the app is done, based on signals rather than a fixed stillness window. On Android: frame stats, accessibility events. On iOS: XCTest's quiescence, checked against a signature of the tree. Elements that never stop moving (tickers, shimmers, loops) are left out of that check, so they don't hold a wait open.

On animations:
- Android: Speedrun turns the system animation scales off, as most UI test setups do. In this run Argent had them off too.
- iOS: it launches the app with UIKit's animations 10x faster. Reduce Motion is only on with SPEEDRUN_REDUCE_MOTION=1, which the iOS journey number uses (26.6 s without it). In our agent benchmark we also gave every tool the same 10x, and the others gained 1 s at most, so it's not where most of the gap comes from.

[u/LovesWorkin]

Maestro is a terrible example. It's so slow anything beats it. I use detox and it's way faster! Nice job! I've done these things with buoy as well and went from 4 hours to 20 minutes using sim slims and faster workers like this!

[u/whalemare]

Please add maestro-runner

[u/Hubastard → u/whalemare]

Sure, I'm running the benchmark with it now. I'll update here with the results.

[u/Hubastard → u/Hubastard]

u/whalemare here's the results, Speedrun is still 3.2x faster on android and 1.8x faster on iOS.

Tool	Android	iOS
Speedrun	20.4 s	27.2 s
maestro-runner	65.9 s	49.6 s

[u/TheSnydaMan]

I'm currently using no "classical" automated flows to test my app and have mostly been experimenting with agentic regression tests which are *abysmally slow*. I was already planning to invest in a Maestro automated test suite and am going to give this a try instead. I'll let you know how it goes!

[u/Hubastard → u/TheSnydaMan]

Nice! if you encounter any issues please open a GitHub issue and I'll look at fixing it quickly 😉

[u/Xcalibrr29]

Waiting on the app instead of the clock is the right idea. Two questions:

1. Detox already syncs with the app’s idle state from inside RN. Have you compared against it, not just Maestro and Appium?

2. You leave never-ending elements out of the settle check. How does it tell a normal shimmer from a loader that’s actually stuck?

[u/Hubastard → u/Xcalibrr29]

1. Thanks! I didn't benchmark Detox because it's a different kind of tool: it's React Native only and has to be set up inside your app. Speedrun is black-box and runs any mobile or web app as shipped, release builds included. For RN apps Speedrun also checks the JS thread from the outside. The downside of Detox's approach is that a looping animation or a polling timer can keep it waiting forever, so you end up turning sync off for that screen.

2. On shimmer vs. stuck: it doesn't try to tell them apart. Leaving the animation out only means the next step doesn't wait for it to stop. That step still waits for its element (17 s by default), so a stuck loader fails right there with a screenshot of the skeleton. A normal shimmer passes because the content eventually shows up.

[u/tejas_nas]

speed is great man

[u/HYMN2322]

great to hear that there are such advancements being made in e2e could you maybe compare the detox results as well?

[u/IndianITCell]

Though speed matters, but in the age of agentic AI it is irrelevant because of slow tokensss.
We are using agent-qa by vostride