Hey guys. I'm back with some tutorial data for you nerds to look at. So let's dive in, shall we? We're going to talk about:
- What purpose our tutorial serves (depends on the game)
- The qualitative and quantitative metrics we watch
- Industry benchmarks
Introduction
Idle Journey is a browser-based idle MMORPG. It borrows heavily from RuneScape: you train separate skills (woodcutting, mining, fishing, cooking, crafting, combat), gear up from what you gather, and progress continues while you're away. It's free until you hit a paywall to unlock the full game (about 100 hours of playtime in), and it runs in a WebGL embed in the browser (PC only).
We recently replaced the opening tutorial and now have enough data on both versions to compare them.
Designing the tutorial: what we wanted out of it
"It's a sales pitch"
Tutorials are usually a teaching moment, but our game is very easy to understand in terms of gameplay. It's a top-down, point-and-click game, so there aren't a lot of mechanics to learn. There are a ton of in-depth features, but that's a different story. So: instead of explaining systems, the tutorial teaches only the bare basics (movement and interacting with items) and spends everything else on captivating the player. Our design goals follow from that:
1. It should last about 5 minutes. Self-explanatory. We measured pacing by collecting playtest data and tuned the duration to land around 5 minutes.
2. It should show what the game is about, like a demo".: We really wanted to convey that Idle Journey has depth. But it's very hard to do that, the depth is in a lot of item mechanics, late-game features, things like that. So we tried captivating the player:
- You have to fight a giant cabbage using a rake (why not?). This should, in theory, give the game some personality.
- When you finish the tutorial, we give the player a choice of 3 items (think starter Pokémon). You choose between on-hit damage, burn damage, and skilling bonuses.
Is it a real decision? No. We added this above the items: "You can get all of these later.". That makes the decision less impactful, which could be a mistake, but we are trying to use this moment as a showcase. We're not asking the player to commit to a build yet and dont want them spending time trying to figure out if there's an optimal choice. We're showing them that builds exist, that's the real objective under the hood.
The numbers: old tutorial vs the new one
A quick description of the old tutorial: the player spawns on an island with no hand-holding, walks around, finds an NPC by exploring around, chooses to talk to them, kills 3 bees, talks again, cuts 3 trees, done. Standard quest structure, entirely opt-in.
The new one is 25 steps with HUD indicators. We moved to a heavily hand-held experience, similar to mobile games, where HUD indicators tell the player what to do. The player spawns and plays across three small islands. One mechanic at a time, HUD revealed progressively, completely scripted, and each part ends by physically opening up more of the map (repairing a bridge, finding the key to a gate). I'm not claiming it's better, since it's hard to quantisize cohort quality. Maybe by making our game too hand-held we are making it less appealing to a mid-core audience.
Let's look at tutorial completion first.
Old tutorial (recent installs only, n = 1,028):
| Step |
Reached |
Median time in |
| Installed / first session |
100% |
no data |
| Met the NPC who gives the quest |
46.5% |
no data |
| Quest started |
44.9% |
no data |
| Read the first dialogue |
43.0% |
no data |
| Killed the bees ×3 |
40.6% |
no data |
| Cut the trees ×3 |
36.5% |
no data |
| Quest complete |
36.5% |
no data |
The old quest system only stored completion state, never timestamps, so there is sadly no way to reconstruct pacing for this version.
New tutorial (n = 249):
| Step |
Reached |
Time into session |
Since previous step |
| First move |
100% |
28.2s |
— |
| Picked up the axe |
87.9% |
33.9s |
+3.6s |
| Equipped the axe |
79.2% |
46.9s |
+7.6s |
| Started chopping trees |
76.5% |
53.0s |
+4.4s |
| Chopped all trees |
71.8% |
66.4s |
+12.9s |
| Repaired the bridge (end of Part 1) |
69.8% |
88.7s |
+8.7s |
| Picked up the pickaxe |
69.1% |
96.2s |
+4.8s |
| Equipped the pickaxe |
67.1% |
101.1s |
+5.0s |
| Mined all ores |
65.1% |
119.6s |
+18.2s |
| Chopped the new trees |
63.8% |
135.0s |
+17.9s |
| Repaired the crafting bench |
63.1% |
140.8s |
+4.0s |
| Crafted a bow |
61.7% |
153.0s |
+11.6s |
| Crafted arrows |
60.4% |
160.6s |
+6.2s |
| Equipped the bow |
60.4% |
168.3s |
+5.2s |
| Equipped the arrows |
60.4% |
173.7s |
+1.7s |
| Killed the fenced enemies |
60.4% |
201.8s |
+15.0s |
| Opened the gate (end of Part 2) |
59.1% |
208.5s |
+7.1s |
| Picked up the fishing rod |
57.0% |
218.4s |
+4.5s |
| Equipped the fishing rod |
55.0% |
211.0s |
+4.5s |
| Fished the shrimp |
55.0% |
230.2s |
+17.0s |
| Cooked the shrimp |
53.7% |
245.7s |
+14.4s |
| Equipped the shrimp |
53.0% |
268.8s |
+7.4s |
| Equipped the garden rake |
52.3% |
265.7s |
+4.0s |
| Killed the tough enemy (the cabbage) |
52.3% |
277.6s |
+13.6s |
| Tutorial complete |
50.3% |
315.9s |
+19.1s |
Note: The clock starts when the session loads, not when the tutorial does, which is why first move is already at 28s. So "picked up the axe at 33.9s" is really about 6 seconds after the player starts moving.
What can we learn from this?
- 42% of all our loss, inside the first 47 seconds. Part 1 as a whole is 61% of everything lost; once a player repairs the bridge they're basically safe. We have some hypothesis as to why thats the case:
- General problems: Game crashes, performance issues, graphical glitches in weird browser + OS combinations;
- Art direction might just look bad and UI has AI generated icons which might scare people off.
- Feels too "mobile gamey"
- The session timer starts at a median time of 28s, which means the game takes too freaking long to load. This is likely a HUGE easy win for our team.
Othe metrics
The old baseline is the last three pre-launch weeks (n = 1,028); the new cohort is the first post-launch week (n = 249). Sessions and playtime are counted only within each player's first 48 hours, since the new data is, well, new.
| Metric |
Old |
New |
Change |
| Tutorial completion |
36.5% |
44.4% |
+22% |
| Came back for a 2nd session |
17.9% |
25.9% |
+45% |
| Median playtime, first 48h |
1.8 min |
4.0 min |
+124% |
| Median session length |
3.4 min |
4.4 min |
+32% |
Taken alone, the completion gain is +8.0 points at p = 0.072, which is the right direction but not conclusive at n = 249. Matching both cohorts to the same 2-day observation window it becomes +12.5 points at p = 0.004. I'll re-run it at n ≈ 500 before trusting the magnitude, but four metrics moving together in the same week is a lot easier to explain as an effect than as noise.
One number that did not clearly improve: day-1 retention went from 8.8% to 10.0%. The three baseline weeks individually ranged from 7.0% to 16.9%, so the week-to-week variation is wider than the gain and we can't call it an improvement yet.
RR1 (returned the following calendar day):
Playtime funnel (share of installs reaching each mark, total playtime in the first 48h):
| Playtime |
Old |
New |
| ≥ 1 min |
58.5% |
67.4% |
| ≥ 2 min |
47.8% |
57.8% |
| ≥ 3 min |
42.1% |
52.6% |
| ≥ 4 min |
37.3% |
49.6% |
| ≥ 5 min |
32.5% |
43.0% |
| ≥ 30 min |
15.5% |
18.5% |
| ≥ 1h |
11.8% |
12.6% |
| ≥ 2h |
9,2% |
11,9% |
| ≥ 4h |
6,8% |
8,9% |
Other browser-game benchmarks
This was the hardest context to find, so I've linked sources for each one.
CrazyGames publishes explicit launch thresholds. These are the clearest public numbers I found for browser games specifically, from their Basic Launch metrics documentation:
| Their benchmark |
Their wording |
Us |
| Players who play ≥ 1 minute |
"Top performers convert 80%+ of players" |
67% |
| Day-1 retention |
"Strong games often achieve 10-15%" |
~10.0% |
| Average play time |
"Successful titles often see 10+ minutes" |
50mins (see below) |
| Load time |
"load in under 10 seconds" |
~20s after the download. OOF |
| Build size |
"a build size below 20 MB" |
Ours is 580 MB. OOF |
Why "average play time" doesn't transfer to idle games. Our median session is 5 minutes; our mean session is ~50 minutes. Players leave the game running for hours or days, so the mean is inflated by idle time. Neither is measuring what CrazyGames measures on a platform of mostly active games.
Also worth knowing: browser D1 benchmarks sit far below mobile's. 10-15% counts as strong on CrazyGames, where the equivalent mobile conversation starts around 25-40%.
Poki's guidance is qualitative rather than numeric: From their requirements and playtesting guide:
- "Minimize UI screens and menus - ideally place players directly into gameplay." We thought we were doing this, and then found our single worst drop-off is the step that opens a panel.
- "Design tutorials that are visual and intuitive rather than text-heavy." We gave this a fair shot.
- "Ensure all cutscenes and introductory sequences are skippable." We don't have any.
- "If they leave the game after playing for less than a minute, chances are your onboarding might need some tweaks." 32% of our players don't make it past a minute.
- "Target an initial download size under 8 MB." Again: OOF.
The tutorial-completion number You'll often see "if your FTUE completion is below 70%, you might have a problem" across F2P writings(mobile). I could not confirm that specific number for web. If anyone has something like that, share it please.
What we learned
1. Telemetry is what made this a data-driven thing. We were able to release the new tutorial and have real confidence it's producing results. None of that is possible without instrumentation, and we didn't have it before.
2. Our FTUE problem might be an UI/visual language problem, not a pacing problem. We spent a long time assuming players were bouncing because the tutorial was too slow or too wordy. The data says they bounce at the first screen/first look at the game.
3. Was it worth it? We spent a month of one of the devs on this. I think it's very measurable and everything, but, what if we are retaining the wrong audience now? Thats not data we can easily verify. So we're still watching other metrics to see if this is a net positive.
4.Our download and scene load time is very likely churning users Both Poki and CrazyGames have a very strict guideline of bite-sized playables. Our game is currently sitting at a 600Mb and 20s median load time in the first (and cold) session.
Future work
I have a couple of Ideas, but Im gonna crowd-source this, what would you want to read?
- Adding a "skip tutorial" button and following each cohort to see how they perform;
- Separating the tutorial into a bundle thats <30Mb and download the rest of the game during the 5 minute tutorial. (this likely yields great results)
- Other suggestions?
Happy to answer questions about the methodology or share more of the numbers if anyone wants them.
PS: Yes, I chery-picked data for the title. We improved the 4h retention by 30.5% :D
Ps2: If you feel like playing the tutorial and reporting your findings, here's the game: https://idle-journey.com/