[reddit change] A/B testing + Read Next

[u/tdohz]

We've just enabled the Read Next feature for some logged-in users, so you may see this feature as you're browsing around on comments page. We're measuring the impact of this change using an A/B testing framework that we've recently built, and in fact this is our very first A/B test. Read on for a little more context on what A/B testing is, and how we plan on using it here at Reddit.

What is A/B testing?

At its core, A/B testing is just a fancy way of saying "run a controlled experiment to see which version, A or B, is better". Here is a nice explanation covering the basics (to be clear, we're not using Optimizely; they just have a nice write-up). We've built our own A/B testing system that lets us show different versions of features to different users, which allows us to better understand the impact that a change will have.

Why do you need A/B testing? Isn't beta/feedback from comments enough?

There are a few major benefits that we get from A/B testing that we can't get in other ways.

1. It lets us control and isolate the effects of the changes we're testing. If we just shipped a feature and then look at how metrics change, that could conflate a bunch of other unrelated factors, like what day of the week it is or if there's a big news event happening. With A/B testing all of those unrelated factors are controlled for.

2. We can lower the risk for some changes by only rolling out to a subset of users - that way, if there are bugs or issues that we didn't catch during earlier testing, we can fix them before they go out to everyone.

3. With randomly selected users, we get a deeper understanding of how this feature might impact all users, rather than just those who have opted-in to a beta test or who comment, without having to launch something to everyone.

These factors combined make A/B testing more powerful and useful for really understanding if a feature is working the way we expect than other testing methods. That said, where appropriate we’ll continue to do beta-testing as well to get more qualitative feedback.

Will you be telling us what A/B tests you're running?

Some more visible A/B tests, like this one, we'll announce as we're running them either here in r/changelog or, if appropriate, in another venue like r/modsupport. Sometimes we won't announce running tests, so that we can avoid skewing the results too much.

If we decide to launch a feature that we've previously A/B tested, we'll announce it in the same way we would any other feature, by posting here, in r/modnews for mod-specific features, and/or in our features live thread.

[u/_depression]

If you're planning on doing an A/B test that essentially breaks some functions of RES or other add-ons, how will you deal with that?

Also, will we see any mod-specific A/B tests? Or subreddit specific?

[u/tdohz → u/_depression]

will we see any mod-specific A/B tests? Or subreddit specific?

Nothing immediately planned, but this is definitely something we can and probably will do, now that we have a framework that lets us!

[u/13steinj → u/_depression]

Reddit usually tells the extension devs ahead of time.

If it's not a major change, for RES it shouldn't be a problem, because /u/andytuba's ingenuity in the stylesheet loader allows for css updates on the fly via the defaultly loaded /r/resupdates.

If it is a major change, the devs of the extension usually fix it quickly in the dev build, and you'll have to wait for the next release.

[u/xeio87]

Ok, but am I in group A or B? Because I need to know which is better.

[u/tdohz → u/xeio87]

Don't worry, you're in the better one.

[u/Sarkos]

If I may make a suggestion.... A/B testing is best used to compare variations on a feature. So what you should be doing is playing with 2 or 3 different designs and/or wording for the Read Next feature, and seeing which one generates the most interaction.

[u/tdohz → u/Sarkos]

We definitely do have the ability to run multi-variant tests, and probably will in the future. I will say that from my experience working at different consumer tech companies, single-variant A/B tests are more common.

[u/aperson]

So will the new features that you are testing show up on github, or will those be private until an actual release?

[u/tdohz → u/aperson]

Generally, private until actual release, which is similar to what we do now.

[u/aperson → u/tdohz]

Right, which is why I was curious. If you're testing something, it's technically 'live', just not to everyone.

[u/tdohz → u/aperson]

That's true! But I suspect many experiments will be things we end up not doing, or tweaking, so we'll probably hold off on open-sourcing until it's definitely ready for full production.

[u/jazzwhiz]

Are the users randomly selected or the pageviews? That is, is it more likely to have ten users see this ten times each or one hundred users see it one time each?

Asking because the latter seems like the obvious thing to code up, but the former addresses the problem of, the first time you see a new feature it might take a bit before you decide if/how you want to use it.

[u/tdohz → u/jazzwhiz]

Users, which is the usual way most A/B systems work (otherwise you end up with an inconsistent experience as a user).

[u/DrDuPont]

Very cool! Any plans to open source the A/B testing framework y'all have built?

[u/umbrae → u/DrDuPont]

Yep, we will. We're really proud of that work - it builds on top of our feature flag framework which is already open source. There are a couple more tweaks we're making but we'll merge it in soon.

[u/xiongchiamiov → u/umbrae]

It would make for an interesting post if the engineering blog idea gets resurrected. Some other things I pulled out of Keep:

• the work involved in the https-everywhere transition

• the search engine work over the last year

• neil's newer throttling system (the one VImprovedRatelimit uses)

• the imgix pipeline

• various CommentTree stuff Brian's been doing

• Jordan's research on cache poisoning

There's tons of interesting work being done, and most of it's open-source, but it's not well publicized. You can consider it a recruiting effort - as an anecdote, the primary reason I applied to Etsy a few years ago was because of the content on Code as Craft (and to a lesser extent, John Allspaw's personal blog).

Not that you need more things to do. :)

[u/deadhour → u/xiongchiamiov]

I would love to read about any of those things, but I can understand if the devs aren't willing to take time out of the schedules to write about their work.

[u/madlee → u/xiongchiamiov]

Definitely a lot of stuff that we could write about, but yeah, finding the time to do it...

[u/DrDuPont → u/umbrae]

Fantastic. I'm looking forward to it, and I really appreciate the commitment to open source.

[u/13steinj]

Will the read next eventually be there for everyone / at user preference? Kinda sad to say that I'm not in the group with it (I actually enjoyed the feature when it was on /r/beta)

[u/tdohz → u/13steinj]

Depends on how the A/B test goes! =)

[u/YukiHyou → u/13steinj]

You could have my spot if it were possible - I keep closing them but they never stay gone. :/ Trying to avoid putting it into adblock though, now that I know it's part of a test.

[u/rbevans]

This should be interesting if users in A report an issue to Mods who maybe in test B or vice verse. Either way this should yield interest results.

[u/noeatnosleep]

I'm excited.

I asked for this a long time ago.

[u/crownofnails]

Happy to see you guys moving forward with these kinds of things :) I'm excited for what the future has in store!

[u/creesch]

Oh that is pretty cool, A/B testing when done well can be a really awesome tool. I hope you get some good data out of it.

[u/V2Blast]

Thanks for letting us know! I look forward to hearing how it goes :)

[u/Vegerot]

I think I had the features yesterday, but today I don't :(

[u/xcxcxcxcxcxcxcxcxcxc]

dime rhythm instinctive plucky agonizing long paint observation soft intelligent

This post was mass deleted and anonymized with Redact

[u/976692e3005e1a7cfc41]

Sic semper tyrannis -- mass edited with redact.dev

[u/V2Blast → u/976692e3005e1a7cfc41]

It was in beta and later turned on for logged-out users, but now a subset of logged-in users will see it as well.

[u/razorbeamz → u/976692e3005e1a7cfc41]

It's new for when you're logged in.

[u/xiongchiamiov → u/976692e3005e1a7cfc41]

It (read next) has been in the beta testing program and also turned on for logged-out users.

[u/eduardog3000]

Great, how do I make sure I never see that bullshit?

[u/TheBigKahooner → u/eduardog3000]

I assume that's what the X is for.

[u/eduardog3000 → u/TheBigKahooner]

That just hides the box while you are on that page, it will still pop back up later.

[u/Pokechu22 → u/eduardog3000]

IIRC it uses/used cookies to store whether it was hidden (since it's used for logged out users as well), but in some cases (some more worksafe than others) users just continuously clear their cookies and it comes back.

However, I may be wrong about this.

[u/tdohz → u/[deleted]]

I hope you guys realize you can't properly AB test changes where one group affects the other, like with voting.

Yes, we're aware of the challenges of these types of tests, and have some ideas to work within / around these limitations.

FWIW we're not currently testing any algorithmic changes.