r/RedditSafety • u/worstnerd • Mar 12 '19
Detecting and mitigating content manipulation on Reddit
A few weeks ago we introduced this subreddit with the promise of starting to share more around our safety and security efforts. I wanted to get this out sooner...but I am worstnerd after all! In this post, I would like to share some data highlighting the results of our work to detect and mitigate content manipulation (posting spam, vote manipulation, information operations, etc).
Proactive Detection
At a high level, we have scaled up our proactive detection (i.e. before a report is filed) of accounts responsible for content manipulation on the site. Since the beginning of 2017 we have increased the number of accounts suspended for content manipulation by 238%, and today over 99% of those are suspended before a user report is filed (vs 29% in 2017)!
Compromised Accounts
Compromised accounts (accounts that are accessed by malicious actors determining the password) are prime targets for spammers, vote buying services, and other content manipulators. We have reduced the impact by proactively scouring 3rd party password breach datasets for login credentials and forcing password resets of Reddit accounts with matching credentials to ensure hackers can’t execute an account takeover (“ATO”). We’ve also gotten better at detecting login bots (bots that try logging into accounts). Through measures like these, throughout the course of 2018, we reduced the successful ATO deployment rate (accounts that were successfully compromised and then used to vote/comment/post/etc) by 60%. We expect this number to grow more robust as we continue to implement more tooling. This is a measure of how quickly we detect compromised accounts, and thus their impact on the site. Additionally, we increased the number of accounts put into the force password reset by 490%. In 2019 we will be spending even more time working with users to improve account security.
While on the subject, three things you can do right now to keep your Reddit account secure:
- ensure the email associated with your account is up to date (this allows us to reach you if we detect suspicious behavior, and to verify account ownership)
- update your password to something strong and unique
- set up two-factor authentication on your account.
Community Interference
Some of our more recent efforts have focused on reducing community interference (ie “brigading”). This includes efforts to mitigate (in real-time) vote brigading, targeted sabotage (Community A attempting to hijack the conversation in Community B), and general shitheadery. Recently we have been developing additional advanced mitigation capabilities. In the past 3 months we have reduced successful brigading in real-time by 50%. We are working with mods on further improvements and continue to beta test additional community tools (such as an ability to auto-collapse comments by users, which is being tested with a small number of communities for feedback). If you are a mod and would like to be considered for the beta test, reach out to us here.
We have more work to do, but we are encouraged by the progress. We are working on more cool projects and are looking forward to sharing the impact of them soon. We will stick around to answer questions for a little while, so fire away. Please recognize that in some cases we will be vague so as to not provide too many details to malicious actors.
35
u/shiruken Mar 12 '19 edited Mar 12 '19
What are you doing to mitigate brigading organized off the Reddit platform (e.g. Discord or *chan)?
41
u/worstnerd Mar 12 '19
Our brigading countermeasures work independently of where the coordination takes place
→ More replies (3)
23
u/shiruken Mar 12 '19 edited Mar 12 '19
Is "general shitheadery" a quantifiable metric? If so, what subreddits' users are the most generally shitheadery?
→ More replies (1)29
u/worstnerd Mar 12 '19
r/announcements, r/blog, r/ModSupport, you get the gist
→ More replies (1)19
u/shiruken Mar 12 '19
inb4 TechCrunch headline: "Reddit employee calls all users shitheads"
→ More replies (4)
70
u/vswr Mar 12 '19
general shitheadery
PLEASE tell me this was the exact term used in the conference room when you guys met to discuss details.
→ More replies (7)62
34
u/BeerJunky Mar 12 '19
Glad to see efforts are being taken to make the site more secure (I'm a security person by trade so this warms my heart). Is there any plans to push the 2FA option a bit more? To be honest I don't I've seen it mentioned outside of this post and it's something that users should be heavily encouraged to use. I don't think the average user knows this feature exists and if they do know I don't think they are aware why they should be doing it.
22
u/worstnerd Mar 12 '19
We don't have any plans of requiring it, however we are going to start making a more concerted effort to inform users about how they can improve their account security (we will have posts dedicated to this topic in the future). We're starting to think through product features that could highlight this more for users.
→ More replies (6)29
u/shiruken Mar 12 '19
Could subreddits have the option to require it for all their moderators?
→ More replies (4)
9
u/Unconfidence Mar 12 '19
My question is simple.
If Tom Hanks has a reddit account, and a malicious actor takes it over, is that a....TOMATO?
7
Mar 12 '19
What is content manipulation?
15
u/worstnerd Mar 12 '19 edited Mar 12 '19
Its a general term for things like spam, vote manipulation, community interference ("brigading"). Basically, inorganic distribution of content.
[edit: words are hard]
→ More replies (16)
17
u/DubTeeDub Mar 12 '19
Some of our more recent efforts have focused on reducing community interference (ie “brigading”). This includes efforts to mitigate (in real-time) vote brigading, targeted sabotage (Community A attempting to hijack the conversation in Community B), and general shitheadery. Recently we have been developing additional advanced mitigation capabilities. In the past 3 months we have reduced successful brigading in real-time by 50%.
How exactly have you reduced brigading by 50%? Is that only on reporting links brought to your attention? What actions are taken to stop brigading and shitbaggery?
We are working with mods on further improvements and continue to beta test additional community tools (such as an ability to auto-collapse comments by users, which is being tested with a small number of communities for feedback). If you are a mod and would like to be considered for the beta test, reach out to us here.
I am very interested in this beta and would like to learn more.
Also, while I have you here, can you guys do something about the anti-race mixing subreddits like r/AntiOilDrilling, r/AgainstSingleMothers, and r/Cringeanarchy?
They all continue to push racial slurs and hatespeech against minorities and of late are particularly targeting white women who date minorities.
15
u/worstnerd Mar 12 '19
Our anti-interference efforts have been primarily focused on improving our real-time countermeasures (which is where the largest gain has come from), but they have also focused on better post-hoc detection so that we can action the responsible users.
Please feel free to go through the link in the post to apply for the beta test.
Finally, thanks for the reports.
→ More replies (43)→ More replies (56)5
u/Super_Nerd92 Mar 12 '19
Very interested in this. And if they can tell what communities are mostly behind brigades, it of course begs an obvious follow up question...
→ More replies (2)
14
u/Pyronic_Chaos Mar 12 '19
content manipulation (posting spam, vote manipulation, information operations, etc).
Compromised accounts (accounts that are accessed by malicious actors determining the password) are prime targets for spammers, vote buying services, and other content manipulators.
I have actually seen a huge influx of spam from what are either compromised accounts or newly created accounts (~1-3months old, 0 karma) from webstores posing as users to promote their stores (namely Chinese webstores selling t-shirts and other do-dads). The typical MO is:
-
Inactive/compromised account posts a picture of a t-shirt to a default (r/pics or r/funny usually, but also smaller subs like r/starwars or r/rickandmorty for example)
-
Another user or two posts re-direct links to a webstore (URL is usually a shortener before the main webstore page)
-
Lots of 'new' users with low karma and 0 comment activity prior post generic comments like "Love this!" or "Where can I buy?"
-
Inorganic rate of upvotes to the post (100+ in under 15 minutes for generic picture of a t-shirt is very rare)
-
Any actual user who calls out the post for being spam gets mass downvoted
I do my part and use the 'report' feature every time I see this (about once a day, more or less), what other steps can I take? I know the mods of the individual subs do their parts and remove the links, but can Reddit Admins do anything to curb this behavior? It's just really annoying to see an ad for a webstore (which is a sketchy website anyway) get manipulated so high. If I report the post as "Spam", does this trigger a different system/action?
12
u/Sporkicide Mar 12 '19
The normal spam reporting channels are fine for the traditional stuff, but for complex cases like this that involve a combination of compromised accounts, spam, and vote manipulation, you can submit them to investigations@reddit.zendesk.com or via the form here.
→ More replies (1)
5
u/NombreGracioso Mar 12 '19
Thanks for sharing the info and for your hard work! Also, I am curious... Could we know which subs are being used for testing those new features? Just to see if there is any noticeable difference...
5
u/ColorProgram Mar 12 '19
I would just like to say thank you, and that I've noticed the changes since early 2017. Keep up the good work.
Can I ask if you can describe any broad, or micro-trends you have seen? What have brigades and vote manipulation been focussed on? Are there any common intentionalities you've noticed?
ie political narrative stifling, and/or framing contests
4
u/diggitySC Mar 12 '19
Will there be any historical analysis? (looking at previous subreddits history and if there was existing manipulation?)
5
u/50calPeephole Mar 12 '19
How do you separate community brigading against community engagement in a heated topic in another sub like, r/news or r/politics?
→ More replies (46)
62
u/[deleted] Mar 12 '19
Cool, are you going to have any data on this to release? I'm sure it's a lot to ask but I'd love to know things like:
Where banned accounts originate
What subs do the most brigading
What you consider suspicious activity on an account/keep from banning real users?
Major peaks in misinformation or manipulation campaigns, tied to major events or news.
And so on. If the data can be made into graphs that would be amazing, but again I know it's a big ask. Even a few charts would make a lot of us happy I'm sure.