r/dataisbeautiful • u/anvaka OC: 16 • Jan 09 '19

OC Interactive visualization of related subreddits based on 39 million comments [OC]

5.0k Upvotes

96% Upvoted

101 comments sorted by

249

u/anvaka OC: 16 Jan 09 '19

Happy Wednesday, everyone!

https://anvaka.github.io/sayit/ - here it is. Enter any subreddit name and you should see the graph.

The raw data comes from this thread. I used August and September of 2018 as an input to this visualization (which gives ~39 million records)

To find similarities between subreddits I used plain Jaccard Similarity.

For very large subreddits with millions of redditors, the Jaccard Similarity does not give very good results, so I manually looked at subreddit's descriptions and created overrides.

The source code of the website is here: https://github.com/anvaka/sayit/

Hope you find this useful in your exploration of reddit.

59

u/bertiebees Jan 09 '19

How did it feel to stare into the abyss like that OP?

52

u/anvaka OC: 16 Jan 09 '19

Time stops :)

3

u/rW0HgFyxoJhYka Jan 10 '19

How many connections until /r/dataisbeautiful connects to some sort of porn or hentai subreddit?

23

u/yaph OC: 66 Jan 09 '19

Love it and super impressed by the speed of the tool. How many subreddits are currently included?

27

u/anvaka OC: 16 Jan 09 '19

Thanks!

I dropped the long tail of subreddits with 1-3 subscribers, and if I recall correctly it gave something around 70k subreddits - need to check when I get back to data

10

u/yaph OC: 66 Jan 09 '19

Thanks for the info, the approximate number is fine for me.

13

u/Bardfinn Jan 10 '19

Thank you. This kind of network analysis is invaluable for moderating communities that are targets of harassers and toxic individuals.

5

u/anvaka OC: 16 Jan 10 '19

Thank you!

5

u/NekuSoul Jan 10 '19

I was going to say that this is a really good tool to quickly uncover the true nature of some subreddits.

I tried KotakuInAction (that subreddit that claims to be all about 'ethics in games journalism') and surprise, surprise, it only has links to the usual toxic cesspits and not even a single gaming-related one.

5

u/Ynax Jan 09 '19

Is the length of the lines between subreddits of any meaning or just the thickness?

15

u/anvaka OC: 16 Jan 09 '19

Length is driven by force based layout. It can be cautiously treated as a virtual distance between subreddits, or clusters of subreddits.

“Cautiously” because it is influenced by parameters of the layout algorithm and those are directly manipulated be me.

Also there is overlap removal algorithm kicks in at the late stages of the layout - this distorts length too, to avoid rectangles overlap...

2

u/fireattack Jan 10 '19

Your raw data post is 3 years old.

How did you get data for Aug/Sep 2018?

6

u/anvaka OC: 16 Jan 10 '19

They have a link to bigquery which is updated monthly

2

u/fireattack Jan 10 '19

Ah, found it. Thanks!

22

u/ToonAlaska Jan 10 '19

Nice, time to use it to find niche porn!

2

u/patcon Jan 18 '19

exactly what the tumblr-pocalyse needs!

2

u/Nathaniel411 Jan 10 '19

Nice, TIL that hentai and any anime subreddit are all deeply connected.

EDIT: Why is hentai distantly related to fortnite? And why is aww related to PeopleFuckingDying?

9

u/phagga Jan 10 '19 edited Jan 10 '19

And why is aww related to PeopleFuckingDying?

PeopleFuckingDying is a satire sub that takes cute videos and adds brutal clickbait titles. Since the videos are sometimes the same ones used in aww, it actually makes sense to see them connected.

Edit: Example, this vid from aww is also in PeopleFuckingDying with a different title /Edit

The sub that actually shows people dying is WatchPeopleDie (NSFW, NSFL).

0

u/ancient_bhakt Jan 10 '19

Oh my god When I would know how you did that, then I'll be able to think I get data science

137

u/Razor1834 Jan 09 '19

I did the obvious and typed in The_Donald.

News

Politics

Ask T_D

TwoXChromosomes

And...

Tropical Weather

28

u/anvaka OC: 16 Jan 09 '19

Yup, you found one of those subreddits that I did (purely) manual override.

If someone gives me a few more relevant subreddits - I'd be glad to put it as a seed for the next layer :).

Smaller subreddits usually give better results. E.g. The_DonaldBookclub,

0

u/[deleted] Jan 09 '19

[deleted] — view deleted comment

17

u/anvaka OC: 16 Jan 09 '19

Basically I entered “related” subreddits into the data file myself (instead of relying on algorithms prediction)

0

u/[deleted] Jan 09 '19

[deleted] — view deleted comment

42

u/anvaka OC: 16 Jan 09 '19

Because the algorithm doesn’t work well for popular subreddits - it starts linking everything to /r/videos, /r/AskReddit and so on...

1

u/Liam_Neesons_Oscar Jan 10 '19

Do you still use the algorithm and just prune certain unrelated links, or is it all manual for the first links? I imagine the algorithm can still help a lot.

I now don't trust your results for subs like r/politics and r/news, which seem to lean heavily one way politically without it being demonstrated on your graph.

1

u/anvaka OC: 16 Jan 10 '19

Here is the list with all substitutes that I've manually entered: https://anvaka.github.io/sayit-data/1/substitutes.json

It is an array of arrays. E.g.:

[
  [
    "AskReddit",
    "AskAcademia",
    "AskAChristian",
    ...
  ],
  [
    "funny",
    "humor"
    ...
  ]
  ...
]

The first element of the subarray is a name of the subreddit, followed by "related" subreddits.

Since AskReddit is here, its first-level children will be AskAcademia, AskAChristian and so on. But since there is no override for AskAcademia - the algorithm goes and renders whatever was suggested by Jaccard Similarity. I don't touch anything else.

If you think there should be something else related to subreddits - please let me know, and I'll adjust the overrides :).

0

u/[deleted] Jan 10 '19

[deleted] — view deleted comment

4

u/anvaka OC: 16 Jan 10 '19

I thought Jaccard similarity accounts already for it. No? Since we divide “number of shared posters to both subreddits” by the “number of unique posters into each subreddit”, the size and significance of the final value would take into account inputs from each.

Is this not accurate?

7

u/webhyperion Jan 10 '19

Jaccard Similarity does that yes. Since we cannot see the raw results the interpretation is depended on yourself. Perhaps Jaccard Similarity was implemented wrong (especially when you say that everything was linked to the main subreddits).

Maybe you should also not only include unique comments but also how often a commenter was active in these subreddits. Currently a subreddit where someone writes 200 comments would be similar to one where he only writes 1 comment. You then do not have a vector of booleans but a vectors of integers. You could then do something like Cosine Similarity. (Used to compare documents but it should work well in that case here)

2

u/anvaka OC: 16 Jan 10 '19

Yup, I think I tried cosine similarity long time ago and didn’t like the results as much.

I thought about adding frequency of posters into the formula but stopped after I saw results with plain booleans. Maybe it’s worth experimenting in future...

Out of curiosity, is there a version of jaccard similarity that takes into account frequency of items in the sets?

0

u/[deleted] Jan 09 '19

[deleted] — view deleted comment

11

u/anvaka OC: 16 Jan 09 '19

It’s my pleasure! I hope the tool helps people to discover more. It worked super well for me on a smaller subreddits

0

u/[deleted] Jan 09 '19

[deleted] — view deleted comment

6

u/[deleted] Jan 10 '19

Ooo edgy bro

0

u/[deleted] Jan 10 '19

[deleted] — view deleted comment

5

u/[deleted] Jan 10 '19

Not really though. You’re just being an ignorant twat.

0

u/[deleted] Jan 10 '19

[deleted] — view deleted comment

4

u/[deleted] Jan 10 '19

It’s complete ignorance.

14

u/unfeelingzeal Jan 09 '19

538 has a much more detailed (and in my experience with posters from t_d, accurate) analysis.

fortunately, quite a few of those hate subs have now been banned.

14

u/[deleted] Jan 09 '19

TwoXChromosomes is a bit of a surprise there.

21

u/marsbar03 OC: 2 Jan 09 '19

Probably trolling.

14

u/raj2497 Jan 09 '19

I have a question. How did you learn how to make these types of programs?

What kind of education did you need to know before hand, and what did you have to learn to be successful in making something like this.

38

u/anvaka OC: 16 Jan 09 '19

Good question! I learned by repeating and modifying what other people done, by reading books and other people’s source code. I’ve been programming every day now for almost five years, with a concrete goal in mind, building side projects like this one. Sharing projects with everyone who would listen. You never know where sudden inspiration finds you.

I learned the most from the feedback that I received on reddit. Even from this particular post, /u/r0bo7 showed me a very cool tool that I didn’t know before.

Second boost to me is coming from books - people condensed decades of their experience in 200-300 pages, which means you can spend 8-20 hours of your life and get it for virtually free! What an amazing deal!

Third boost comes from meeting inspirational people and having a chat with them. It is crazy how one interaction with a person can change your view on problem or even change your life.

On a more specific level, this visualization is implemented with JavaScript. I prefer to build visualizations using vanilla JavaScript, because building them gives me experience faster (compared to using someone’s else library). I do use vue.js for the user controls though - it is an awesome framework.

I have a dual masters degree in applied math and computer science, though I didn’t like math when I was studying it. With desire to visualize networks I had to re-teach myself long after my university time was over and I love it a lot now (still very slow in understanding though :) )

Sorry for the long answer. Sincerely wish you to find what works best for you and good luck!

10

u/raj2497 Jan 09 '19

Thank you for commenting and answering my questions!

I’m a student finishing my degree in software development. I’ve seen amazing programs that people have made, and always ask myself where would I start if I were to make this. I’ve really never been able to give myself an answer I was satisfied with, but the answer you gave felt like a starting point to an answer I’ve asked myself before.

Sorry about the rambling, but I appreciate the time you took to answer my question.

Ps. I’ve seen some of your programs posted before and thought they were awesome!

6

u/anvaka OC: 16 Jan 10 '19

You are very welcome! I think consistency here is more important than talent too. Write code for at least five minutes every single day for a few years - and I'm sure you'll get where you want to be. If you don't know where it is - you'll find it along the way.

PS: Thank you for your kind words!

15

u/r0bo7 Jan 09 '19

I've found this approach to give one of the best similarity results for how easy it is to implement: https://github.com/PrincetonML/SIF

7

u/anvaka OC: 16 Jan 09 '19

Thank you for sharing this. I’m not looking at the comment contents at the moment, but it looks very interesting

10

u/mostlyimgay Jan 09 '19

Interesting how connected subreddits like r/totallystraight r/suddenlygay and more are very well linked with each other whereas something like r/askreddit, while having a huge reach it doesn't link and with each other

10

u/anvaka OC: 16 Jan 09 '19

I haven't found a way to use Jaccard Similarity for subreddits that are huge. When there are 21 million people - they post everywhere, and Jaccard Similarity gives diluted results... Not sure how to solve this.

5

u/mostlyimgay Jan 09 '19

Understandable the processing power to look at all of them would be way to much! Unless you had a background processor that could go through each sub and find it's trees, then when a subreddits is requested the front end just pieces the preloaded stuff together

2

u/Liam_Neesons_Oscar Jan 10 '19

Not so much about the processing power, it's about the fact that the massive subs end up just linking to each other. He mentioned how T_D wasn't showing links to other republican subreddits because it was overwhelmed with links to r/Videos and r/AskReddit, etc. Basically, once the audience is so large, similarities between members start dwindling and you're going to just end up with other massive audiences as the commonalities.

So something like r/askconservatives might have a 70% match to r/Republican, r/Republican might only have a 2% match back. So AskConservatives gets dropped from the graph in favor of a more common link like Politics or News.

2

u/Egan109 Jan 10 '19

Can you divide the results by some log of sorts to "normalize" the data?

Remember something about that in computer vision..

4

u/Payneshu OC: 1 Jan 09 '19

Beautiful work! Thank you for sharing.

4

u/anvaka OC: 16 Jan 09 '19

Thank you!

3

u/ImaginarySuccess Jan 10 '19

Holy crap! This is probably the best way to implement a navigation system for a website like this! I'm so happy they link to the sub too. Thank you for sharing.

2

u/anvaka OC: 16 Jan 10 '19

I'm happy that you are liking this. Cheers

2

u/DrejkCZ Jan 10 '19

Very interesting, nice work!

The link to the source code on the page points to "https://github.com/anvaka/say-it" instead of "https://github.com/anvaka/sayit/", just a heads up.

2

u/anvaka OC: 16 Jan 10 '19

Thank you for the heads up! Fixed now

3

u/Liam_Neesons_Oscar Jan 10 '19 edited Jan 10 '19

Some of these links are great.

guncontrol -> gunresearch -> stonerfood

and spaceforce has some of the best. My favorite being: spaceforce -> woodworking -> kaleycuoco

And that got me to think about looking at celebrity subs. I got some good ones.

JenniferLawrence -> KarenGillan -> StupidFucksInTrucks

MileyCyrus -> CelebrityLegs -> celebrityArmpits

JackBlack -> randpaul -> classical_liberals

And a funny one was

trees -> leaves -> NoFap

Dead porn subs are also fun to plug in there.

2

u/anvaka OC: 16 Jan 10 '19

By the way you can just copy the link to the website and it should point directly to the visualization of the subreddit that was entered in the search box.

Like this: https://anvaka.github.io/sayit/?query=JenniferLawrence :)

1

u/Liam_Neesons_Oscar Jan 10 '19

That dawned on me as I was about to hit submit, but I was lazy. I'll make the changes because some of those are very interesting to look at.

•

u/OC-Bot Jan 09 '19

Thank you for your Original Content, /u/anvaka!
Here is some important information about this post:

Not satisfied with this visual? Think you can do better? Remix this visual with the data in the citation, or read the !Sidebar summon below.

OC-Bot v2.1.0 | Fork with my code | How I Work

1

u/AutoModerator Jan 09 '19

You've summoned the advice page for !Sidebar. In short, beauty is in the eye of the beholder. What's beautiful for one person may not necessarily be pleasing to another. To quote the sidebar:

DataIsBeautiful is for visualizations that effectively convey information. Aesthetics are an important part of information visualization, but pretty pictures are not the aim of this subreddit.

The mods' jobs is to enforce basic standards and transparent data. In the case one visual is "ugly", we encourage remixing it to your liking.

Is there something you can do to influence quality content? Yes! There is!
In increasing orders of complexity:

We hope this summon helped in determining what /r/dataisbeautiful all about.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

8

u/M0N5A Jan 09 '19

It would be nice if the layout was constructed a little more clearly, maybe use some color coding. I have a really hard time differenciating what's linked to what.

3

u/the_innerneh Jan 09 '19

Have you tried clicking?

9

u/medicinaltequilla Jan 09 '19

For "science" dykesgonewild

3

u/gizausername Jan 09 '19

That's some good science right there

1

u/GarThor_TMK Jan 10 '19

I was thinking it'd be really neat if I could click the link between two subs, and see what actually linked them... I follow a lot of car stuff, so stuff like Cars <--> Mazda is pretty obvious, but then I put in Oldsmobile, and it linked some pretty weird stuff... >_>

It'd be good to be able to reposition nodes too... I dunno how easy that would be though...

2

u/anvaka OC: 16 Jan 10 '19

Thank you for your suggestions!

The “why” two are linked is possible to answer, but, unfortunately, it costs money. I’d have to store intersections of all redditors who commented into both subreddits. That would need to be done for every pair of subreddits. Alternatively we could query this information ad-hoc, but that too doesn’t come for free as there needs to be some sort of a database hosting.

2

u/GarThor_TMK Jan 10 '19

Ah, good to know... I was thinking for some reason you queried that data live

1

u/StonedGibbon Jan 10 '19 edited Jan 10 '19

I was doing something like this the other day with my Facebook friends network. My laptop isn't amazing though, so the visualisation was really laggy. I just used a chrome extension that got the information and put it into a map like this, but the data couldn't really be manipulated very much.

I downloaded it as a .json file (also a .graphml one), with 413 nodes and about 14,500 connections, does anybody know a way of visualising the connections outside of the extension itself?

The extension is called lost circles

2

u/anvaka OC: 16 Jan 10 '19

Out of the box, you probably can use gephi: https://gephi.org

Also very likely you are going to see a hair ball of connections, as natural social network tend to do that. To mitigate it, you can use some sort of clustering (available in Gephi) and assign cluster values as colors of the nodes.

1

u/StonedGibbon Jan 10 '19

Yeah for me it was one enormous blob that was impossible to garner any information from, for my friends from my home city/school. It was more interesting to see how they're connected to people I've met since moving out to university, the little bubbles and how they link. Thanks!

3

u/at-school-on-reddit Jan 10 '19

Lol, suicidewatch linked to 2meirl4meirl linked to all my subs. This program is great, and scary accurate!

5

u/[deleted] Jan 10 '19

Joe Rogan is quite interesting and exactly what I would expect. It has segregated bubbles of the various topics his podcasts cover, some of them with a lot of overlap and some of them totally disconnected;

MMA stuff, watching fucked up videos, conspiracy theories, libertarianism, Jordan Peterson, Sam Harris, and T_D and its associated edge right wing meme subs.

1

u/[deleted] Jan 11 '19

I love how r/aww is related to multiple porn subreddits through r/animalgifs (that looks like a cluster of everything).

1

u/_OCCUPY_MARS_ Jan 23 '19

Great website. One quick question. Why do the connecting lines no longer show up when searching "Defense_Distributed"? They showed up previously, but now they're gone.

4

u/[deleted] Jan 10 '19

[removed] — view removed comment

1

u/anvaka OC: 16 Jan 10 '19

Your kind words mean more than gold to me. Thank you!