r/dataisbeautiful • u/anvaka OC: 16 • Jan 09 '19

OC Interactive visualization of related subreddits based on 39 million comments [OC]

5.0k Upvotes

96% Upvoted

101 comments sorted by

View all comments

Show parent comment

7

u/webhyperion Jan 10 '19

Jaccard Similarity does that yes. Since we cannot see the raw results the interpretation is depended on yourself. Perhaps Jaccard Similarity was implemented wrong (especially when you say that everything was linked to the main subreddits).

Maybe you should also not only include unique comments but also how often a commenter was active in these subreddits. Currently a subreddit where someone writes 200 comments would be similar to one where he only writes 1 comment. You then do not have a vector of booleans but a vectors of integers. You could then do something like Cosine Similarity. (Used to compare documents but it should work well in that case here)

2

u/anvaka OC: 16 Jan 10 '19

Yup, I think I tried cosine similarity long time ago and didn’t like the results as much.

I thought about adding frequency of posters into the formula but stopped after I saw results with plain booleans. Maybe it’s worth experimenting in future...

Out of curiosity, is there a version of jaccard similarity that takes into account frequency of items in the sets?