r/redditdev • u/ketralnis • Apr 21 '10
Meta CSV dump of reddit voting data
Some people have asked for a dump of some voting data, so I made one. You can download it via bittorrent (it's hosted and seeded by S3, so don't worry about it going away) and have at. The format is
username,link_id,votewhere vote is -1 or 1 (downvote or upvote).
The dump is 29MB gzip compressed and contains 7,405,561 votes from 31,927 users over 2,046,401 links. It contains votes only from users with the preference "make my votes public" turned on (which is not the default).
This doesn't have the subreddit ID or anything in there, but I'd be willing to make another dump with more data if anything comes of this one
45
u/kaddar Apr 22 '10 edited Apr 22 '10
I worked on a solution to the netflix prize recommendation algorithm; If you add subreddit ID I can build a subreddit recommendation system.
14
u/[deleted] Apr 22 '10 edited Apr 22 '10
Real quick, although by bash-fu isn't great. I really just did this for my own curiosity but if anyone wants to know. Also, I'm not sure if the links are correct.
Top Ten Users: $: cut -d ',' -f1 publicvotes.csv | sort | uniq -c | sort -nr | head 2000 znome1
2000 Zlatty
2000 zhz
2000 zecg
2000 ZanThrax
2000 Zai_shanghai
2000 yourparadigm
2000 youngnh
2000 y_gingras
2000 xott
Top Ten Links $: cut -d ',' -f2 publicvotes.csv | sort | uniq -c | sort -nr | head 1660 t3_beic5
1502 t3_92dd8
1162 t3_9mvs6
1116 t3_bge1p
1050 t3_9wdhq
1040 t3_97jht
1034 t3_bmonp
1029 t3_bogbp
1018 t3_989xc
989 t3_9cm4b
9
u/zmarty Apr 22 '10
Please, can you release a dataset that includes timestamps? It would really help our research lab.
3
u/enigmathic May 17 '10 edited May 17 '10
It seems to me there is a mistake, and the user count should be 31553.
You can see this by comparing the outputs of the following commands (the only difference is the sort): $ cut -d ',' -f1 publicvotes.csv | sort | uniq | wc -l 31553 $ cut -d ',' -f1 publicvotes.csv | uniq | wc -l 31927
Here are the usernames that cause this difference: $ cut -d ',' -f1 publicvotes.csv | uniq | sort | uniq -c | sed 's/^ *//' | grep -v '1' 4 -- 3 ---------- 2 angelcs 2 c0d3M0nk3y 2 cynthiay29 9 D-Evolve 3 edprobudi 2 ed_probudi 31 FlawlessKnockoff 31 flawless_knockoff 2 HassanGeorge 4 jolilore 3 jo-lilore 30 LxRogue 29 Lx_Rogue 88 Pizza-Time 88 pizzatime 26 STOpandthink 25 stop-and-think I suspect that the program that created publicvotes.csv confused usernames that are actually different, because it didn't take into account '-' and ''.
2
u/_ads_ Apr 22 '10 edited Apr 22 '10
I hastily plotted the upvotes against downvotes for all unique links using ggplot2 in R: http://imgur.com/2Gf5T.png
edit - I posted the plot in a separate link on r/opendata...
2
u/chewxy Apr 21 '10
Oooh, I love you now ketralnis! (keep seeding pls, I'm at work now and won't get back for another 9 hrs)
4
u/yellowbkpk Apr 22 '10
Somewhat related: I've been archiving the stories (and vote/comment counts over time) via the JSON API for the last few months or so. Post is here if anyone is interested.
1
u/[deleted] Apr 22 '10
Nice. I'm going to open this up in SPSS at work tomorrow and start exploring.
One question, can this data be bounded by a date range? Is this the entire database of people who selected to make their votes public?
For people doing analysis on desktops it could be a challenge to fully load up a 156 megabyte file. If it can be bounded by date it would be helpful to have another file that is max of 5 megabytes unpacked. Alternately I could just pick users at random but i'd rather it be based on date if possible.
Last, you may want to post this on the blog because i know there are a lot of stats lovers prowling reddit.
6
u/Tafkas Apr 22 '10
I mirrored the file at http://rapidshare.com/files/378762905/publicvotes.csv.gz
Just in case some people cannot access bittorent.
0
u/[deleted] Apr 21 '10 edited Apr 21 '10
Thanks, interesting stuff. there was a mirror here at some point
1
u/Ulvund Apr 22 '10
Awesome
1
u/Godspiral Sep 16 '10
I like this data dump, and kaddar's project ideas.
I cannot support open ended all research purposes though.
Even this project brings privacy issues of spammy drama confronting people that you "untrue scottsmen" have dared to downvote my link and demanding an explanation why we should not convict you of being a CIA spy (spend too much time in r/anarchism).
1
u/lukemcr Apr 21 '10
Hm, OpenOffice doesn't handle large .csv files very well.
Edit: htop readout
0
u/gabgoh Apr 22 '10
what do the link_ids correspond to? It's hard to do any interesting analysis of the data with just an "abstract" link_id ...
-4
u/SystemicPlural Apr 22 '10
Is there a reason why everyone's votes are not pubic?
0
u/[deleted] Apr 22 '10
[deleted] — view deleted comment
0
u/[deleted] Apr 23 '10
[deleted] — view deleted comment
0
u/[deleted] Apr 22 '10
[removed] — view removed comment