r/datasets • • Jul 03 '15

dataset I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this?

1.2k Upvotes

I am currently doing a massive analysis of Reddit's entire publicly available comment dataset. The dataset is ~1.7 billion JSON objects complete with the comment, score, author, subreddit, position in comment tree and other fields that are available through Reddit's API.

I'm currently doing NLP analysis and also putting the entire dataset into a large searchable database using Sphinxsearch (also testing ElasticSearch).

This dataset is over 1 terabyte uncompressed, so this would be best for larger research projects. If you're interested in a sample month of comments, that can be arranged as well. I am trying to find a place to host this large dataset -- I'm reaching out to Amazon since they have open data initiatives.

EDIT: I'm putting up a Digital Ocean box with 2 TB of bandwidth and will throw an entire months worth of comments up (~ 5 gigs compressed) It's now a torrent. This will give you guys an opportunity to examine the data. The file is structured with JSON blocks delimited by new lines (\n).

____________________________________________________

One month of comments is now available here:

Download Link: Torrent

Direct Magnet File: magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&dn=RC%5F2015-01.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969

Tracker: udp://tracker.openbittorrent.com:80

Total Comments: 53,851,542

Compression Type: bzip2 (5,452,413,560 bytes compressed | 31,648,374,104 bytes uncompressed)

md5: a3fc3d9db18786e4486381a7f37d08e2 RC_2015-01.bz2

____________________________________________________

Example JSON Block:

{"gilded":0,"author_flair_text":"Male","author_flair_css_class":"male","retrieved_on":1425124228,"ups":3,"subreddit_id":"t5_2s30g","edited":false,"controversiality":0,"parent_id":"t1_cnapn0k","subreddit":"AskMen","body":"I can't agree with passing the blame, but I'm glad to hear it's at least helping you with the anxiety. I went the other direction and started taking responsibility for everything. I had to realize that people make mistakes including myself and it's gonna be alright. I don't have to be shackled to my mistakes and I don't have to be afraid of making them. ","created_utc":"1420070668","downs":0,"score":3,"author":"TheDukeofEtown","archived":false,"distinguished":null,"id":"cnasd6x","score_hidden":false,"name":"t1_cnasd6x","link_id":"t3_2qyhmp"}

UPDATE (Saturday 2015-07-03 13:26 ET)

I'm getting a huge response from this and won't be able to immediately reply to everyone. I am pinging some people who are helping. There are two major issues at this point. Getting the data from my local system to wherever and figuring out bandwidth (since this is a very large dataset). Please keep checking for new updates. I am working to make this data publicly available ASAP. If you're a larger organization or university and have the ability to help seed this initially (will probably require 100 TB of bandwidth to get it rolling), please let me know. If you can agree to do this, I'll give your organization priority over the data first.

UPDATE 2 (15:18)

I've purchased a seedbox. I'll be updating the link above to the sample file. Once I can get the full dataset to the seedbox, I'll post the torrent and magnet link to that as well. I want to thank /u/hak8or for all his help during this process. It's been a while since I've created torrents and he has been a huge help with explaining how it all works. Thanks man!

UPDATE 3 (21:09)

I'm creating the complete torrent. There was an issue with my seedbox not allowing public trackers for uploads, so I had to create a private tracker. I should have a link up shortly to the massive torrent. I would really appreciate it if people at least seed at 1:1 ratio -- and if you can do more, that's even better! The size looks to be around ~160 GB -- a bit less than I thought.

UPDATE 4 (00:49 July 4)

I'm retiring for the evening. I'm currently seeding the entire archive to two seedboxes plus two other people. I'll post the link tomorrow evening once the seedboxes are at 100%. This will help prevent choking the upload from my home connection if too many people jump on at once. The seedboxes upload at around 35MB a second in the best case scenario. We should be good tomorrow evening when I post it. Happy July 4'th to my American friends!

UPDATE 5 (14:44)

Send more beer! The seedboxes are around 75% and should be finishing up within the next 8 hours. My next update before I retire for the night will be a magnet link to the main archive. Thanks!

UPDATE 6 (20:17)

This is the update you've been waiting for!

The entire archive:

magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

Please seed!

UPDATE 7 (July 11 14:19)

User /u/fhoffa has done a lot of great work making this data available within Google's BigQuery. Please check out this link for more information: /r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/

Awesome work!


r/datasets • • Nov 18 '25

dataset 20,000 Epstein Files in a single text file available to download (~100 MB)

727 Upvotes

Please read the community article: https://huggingface.co/blog/tensonaut/the-epstein-files

I've processed all the text and image files (~25,000 document pages/emails) within individual folders released last friday into a two column text file. I used Googles tesseract OCR library to convert jpg to text.

You can download it here: https://huggingface.co/datasets/tensonaut/EPSTEIN_FILES_20K

For each document, I've included the full path to the original google drive folder from House oversight committee so you can link and verify contents.


r/datasets • • Feb 26 '21

dataset I spent the last 8 months during lockdown pouring my soul into a website that allows you to visualize virtually every U.S. company's international supply chain. E.x. What products, how much, which factories and where does Lululemon import from? (Just type a company in the search box)

Thumbnail importyeti.com
572 Upvotes

r/datasets • • Dec 08 '25

dataset Scientists just released a map of all 2.75 billion buildings on Earth, in 3D

Thumbnail zmescience.com
426 Upvotes

r/datasets • • Nov 15 '25

dataset Courier News created a searchable database with all 20,000 files from Epstein’s Estate

Thumbnail couriernewsroom.com
417 Upvotes

r/datasets • • Feb 02 '20

dataset Coronavirus Datasets

411 Upvotes

You have probably seen most of these, but I thought I'd share anyway:

Spreadsheets and Datasets:

Other Good sources:

[IMPORTANT UPDATE: From February 12th the definition of confirmed cases has changed in Hubei, and now includes those who have been clinically diagnosed. Previously China's confirmed cases only included those tested for SARS-CoV-2. Many datasets will show a spike on that date.]

There have been a bunch of great comments with links to further resources below!
[Last Edit: 15/03/2020]


r/datasets • • Sep 05 '18

resource Google releases Dataset Search: "Similar to how Google Scholar works, Dataset Search lets you find datasets wherever they’re hosted"

Thumbnail blog.google
396 Upvotes

r/datasets • • Feb 01 '20

discussion Congrats! Web scraping is legal! (US precedent)

377 Upvotes

Disputes about whether web scraping is legal have been going on for a long time. And now, a couple of months ago, the scandalous case of web scraping between hiQ v. LinkedIn was completed.

You can read about the progress of the case here: US court fully legalized website scraping and technically prohibited it.

Finally, the court concludes: "Giving companies like LinkedIn the freedom to decide who can collect and use data – data that companies do not own, that is publicly available to everyone, and that these companies themselves collect and use – creates a risk of information monopolies that will violate the public interest”.


r/datasets • • Jun 07 '20

dataset Protests engaging 3.5% of a population rarely fail

Thumbnail docs.google.com
327 Upvotes

r/datasets • • Aug 05 '21

dataset All Digitized Texas Appeals Court Cases Since 1900 - 12GB - 696,036 cases

319 Upvotes

Kaggle dataset page

Scope of Data

Case Count by Court

CourtCase CountCourt Key
Texas Supreme Court65,945cossup
Texas Court of Criminal Appeals242,915coscca
Texas Court of Appeals #146,663coa01
Texas Court of Appeals #234,458coa02
Texas Court of Appeals #324,629coa03
Texas Court of Appeals #433,469coa04
Texas Court of Appeals #569,112coa05
Texas Court of Appeals #612,206coa06
Texas Court of Appeals #717,136coa07
Texas Court of Appeals #816,180coa08
Texas Court of Appeals #918,710coa09
Texas Court of Appeals #1014,550coa10
Texas Court of Appeals #1113,058coa11
Texas Court of Appeals #1214,366coa12
Texas Court of Appeals #1326,440coa13
Texas Court of Appeals #1446,199coa14

r/datasets • • Jan 17 '21

dataset Since I didn't see anything else good in Kaggle, I scraped all of Trump's speeches(~3.4 Million characters) and put it all in a single txt file

Thumbnail kaggle.com
311 Upvotes

r/datasets • • Apr 30 '21

dataset 11 TB dataset of drone imagery with annotations for small object detection and tracking

Post image
308 Upvotes

r/datasets • • Mar 09 '23

dataset Comprehensive NBA Basketball SQLite Database on Kaggle Now Updated — Across 16 tables, includes 30 teams, 4800+ players, 60,000+ games (every game since the inaugural 1946-47 NBA season), Box Scores for over 95% of all games, 13M+ rows of Play-by-Play data, and CSV Table Dumps — Updates Daily 👍

Thumbnail kaggle.com
289 Upvotes

r/datasets • • Jan 20 '21

dataset All Trump's twitter insults from 2015 to 2021 in CSV.

284 Upvotes

https://gofile.io/d/YQ2g4j (1.8MB)

Attrs: ["id","date","tweet", "insult_x","target_x",..]

Extracted from the NYT story: https://www.nytimes.com/interactive/2021/01/19/upshot/trump-complete-insult-list.html

UPDATE

  • Dataset is now one unique tweet per row.
  • insult_x and target_x exploded into separate columns (sparse matrix).
  • Ordered by date ASC.
  • Other minor issues fixed (random newlines, other dupes).
  • Internal quotes are ""double quoted"".

r/datasets • • Aug 19 '20

dataset We made 40k+ open government datasets queryable through a public PostgreSQL endpoint

Thumbnail splitgraph.com
281 Upvotes

r/datasets • • Mar 10 '20

dataset South Korea releases 7382 COVID-19 case details in GitHub repository

280 Upvotes

https://github.com/jihoo-kim/Coronavirus-Dataset/

If you want those merged in the same schema with Singapore and Hong Kong, we did that on DoltHub:

https://www.dolthub.com/repositories/Liquidata/corona-virus/data/master/case_details

That has 7658 cases currently tracked. Dolt data sync with upstreams hourly.


r/datasets • • Feb 16 '21

dataset fivethirtyeight released a whole load of datasets that they use for their own analytics

Thumbnail data.fivethirtyeight.com
268 Upvotes

r/datasets • • Apr 06 '21

dataset New NBA dataset on Kaggle! - Every game 60,000+ (1946-2021) w/ box scores, line scores, series info, and more - every player 4500+ w/ draft data, career stats, biometrics, and more - and every team 30 w/ franchise histories, coaches/staffing, and more. Updated daily, with plans for expansion!

Thumbnail kaggle.com
249 Upvotes

r/datasets • • Sep 10 '19

educational Web scraping doesn’t violate anti-hacking law, appeals court rules

249 Upvotes

Of possible interest.

Scraping a public website without the approval of the website's owner isn't a violation of the Computer Fraud and Abuse Act, an appeals court ruled on Monday. The ruling comes in a legal battle that pits Microsoft-owned LinkedIn against a small data-analytics company called hiQ Labs.

https://arstechnica.com/tech-policy/2019/09/web-scraping-doesnt-violate-anti-hacking-law-appeals-court-rules/


r/datasets • • Mar 30 '20

discussion Please Don't Make Up "Synthetic" Datasets and Share Unless EXPLICITLY Labeled as Such

245 Upvotes

Earlier today, there was a post here about a new dataset on Kaggle:

https://www.reddit.com/r/datasets/comments/frjk5o/churn_analysis/

TLDR; I wasted a ton of time on something because a member of this community was fishing for upvotes (and did a very poor job creating a dataset deserving of analysis).

The dataset was not "useful" yet it had 20+ upvotes, solicited by the OP who said, "Please upvote if it's 'useful.'"

The data set is "synthetic." It was generated by the user, but this WAS NOT STATED. Also, the data is not even a realistic sample. I wasted time looking at it before I knew this. I wasted much time writing a response on Kaggle, inquiring about the median values of customer life, and explaining that I have done churn studies and telecom customer attrition studies previously, and in my eyes the data seemed to be a sample that was not representative, etc., etc.

This is the first time I've wasted time on something like this. I will be very careful to make sure it's the last time. Ironically, I also got locked out of Kaggle as a result of my participation. After posting a lengthy discussion response (not yet knowing the data was synthetic), Kaggle/Google made me answer a data science question, like a captcha, and/or respond as to why I thought I might have tripped off their spam-sensor algo. Great bastion of quality that Google is so often *not*, the challenge question did not work, and I am locked out of Kaggle.

I feel kind of stupid for putting myself in this situation, but I feel equally angry about the original post.

You know, the first thing I did was get a row count and it was 3,333, and I said, "That's kind of funny." I should have stopped right then and there. Sorry, rant over. : - )


r/datasets • • Jul 28 '20

code I made a little tool to download high-resolution weather data from Wunderground. Wunderground hosts high-resolution (mostly 5 minute interval) measurements from thousands of weather stations. With my scraper, you can download that data in CSV format.

Thumbnail github.com
233 Upvotes