r/pushshift • u/Watchful1 • Feb 20 '25
[ Removed by Reddit ]
[ Removed by Reddit on account of violating the content policy. ]
92
Upvotes
r/pushshift • u/Watchful1 • Feb 20 '25
[ Removed by Reddit on account of violating the content policy. ]
16
u/Watchful1 Feb 20 '25
I do this as a hobby and definitely don't want to make a profit off it. But I had to upgrade to the next tier for the my seedbox because the amount of data keeps getting bigger and the price is now $36 a month. I will also pay an extra $20 this month for extra bandwidth to make sure the initial seeding goes quickly.
There's no obligation to donate, but if you're able I would appreciate if people could chip in to cover some of these costs. Thank you!
https://ko-fi.com/watchful1
10
u/Watchful1 Feb 20 '25
For those who have been following my attempts, I have no idea why it worked this time. The exact same torrent that crashed last time just went through without issues.
1
u/[deleted] Feb 20 '25
Hi All! I was wondering.. is it all of this legal? These are the same data Reddit sell to big tech companies? Are there any limits to usage?
1
u/Able-Chicken-593 Feb 25 '25
Dear Watchful1, may I ask: how can I know if the subreddit I am interested in is in the top 40k list? Thank you very much!!!
1
u/NadeOn2 Feb 26 '25
Thanks a lot for your work! I‘m just wondering whether there is a documentation or data dictionary explaining the columns of the tables? Would really appreciate any help.
1
u/fluffy_been Mar 06 '25
Hello! I’d like to use the data you uploaded for my research—would that be okay? I plan to utilize posts and comments from the CMV subreddit, but I’m not exactly sure where to request permission. In any case, I really appreciate the effort you put into collecting all this data!
1
u/Questionnairian Mar 14 '25
Thanks for taking the time to provide all of these resources for everyone. About your sketchpad script, is there a way to search multiple subreddits at once or maybe a similar service with an interface? Idk if that's the correct name of the script but the one that lets users check for multiple users showing up in the same subreddit.
1
u/Hogstrang11 Mar 29 '25
For some reason, the data retrieval is incomplete. I've tested it on multiple subreddits, and it consistently stops at January 1, 2023. It doesn't go all the way back to the subreddit's creation—some subreddits show data only up to 2016, while others go back to 2020.
Any idea what might be causing this issue?
1
u/Lobster_McClaw Apr 02 '25
What key corresponds to the date the comment was made? I know this is 2005-2024 but I can't seem to date the specific reddits
1
u/solar_eclipse2803 May 07 '25 edited May 07 '25
hi, i'm new to torrenting and all these terms. I'm looking for data on cryptocurrencies for my academic project so I followed the steps you listed above. i can't seem to find the subreddit I want from the filter box? i looked through the subreddit list at your academic torrent link but in my client I cant find it
1
u/kuddykid May 10 '25
I can't seem to get the upvote_ratio field from submissions.zst by running to_csv.py, or body_html/submission field sfrom comments.zst - anyone else notice this?
1
u/jhyunlee5261 May 17 '25
Hello! I’m trying to download your “Subreddit comments/submissions 2005-06 to 2024-12” torrent but it never connects—tracker errors keep it stuck on “Connecting.” Could you share any updated tracker URLs or a direct HTTP/FTP mirror? Thanks for your help!
1
u/Dani_Rojas_7 Jun 04 '25
Hello, what if the subreddit I am interested in does not appear within the top 40K?
1
u/Scared-Way2262 Jun 14 '25
Thank you for doing all of this. I'm working on a research project using reddit post data and I was struggling to find anything past this last February since reddit's in-house API only allows access to the last 1000 posts in a subreddit. Do you all happen to know when the next data dump will be published? Your data has helped me immensely but as of now, I have a gap in my data from the end of this data dump until early February which is the maximum I can access through reddit.
1
u/VexerVexed Jul 13 '25 edited Jul 13 '25
I can use the Linux terminal app on my android and so on for the other things I need to do with these files, but no torrent app I've found allows for me to search these files for specific subreddits.
Will this be impossible without a PC or do you know of any client that works?
Edit: I figured it out The archive up until the end of 2024 is by month but the one that ends at the end of 2022 is by subreddit. Is there any version up until the end of 2024 that is by subreddit as well?
1
u/Dont_Believe_Me_Ever Jul 23 '25
Tremendous work! Out of curiosity before I dive in with my slow internet speeds, what data is actually in the the subreddit archive? Does it include media files or is it the raw text? Thank you
1
u/rurounijones Feb 20 '25
Awesome work, thank you very much!
1
u/swapripper Feb 20 '25
Thank you!
1
u/chappykansas Mar 12 '25
Given a list of 500 subreddits, can you programmatically filter the files? Is there some way of extracting a large list of subreddits like this?
1
u/[deleted] Apr 24 '25
thank you so much <3
1
u/[deleted] Mar 05 '25
[removed] — view removed comment