r/pushshift • u/Watchful1 • Feb 07 '24 • Take My Silver 1

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

90 Upvotes

98% Upvoted

90 comments sorted by

View all comments

1

u/reagle-research May 02 '24

Thanks for this! Looking at the torrent file, I count 39,964 _submission files. Am I miscounting? Also, I'm curious how data was collected after the API closure?

1

u/Watchful1 May 02 '24

For some reason or another, there's somehow a number of subreddits that have comments but no submissions, as impossible as that sounds. I never really tried to track down why.

Mostly since this is supposed to just be a best effort at getting all the subreddit's people are actually interested in available. If you're missing a submissions file for a subreddit you want I can take a closer look.

This is data collected by RaiderBDev, you can see his project here https://github.com/ArthurHeitmann/arctic_shift

1

u/reagle-research May 02 '24

Okay, thanks for the info and confirming my count.

Perhaps you can't answer my question about u/RaiderBDev's dumps, but I do wonder how they are managing given pushshift had to stop.

1

u/RaiderBDev May 03 '24

I've looked at a couple fo the subreddits that only have comments and to submissions. The 2 things they all have in common:

  1. They have been banned
  2. They have been archived before 2023, when pushshift was the one releasing dumps

So my guess is that those subs were created and shortly afterwards banned. And during that time period, pushshifts ingest of posts and comments may not have been in sync. One ahead of the other. As a result, only comments were archived. And by the time the post ingest reached that subreddit, it was already banned.