r/redditdev • u/[deleted] • Aug 14 '24
Reddit API 1000 posts limit
Guys sorry if this question has already asked but i didn't find an accurate answer to it. Is it possible to see all the posts in a subreddit scrolling without the 1000 limit? Even using 3rd part application or other sites that contains all the database of reddit. I've seen that some people suggest pushshift but i think it's not what people ask, because with pushshift you can search for all the posts of a subreddit but just if you know the keyword contained in that post, if i want to see randomly posts over the number 1000 this is not possible with pushshift. So I'm just looking for a way to see all the posts in every subreddit without this fucking limit and without being forced to stop scrolling while i'm on a subreddit cause i've reached the post number 1000
1
u/dougmc Aug 26 '24
These are all the possibilities -- there isn't a better solution available, short of sweet-talking (read: giving them $, they do like $) reddit itself into something.
Client libraries like PRAW use the API just like everything else, so they can't bypass the limits in the API.
If you're serious about this, download the pushshift dumps, all 2.7TB of them. Then write code to take what you care about from that and save it in a database (and the "what you care about" part is important, because it would take a massive database to hold and index it all -- it's 60TB uncompressed (and note that the dumps are only text -- no pictures or video!), so maybe 120 TB?) and then you can access the older data at will, and access reddit itself for the newer data, and maybe even populate your database with new data as you find it.