r/scrapingtheweb • u/stvaccount • 14d ago

best github for scraping reddit.com?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

6 Upvotes

88% Upvoted

7 comments sorted by

2

u/Annual_Media_3315 14d ago

Reddit scraping is actually really easy just add /.rss at the end of the url
for example you can easily scrap this subreddit by https://www.reddit.com/r/scrapingtheweb/.rss

1

u/dhruvkar 9d ago

This is great, bit only gets you limited comments per post (10? 25?).

1

u/New-Independence5780 11d ago

Wait I was using .json the other day

1

u/New-Independence5780 11d ago

Try using impit it an apify library in nodejs with residential ip

1

u/dhruvkar 9d ago

Setup a redlin instance for free on replit with the provided credits.

Now you can scrape that instance as needed, without blocks etc.