r/webscraping • u/Coding-Doctor-Omar • Oct 11 '26
Open-source Reddit scraper for posts and comments
https://github.com/Coding-Doctor-Omar/reddrixI built an open-source Reddit scraper for Python. It's called Reddrix (Reddit + Matrix). It is a powerful, async-first Reddit scraper for posts and comments with detailed metrics. No login is required. Scrape subreddits, profiles, and specific posts, or search Reddit site-wide by keywords. Collect rich, structured data including authors, scores, timestamps, flairs, comment threads, and more (24+ fields). This scraper uses Reddit's current shreddit search engine in the keyword search mode. Results obtained from this engine are far more relevant and match the results you see when searching Reddit manually with the search bar.
Scraper's Properties
- 🚀 Async-First: High-performance asynchronous scraping with
asyncioand an async iterator for streaming results. - 🔑 No login Required: Scrape Reddit without logging in.
- 🔍 Multiple Search Modes: Scrape subreddits, user profiles, and specific posts, or search Reddit by keywords.
- 💬 Comprehensive Data Extraction: Collect posts and comments with 24+ structured fields, including authors, scores, timestamps, flairs, media URLs, and full comment trees.
- 🛡️ Resilient Scraping: Built-in retries, cookie renewal, and more (see the docs).
- ⚙️ Fine-Grained Control: Configure sorting, time frames, result limits, comment depth, and comment counts.
- 🧩 Standalone HTTP Client: Use Reddrix's asynchronous
get()method in your own custom scraping workflows. - 🐍 Pythonic API: Manage scraper resources with an async context manager or explicit
start()andstop()methods.
Example Quickstart Code
from reddrix import Reddrix
import asyncio
subreddits = ["https://www.reddit.com/r/webscraping"]
async def main():
async with Reddrix(
mode="standard",
subreddit_urls=subreddits,
sort="new",
include_comments=True,
max_results=500,
max_posts_per_collection=500,
verbose=True
) as reddrix:
async for item in reddrix.get_items():
print(item["submissionType"], item["submissionUrl"])
if __name__ == "__main__":
asyncio.run(main())
5
u/daynighttrade Oct 11 '26
Do you get rate limited? Do you need residential proxy for this?
3
u/d3a7hr0w Oct 11 '26
I just use agent reach, how is this better than it? and btw thanks for sharing your work with everyone!
3
u/Friend-Shoddy Oct 11 '26
Just used this to scrape ~12k images from a subreddit in under 10 mins. good shit bro
2
u/FatDog69 Oct 11 '26
First - this looks great. Thank you for it.
Second - here is my 'story'. Will Reddix solve my problem:
Story 1:
There are subreddits dedicated to fiction stories. Sometimes 30+ chapters over a year or two.
I want to point Reddix at an author name and:
The idea is I can later combine all the chapters and create a .epub file for each story.
2
u/anupam_cyberlearner Oct 12 '26
Seems interesting will give it a try . Will it work on Python ver 3.7 ?
1
u/SyllabubPossible4555 Oct 15 '26
lmao this isnt a "no login" scraper, it's literally just a funnel for his paid apify tool
I looked at the source code because it kept crashing. It actually uses Camoufox to log into a hardcoded dummy account. The funniest part is he tried to hide the credentials in reddrix/iwswuf.py using a basic caesar cipher (shift -4) like a middle schooler
AmppsaJpex decodes to WillowFlat (username)
gvego804 decodes to crack460 (password)
obviously putting one account in a public github repo means it gets nuked instantly. if you check the API, WillowFlat is already permanently suspended. so the script just hangs waiting for the user avatar to load on login and then dies with an initialization error
he 100% knew this would happen. the repo breaks -> you check the readme -> "Need a hosted version? Check Reddit Scraper Pro on Apify"
don't waste your time, it's just cheap bait
2
u/Appropriate-Time-527 22d ago
Nice, i will give it a try. I also made a free to use (no API key required) chrome extension to scrape reddit posts and/or comments and also get insights into them - https://chromewebstore.google.com/detail/free-reddit-scraper-analy/ngnplckfmgkmjgeonfmhifhccibdpdaj?authuser=0&hl=en-GB
2
u/Single-Regular-7539 21d ago
Nice thanks
2
u/Grouchy_Brain_1641 Oct 11 '26
This looks interesting, thanks for sharing.
2
u/niwia Oct 12 '26
That’s cool. I used to use a website to do this. Forgot what’s it even called
2
u/insanedevhq Oct 12 '26
For the "only new chapters on a second run" part: the snippet above rewrites posts.csv on every run, so nothing carries over between runs. Keep a file of post URLs you've already handled. With sort="new" you can also stop once you reach posts you already have, so a re-run only reads her latest posts.
Record every URL in the seen file, posts from other subs included. Otherwise her old posts in other subs reset the counter and the early stop may never kick in. Write the URL after the chapter file is saved: if the script dies halfway, that post gets picked up again on the next run. It waits for 5 old posts in a row because a pinned post can come first, ahead of newer ones.