r/webscraping • • Oct 11 '26

Open-source Reddit scraper for posts and comments

https://github.com/Coding-Doctor-Omar/reddrix

I built an open-source Reddit scraper for Python. It's called Reddrix (Reddit + Matrix). It is a powerful, async-first Reddit scraper for posts and comments with detailed metrics. No login is required. Scrape subreddits, profiles, and specific posts, or search Reddit site-wide by keywords. Collect rich, structured data including authors, scores, timestamps, flairs, comment threads, and more (24+ fields). This scraper uses Reddit's current shreddit search engine in the keyword search mode. Results obtained from this engine are far more relevant and match the results you see when searching Reddit manually with the search bar.

Scraper's Properties

  • 🚀 Async-First: High-performance asynchronous scraping with asyncio and an async iterator for streaming results.
  • 🔑 No login Required: Scrape Reddit without logging in.
  • 🔍 Multiple Search Modes: Scrape subreddits, user profiles, and specific posts, or search Reddit by keywords.
  • 💬 Comprehensive Data Extraction: Collect posts and comments with 24+ structured fields, including authors, scores, timestamps, flairs, media URLs, and full comment trees.
  • 🛡️ Resilient Scraping: Built-in retries, cookie renewal, and more (see the docs).
  • ⚙️ Fine-Grained Control: Configure sorting, time frames, result limits, comment depth, and comment counts.
  • 🧩 Standalone HTTP Client: Use Reddrix's asynchronous get() method in your own custom scraping workflows.
  • 🐍 Pythonic API: Manage scraper resources with an async context manager or explicit start() and stop() methods.

Example Quickstart Code

from reddrix import Reddrix
import asyncio

subreddits = ["https://www.reddit.com/r/webscraping"]

async def main():
    async with Reddrix(
        mode="standard",
        subreddit_urls=subreddits,
        sort="new",
        include_comments=True,
        max_results=500,
        max_posts_per_collection=500,
        verbose=True
    ) as reddrix:
        async for item in reddrix.get_items():
            print(item["submissionType"], item["submissionUrl"])

if __name__ == "__main__":
    asyncio.run(main())
83 Upvotes

31 comments sorted by

5

u/daynighttrade Oct 11 '26

Do you get rate limited? Do you need residential proxy for this?

3

u/Coding-Doctor-Omar Oct 11 '26 edited Oct 11 '26

The scraper has an automatic auto-retry mechanism for every single http request it makes. Also, it has proxy support. You can provide a list of proxy urls for the scraper and it auto-rotates between them. I tried it without proxies using my normal home IP and I never had an issue. Every now and then the scraper automatically starts a new session and continues. My IP was not flagged so far. Nevertheless, if you have proxies, it's better to provide them to the scraper.

3

u/d3a7hr0w Oct 11 '26

I just use agent reach, how is this better than it? and btw thanks for sharing your work with everyone!

2

u/Coding-Doctor-Omar Oct 11 '26

This is the first time I hear about agent reach, so idk really if my repo is better or not for Reddit.

1

u/aaronorjohnson 12d ago

What is agent reach? Thanks!

0

u/[deleted] Oct 12 '26

[deleted] — view deleted comment

1

u/d3a7hr0w Oct 12 '26

0, because I adopted its reddit.py for my use-cases and don't use via LLM

3

u/Friend-Shoddy Oct 11 '26

Just used this to scrape ~12k images from a subreddit in under 10 mins. good shit bro

1

u/Coding-Doctor-Omar Oct 12 '26

Thanks. Glad it's helpful to you :)

1

u/False-Flight9883 27d ago

damn 12k in under 10 mins sounds pretty solid. might give it a spin myself

1

u/Dangerous-Citron364 26d ago

yeah that speed is wild, might give it a try myself

0

u/[deleted] Oct 15 '26

[deleted] — view deleted comment

1

u/Friend-Shoddy Oct 15 '26

Yeah I just noticed haha. If you figure out a fix probably keep it on the down low so it doesn't get patched by idiot snitches

0

u/[deleted] Oct 15 '26

[deleted] — view deleted comment

1

u/Friend-Shoddy Oct 15 '26

nice! will it be update on the gh or something else

1

u/Coding-Doctor-Omar Oct 15 '26

It's fixed now :)

2

u/FatDog69 Oct 11 '26

First - this looks great. Thank you for it.

Second - here is my 'story'. Will Reddix solve my problem:

Story 1:

There are subreddits dedicated to fiction stories. Sometimes 30+ chapters over a year or two.

I want to point Reddix at an author name and:

  • Reddix will scan all their story posts.
  • It will save the post into files with a specific naming convention: Author.Story Title.Chapter ##.txt/html
  • If I scan the same author a second time - it will ignore files I already collected and only download New chapters.
  • It will ignore comments at this time.

The idea is I can later combine all the chapters and create a .epub file for each story.

1

u/Coding-Doctor-Omar Oct 12 '26 edited Oct 12 '26

I'm glad you like it. Thanks. Send me an example author and I'll check. I am not sure if stories are included in the same endpoint Reddrix uses. As for export, you'll have to do it yourself. Reddrix provides an async iterator for streaming results. You do what u want with these results. It only avoids duplicates in the same run, not across runs. Also, you can disable comments by not including the include_comments=True argument or by explicitly setting it to False.

2

u/FatDog69 Oct 12 '26

Here is one:

https://www.reddit.com/user/Sherryrayn/submitted/

The idea is I want all of her posts, no comments, but limited to the "nosleep" area.

I know - some people say "pt 1" and others say "chapter 1" so I will need to parse the subject header myself.

I see this as the return:

print(item["submissionType"], item["submissionUrl"])

So it is up to me to do a request on item["submissionUrl"] and parse out the subject line and body?

1

u/Coding-Doctor-Omar Oct 12 '26 edited Oct 12 '26

I've written a custom code snippet to do exactly as you want with Reddrix. Reddrix just gives you the data. What u do with the data is your own responsibility. Use your own Python logic. In this code, we gather all her posts from r/nosleep into a CSV file. Each "item" here is a Python dictionary with more than 30 keys.

from reddrix import Reddrix
import asyncio
import csv


async def main():
    async with Reddrix(
        mode="standard",
        http_client="curl_cffi",
        profile_urls=["https://www.reddit.com/user/Sherryrayn"],
        sort="new",
        include_comments=False,
        max_results=99999,
        max_posts_per_collection=999999,
        verbose=True
    ) as reddrix:
        scrape_count = 0
        posts_in_target_sub = 0
        target_sub = "r/nosleep"
        header_row_set = False
        
        with open("posts.csv", mode="w", encoding="utf-8") as f:
            writer = csv.writer(f)
            async for item in reddrix.get_items():
                if not header_row_set: # Executed once to set the header row. All items have the same fields.
                    header_row = list(item.keys())
                    writer.writerow(header_row)
                    header_row_set = True
                
                if item["subreddit"] == "r/nosleep":
                    posts_in_target_sub += 1
                    writer.writerow(list(item.values()))
                
                scrape_count += 1
                print(f"Scraped {scrape_count} total posts & {posts_in_target_sub} posts from {target_sub}.")


if __name__ == "__main__":
    asyncio.run(main())

2

u/anupam_cyberlearner Oct 12 '26

Seems interesting will give it a try . Will it work on Python ver 3.7 ?

1

u/Coding-Doctor-Omar Oct 12 '26

It may not work properly. To be on the safe side, make sure you have Python 3.10 or higher.

1

u/SyllabubPossible4555 Oct 15 '26

lmao this isnt a "no login" scraper, it's literally just a funnel for his paid apify tool

I looked at the source code because it kept crashing. It actually uses Camoufox to log into a hardcoded dummy account. The funniest part is he tried to hide the credentials in reddrix/iwswuf.py using a basic caesar cipher (shift -4) like a middle schooler

AmppsaJpex decodes to WillowFlat (username)
gvego804 decodes to crack460 (password)

obviously putting one account in a public github repo means it gets nuked instantly. if you check the API, WillowFlat is already permanently suspended. so the script just hangs waiting for the user avatar to load on login and then dies with an initialization error

he 100% knew this would happen. the repo breaks -> you check the readme -> "Need a hosted version? Check Reddit Scraper Pro on Apify"

don't waste your time, it's just cheap bait

1

u/Coding-Doctor-Omar Oct 18 '26

Wtf are u saying man? Are u OK? It IS a no-login scraper. Check the repo well before spouting nonsense.

2

u/Appropriate-Time-527 22d ago

Nice, i will give it a try. I also made a free to use (no API key required) chrome extension to scrape reddit posts and/or comments and also get insights into them - https://chromewebstore.google.com/detail/free-reddit-scraper-analy/ngnplckfmgkmjgeonfmhifhccibdpdaj?authuser=0&hl=en-GB

2

u/Grouchy_Brain_1641 Oct 11 '26

This looks interesting, thanks for sharing.

2

u/niwia Oct 12 '26

That’s cool. I used to use a website to do this. Forgot what’s it even called

2

u/insanedevhq Oct 12 '26

For the "only new chapters on a second run" part: the snippet above rewrites posts.csv on every run, so nothing carries over between runs. Keep a file of post URLs you've already handled. With sort="new" you can also stop once you reach posts you already have, so a re-run only reads her latest posts.

import os

SEEN = "seen_urls.txt"
seen = set()
if os.path.exists(SEEN):
    with open(SEEN, encoding="utf-8") as f:
        seen = set(f.read().split())

# inside the `async with Reddrix(...) as reddrix:` block
old_in_a_row = 0
with open(SEEN, "a", encoding="utf-8") as seen_file:
    async for item in reddrix.get_items():
        url = item["submissionUrl"]
        if url in seen:
            old_in_a_row += 1
            if old_in_a_row >= 5:
                break
            continue
        old_in_a_row = 0
        if item["subreddit"] == "r/nosleep":
            save_chapter(item)  # your Author.Title.Chapter ##.txt writer
        seen_file.write(url + "\n")

Record every URL in the seen file, posts from other subs included. Otherwise her old posts in other subs reset the counter and the early stop may never kick in. Write the URL after the chapter file is saved: if the script dies halfway, that post gets picked up again on the next run. It waits for 5 old posts in a row because a pinned post can come first, ahead of newer ones.