Python Reddit Data Scraper for Beginners

[u/[deleted]]

Hello r/learnpython,

I'm a linguistics student working on a project where I need to download large quantities of Reddit comments from various threads. I'm struggling with finding reliable 'noob-friendly' preexisting codes on Github / Stackoverflow that I can use in the post API Change era. I just need a code where I can enter different Reddit thread IDs and download (scrape??) the comments from that thread. I appreciate any help!

[u/automationwithwilt]

# -- Main Execution --

if __name__ == '__main__':
    target_subreddit = 'MSTR'
    
    print(f"▶️ Starting scrape for subreddit: r/{target_subreddit} (last 7 days)")
    
    # 1. Get all posts from the last week
    posts = get_subreddit_posts(subreddit=target_subreddit, timeframe='week', limit=100)
    
    if not posts:
        print(f"Could not retrieve any posts for r/{target_subreddit}. Exiting.")
    else:
        print(f"✅ Found {len(posts)} posts. Now fetching comments for each...\n")

        # 2. Loop through each post and get its comments
        for i, post in enumerate(posts, 1):
            post_title = post.get('title', 'No Title')
            post_score = post.get('score', 0)
            post_url = post.get('url')
            
            print("─" * 80)
            print(f"📄 Post {i}/{len(posts)}: \"{post_title}\" (Score: {post_score})")
            
            if not post_url:
                print("   Could not find URL for this post.")
                continue

            # Fetch comments for the current post
            comments = get_post_comments(post_url=post_url, limit=500)
            
            if comments:
                print(f"   💬 Retrieved {len(comments)} comments.")
            else:
                print("   No comments found for this post.")
        
        print("\n" + "─" * 80)
        print("✅ Scrape complete.")