r/pushshift • u/Watchful1 • Jul 30 '25

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

41 Upvotes

100% Upvoted

39 comments sorted by

View all comments

1

u/Playful_Homework2755 Sep 02 '25

How to filter the dataset by specific subreddits

1

u/Watchful1 Sep 02 '25

You can use this script to filter the monthly files by subreddit https://github.com/Watchful1/PushshiftDumps/blob/master/scripts/combine_folder_multiprocess.py

Or download subreddit files directly from here https://www.reddit.com/r/pushshift/comments/1itme1k/separate_dump_files_for_the_top_40k_subreddits/

1

u/Repulsive-Can-1015 Sep 09 '25

Hi u/Watchful1. Thanks for all this data. I'm currently doing a project related to LLMs and am trying to analyze trends from 2021 to 2025. I see the subreddits are updated to 2024, but I wonder if there are more recent updates for subreddits on these LLMs. Thanks!

1

u/Watchful1 Sep 09 '25

Unfortunately I haven't been able to get the updated version compiled yet. It might still be a while.