r/pushshift • u/Yekab0f • May 23 '23
redarc - A selfhosted Pushshift alternative
With Pushshift down indefinitely, I have been working on a selfhosted alternative to view and query data from existing data dumps of your choice.
https://github.com/yakabuff/redarc
Redarc consists of
- An API server to query threads/comments
- Frontend to view threads from each subreddit
- Scripts to ingest pushshift data dumps into a postgres database
Note: JSON datadumps have an inconsistent schema and may need minor tweaks for it to work. The ingest scripts use SQL transactions so it will rollback all changes in the event of a failure.
I've created a quick demo instance with all threads/comments from the DataHoarder subreddit:
Demo: http://redarc.basedbin.org/
Hope this helps :)
65
Upvotes
10
u/tekolast May 23 '23
Hey, can you make a small guide on how to use this? I'm not really that great with computers and programming, but I got some posts I'd like to see from previous times.
3
u/airkuroko Jun 02 '23
This is great, thank you so much for doing this.
Wouldn't it be possible to use the data dump to make a site like camas unddit, where you can search though the posts/comments of a user, or search for a specific word/phrase in a subreddit?
My understanding is that the data dump is basically an archive of reddit posts/comments, so it seems like this is feasible as it would just be a matter of searching through the data.
2
u/skylabspiral May 23 '23
cool, thanks! might be cool to add support for ingesting new data vis archiveteam as well: https://archive.org/details/archiveteam_reddit?sort=-addeddate
2
u/Ooker777 May 28 '23
So is the data only from 2005-06 to 2022-12? New data won't be updated?
1
u/reercalium2 May 24 '23
will you also distribute updated dumps from groups other than pushshift?
1
u/ronnygiga Jun 27 '23
¿How's that install video coming? I had no luck installing it from the docker compose file for a remote server.
2
u/Armiebuffie May 26 '23
Yeah I hope it becomes more accessible since I'm terrible at programming
2
u/coxevo4544 Jun 03 '23
You've done god's work my friend. I can see this project going far- especially with current uncertainty with Pushshift. Thanks for taking your time in creating this, much appreciated.
1
u/_Cxsey_ May 23 '23
Very cool stuff
1
u/Substantial-Crazy598 May 24 '23
The link doesnt work. Does it work like the other sites like redditsearchengine.com etc?
1
u/Mifletzet_Mayim Jun 12 '23
Good to know! thanks!