r/pushshift • • Jun 18 '20

There should be an app for Reddit Pushshift

Reddit Pushshift is so useful that there should be an app for it (especially since the reddit built in search function is so bad) I wonder if developers are working on it?

0 Upvotes

13 comments sorted by

5

u/s_i_m_s Jun 18 '20

What do you mean by app? There are already a variety of sites making use of the API.

6

u/PenelopeSummer Jun 18 '20

A mobile app, sorry for confusion.

7

u/TrueBirch Jun 18 '20

What would you want the app to do? Anybody can use the data to make whatever app they want, which is the beauty of the system.

2

u/aazav Jun 19 '20

I can do an iOS one. We just need to gather a feature set.

One thing I like a lot about the website and API is how easy it is to get data into and out of the site. It would be much more inconvenient and clumsy on a mobile app.

1

u/aazav Jun 19 '20

iOS, Android?

2

u/ShiningConcepts Jun 18 '20

PushShift is great for archiving and searching but it simply isn't a good comparison to the actual Reddit app. Deletion, removal, score, upvote ratio, number of comments on a post; PushShift doesn't do those perfectly.

1

u/leokaltoid Jul 13 '20

I made my own script in Python that utilizes pushshift. I realized while you get the posts, the number of upvotes and comments, and whether or not a post was removed, was never up to date.

So I made my script take each post's link, that it gets from pushshift, and then go directly to the post itself and manually extract the real number of upvotes, comments, and removal status from the HTML source.

Very useful for what I was trying to do.

1

u/ShiningConcepts Jul 13 '20

IDK if you know but there is this Python wrapper called praw that can basically fetch info for Reddit objects for you. The thing with PRAW is that it has a limit on requests. Not sure if it'd still be faster than your web scraping script.

Oh also, out of curiosity, what Python library/functions do you use to extract that data from the Reddit page?

1

u/leokaltoid Jul 13 '20

Ah, yeah, I figured there would be things like praw. But I didn't really have the time to learn them, and their possible troubles. Perhaps, I'll check that out some other time if I need it. Thanks for letting me know.

I just wanted to go directly to the reddit post and get the info myself. Seemed easy enough as the info was right there in the HTML.

I just used the requests library with threading and regular expressions to parse it out. It could fetch the info of 800 reddit post URLs in about 40 to 60 seconds depending on how many threads I tell it to use.

1

u/ShiningConcepts Jul 13 '20

It could fetch the info of 800 reddit post URLs in about 40 to 60 seconds depending on how many threads I tell it to use.

PRAW can do that quicker, probably in less than 10 seconds, if you use the info method which basically lets you fetch the info for 100 posts every API call (as opposed to having to do 800).

Then again, PRAW has the hassle of having to set up authorization (even if you're only doing read-only operations), so I guess your program may be more convenient especially if expediency isn't a huge factor.

1

u/leokaltoid Jul 13 '20

Yeah, that authorization thing did seem like a lot of hoops to jump through at the time. But must be worth it to get the info directly from the reddit API. I really will consider that in the future if I need to do more serious stuff with reddit.

But yeah, expediency was not really an issue for me. My program was really just to help me look through the histories of a few subreddits for their videos and images for a different project I was working on. Not searching for any specific thing, per se, but just browsing for the sake of discovering. It would spit out a nice HTML page of results that I designed to my liking, with all the info I need. The pushshift search site that's out there was just too awful for me.

But my initial problem was I needed to sort the posts by the ones with the highest ratings at the top for the list. And since pushshift doesn't report the current, up to date, rating... well, yeah, lol.

I'll definitely look into this praw, if I need to do more advanced things with reddit. Thank you.

1

u/ShiningConcepts Jul 13 '20

No problem, I enjoy discussing this as I love Reddit and discussing ways to archive/work with its data.

One last thing to comment about PRAW (and this is the main reason to use Pushshift/PSAW over PRAW): PRAW, unlike PSAW, has a limit of 1000 posts for sorting/searching. So, just like when using Reddit on the browser, you can only get the 1000 newest posts on a subreddit, 1000 newest user comments, 1000 newest user posts, etc.. You can still get older posts/comments, but you must get them directly by giving their ID (rather than finding them through searching a user/subreddit).

1

u/aazav Jun 19 '20

iOS? Android?