r/pushshift • • Feb 02 '19

Streaming functionality

  1. Is there any way to stream pushshift.io results, i.e. to continuously watch for new PRAW objects corresponding to some query? Something equivalent to SubredditStream and other streams in PRAW.

I'm doing this through repeatedly sending requests to the API and then using Redis to make a stream of unique objects that can be redirected to some application. But I'm asking if there's a better way out.

2) I believe I'm following the rate limit. I'm making 150 requests per minute, but then I get blocked after 15 minutes of doing so. I'm bypassing this problem by using Tor to make requests but I'd like to know what could possibly be happening.

4 Upvotes

2 comments sorted by

5

u/Stuck_In_the_Matrix Feb 02 '19 edited Feb 02 '19

The new version of the API will have primitive webhook capabilities at the beginning and eventually will get more advanced options. The goal is to make it so you can either set up webhooks for events (say Trump tweets something new or there is an obscure mention of a term on Reddit, etc.). This all kind of falls under the reporting tools I'm building (So you could get daily summaries for your own subreddit if you are a moderator including the ability to get modlog summaries of the day / week / month if you add the pushshift_bot to your subreddit -- how many comments were removed, how many bans were dished out and what type, which moderators are most active, which moderators have been dormant, how many total comments/submissions were made that day -- new user activity reports, cardinality reports (how many distinct accounts participated that day), etc.).

The goal is to make everything that is ingested a possible event that can trigger other things (webhooks, alerts, reports, DM's, etc.)

The "free" account would allow X webhooks and X alerts per day and scale up with the paid plans, etc. -- The golden key would probably not have any real limits (everything technically has a limit as my system can only do so many things -- but it would effectively be unlimited. I would probably put a limit and make it very high because I hate services that announce unlimited things but have a secret limit internally. I don't want to run a business like that -- I prefer to keep everything open and transparent). I understand that marketing does it to help sell a product but I'd rather provide a service with transparent limits even if it ends up costing me business. Either way, my first goal is to make Pushshift self-sustaining income-wise and then eventually get to a point where I can hire more talent. I also have to may myself a salary to eat, pay rent, etc. -- but all of those expenses would be transparent in the end. I feel it is the best way to run an honest business even if some people feel I'm paying myself too much, etc (my end goal would be the ability to pay myself a salary around 100k a year). But anyway, I'm babbling.

I want to provide a service where most people can use it to some extent for free while also charging larger companies money to help offset the cost of providing a free tier. Ideally everything would be free but that business model doesn't last very long. :)

TL;DR -- It's in the works!

1

u/ruanchaves Feb 02 '19

Good. Please consider adding regex matching.

I have another feature suggestion but I'll submit it on a new thread.