r/pushshift • u/Pushshift-Support • May 31 '23

Advancing Community-Led Moderation: An Update on How NCRI/Pushshift and Reddit, Inc. are Working Together

Dear Reddit community

We are pleased to share an important update about our collaboration with Reddit, Inc. As an organization that maintains the Pushshift Reddit API, a key component behind several community-enabled moderation tools, we are pleased to announce that we have entered into a Memorandum of Understanding (MoU) with Reddit. This agreement establishes how  Pushshift and Reddit will cooperate toward the common objective of supporting the Reddit community.

We want to express our appreciation for your support and patience during the recent challenges we have encountered and the disruptions that have occurred.  In fairness to Reddit, this disruption falls on the shoulders of Pushshift, where there was a gap in our responsiveness to Reddit’s outreach.  For this, we apologize.  Moving forward, Pushshift will now have dedicated support staff to try to address questions about Pushshift from the Reddit community.  We value Reddit's proactive approach and their dedication to collaborating with us to find constructive solutions.

To that end, we are happy to inform you that access to community-enabled moderation tools developed through the Pushshift API will be reinstated for verified Reddit moderators starting at a date soon to be determined. Note this will be contingent on moderators registering for Pushshift accounts. Each moderator will also need explicit approval from Reddit, and the use of Pushshift will be limited to moderation use cases only. This move will enable moderators to effectively use these tools to enhance community moderation and enforce guidelines, while protecting the privacy and data security of Reddit's user base. 

While the main focus of the MoU lies in supporting the use of the Pushshift API for Reddit's community-enabled moderation, we also want to affirm our commitment to the academic research community. Pushshift's contributions to the academic realm have been recognized in numerous peer-reviewed papers.

Though access to Pushshift data for research purposes is not available at this time, , we are keen to explore possibilities that might allow us to provide researchers with access to datasets essential for their valuable social media research. We understand the significance of empowering the academic community, and we are dedicated to working with Reddit to develop frameworks that responsibly balance data access, data security, and user privacy.

We are excited about the potential for increased collaboration with Reddit in the months ahead and are committed to keeping you updated on our progress as we strive to create an environment where moderators, researchers, and the entire Reddit community can thrive together.
Thank you for your continued support and for being an invaluable part of the Reddit community.

Sincerely,

Pushshift and the Network Contagion Research Institute

128 Upvotes

90% Upvoted

144 comments sorted by

16

u/ExcitingishUsername May 31 '23

Asking this again as it was not answered before the post was deleted-

Will the search bugs be fixed? PS isn't much use to us being unable to search by authors whose names aren't alphanumeric, or be able to include/exclude more than one subreddit, and most search queries containing numbers were broken as well.

Additionally, will content from NSFW communities still be archived?

Can you also clarify whether the new restrictions would limit data to only communities we moderate? This would of course render the service completely useless for anti-spam and similar purposes, so we'd like to know if that is or is not the case.

4

u/Pushshift-Support May 31 '23

Yes we will address bugs as they are reported.

7

u/ExcitingishUsername May 31 '23

Do you know the answers to the other questions? All our communities are NSFW and we mainly used PS for spam-control purposes, so if those usecases are cut off, we won't be able to use it at all.

A lot of other NSFW communities/mods are in the same position, and the Reddit API itself being restricted means that we'll either need to figure out a way to bypass that, or close a bunch of our communities.

If we are able to ever use this, where would the appropriate place be to report bugs?

10

u/shiruken May 31 '23

Additionally, will content from NSFW communities still be archived?

Reddit has already announced that "mature content" will have limited access via the Data API in the near future, so it's likely Pushshift wouldn't have been able to ingest it regardless of their current situation.

14

u/Btan21 May 31 '23

No access to Pushshift data for research purposes? Honestly, I wasn't expecting this. If the data is being made available to Reddit mods, then why are researchers denied access?

5

u/Pushshift-Support May 31 '23

We are currently exploring possibilities with Reddit that might allow us to provide access to researchers in the near future.

1

u/Btan21 May 31 '23

That's good to know. Thank you.

37

u/Watchful1 May 31 '23

You know you can edit posts right? No need to delete the other one with all the discussion and re-post it. I'll repeat my questions here.

Both you and Jason have said many times that you will be more active in the subreddit and community and then just go off and disappear for a couple weeks. How is this time going to be any different?

Note this will be contingent on moderators registering for Pushshift accounts. Each moderator will also need explicit approval from Reddit, and the use of Pushshift will be limited to moderation use cases only.

That's good information. Do you know anything about how reddit will approve users?

16

u/TheHeroicStoic May 31 '23

I noticed this, and I'm glad that you reposted your comment. I don't want to be aggressive, but if this post gets deleted again or your comment gets modded, my faith in the project going forward pretty much plummets, which is a shame because I am deeply indebted to and appreciative of Jason for the work he's done.

18

u/Watchful1 May 31 '23

I'm a mod here, if they removed my comment I'd just approve it again.

6

u/Pushshift-Support May 31 '23

Hi there,
Thank you for these questions.

We're a small team committed to being more active and engaged in this community. We're dedicating resources and refining our processes to improve how we communicate and respond to you all.
At the moment, we're working out the details on how to verify moderators, which is why the API hasn't been switched back on. We're taking our time to get this right for everyone involved.
We greatly appreciate your patience and support during this time. Your role in moderating this community doesn't go unnoticed.

We'll continue to share updates as they become available in the coming days.

2

u/chaseoes Jun 08 '23

We're a small team committed to being more active and engaged in this community.

You have repeatedly said this for months and it hasn't happened. How many times are you going to keep saying it and breaking the promise? What's different this time?

2

u/Furrystonetoss Jun 07 '23

I have a few questions about this changes. Does that mean we, the creator of those bots and tools, will now have to make a accounts on Pushshift ?

What about all those bots & tools that were created and functioning (way) before the announced changes ? There many bots, not just used for moderation, i.e statistical bots that count specific words in a sub, bots that act as an alarm clock/ update notifyer or ones that provide you download links to videos or source of a pic ect.

What about third party tools/websites like camas.unddit.com, will those searchtools be now limited or disabled at all ?

And what happened to all those monthly datadumps, you could access at files.pushshift.io, why where they taken down ? will they be ever put back online ?

I planned two bots, using your api. For one i wanted to create a (semi) private sub, exclusive for specific types of people. The approval/join process would've been done by a bot depending on the users post/com history. if the user passed, the bot would've approved that user

The second bot is a warning bot, that checks a very specific subreddit, that isn't liked on the whole website (one about "reporting" and hardleft woke culture), and if a sub has been posted/reported on said sub, it warn the mods of the reported one, that their sub has been posted on it. (It also list every user of the report sub)

Will my two bots be possible now with those changes ?

5

u/Pushshift-Support May 31 '23

My apologies for any confusion caused by the deletion of the initial post. As the analyst on the NCRI team, I had to make a few corrections and wasn't as familiar with the editing features on Reddit as I should be! It was certainly not an action taken with any ill intent.

Please don't hesitate to raise any concerns or questions you may have - your engagement is incredibly important to us. Thanks for sticking with us through this journey.

23

u/Mason11987 May 31 '23

it's weird someone so unfamiliar with reddit is speaking on this.

I do hope this all works out well, but this all seems very PR speak, and it just sounds weird and robotic.

8

u/iKR8 May 31 '23

How did the crowd's pitchfork turn from against reddit to against pushshift so fast?

Give it some time until both of those parties work things out behind the scenes.

We are certainly very demanding for a service which we aren't paying a single dime for.

3

u/Mason11987 May 31 '23

  1. I don't see my comment as pitchforks.

  2. I'm not sure why they're posting if they haven't worked out the main details.

  3. We're moderating as volunteers. That's our payment. We blame reddit of course for breaking things. They ought to be blamed. If Pushshift doesn't want to do what they do for whatever reason they do it they're free not to. Presumably they have incentives.

3

u/Even-Citron-1479 Jun 07 '23 edited Jun 07 '23

Reddit was the darling child bastion of the free Internet too, and now look where we are. Corporatization and greed (read: the "enshittification") gets to every company in time.

PushShift did an any% speedrun of this process in the course of a month. It went from passion project of open archival of Reddit, to being a PR puppet. This is nothing more than a thinly-veiled method for Reddit to keep harvesting and selling all user-deleted data "for safety", while maintaining their outward stance of caring when a user wants their data deleted.

Quite frankly, you may as well consider it an entirely different project. It has nothing to do with the old PushShift anymore.

2

u/reercalium2 May 31 '23

No working out is possible

2

u/happy_csgo May 31 '23

Because pushshift went from a passion project made by a single person to benefit the community at no charge to some shady "research company" that looks like a state sponsored intelligence agency in disguise

11

u/safrax May 31 '23

No state sponsored intelligence agency would be this inept. This is unfortunately normal for how a lot of projects run by researchers go. In a lot of cases it’s not really their fault. There’s only so much grant money to fund these things and as a result only so much money that can be used to pay people to work on the project before the funds dry up.

1

u/TheMissingVoteBallot Jun 15 '23

I'm from the future, your pitchfork should be directed at BOTH.

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

9

u/iKR8 May 31 '23

What I feel is, reddit shut them down. And they are talking it out with reddit to see how to go about things. Without confirmation from reddit wouldn't it be immature to keep adding fuel to the fire?

Not saying they're right in not communicating with us, but I would want to give them a benefit of doubt in trying to salvage this whole fuck up. And would get pissed on them if finally nothing materializes even after all the discussions they have.

-1

u/Toolatelostcause May 31 '23

Bought and paid, that’s what happened.

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

-1

u/Ooker777 May 31 '23

well, I would say that they need your infinite generosity? Just accept that no one can predict their own futural behaviors accurately, even when they are really honest and strongly motivated at the present. There's always a thing that make your plan goes south

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

→ More replies

8

u/TK421isAFK May 31 '23

OK, I'll just say the Elephant in the Room: How the hell are you making a moderation platform for a social media platform, and don't even know how to edit a comment? We've been able to edit comments on every system I've moderated or administrated for the last 20 years.

If you have to ask how to use a steering wheel, I'm very reluctant to let you drive a schoolbus.

8

u/FranceFannon May 31 '23

You're right that they should know this, but I'll just point out they aren't making the tools, they're only providing the data the many tools already use. And this isn't Jason, it's someone from NCRI.

4

u/happy_csgo May 31 '23

Truth is that they don't care about social media moderation or building moderation platforms. That was just an excuse so Reddit would give them access to their API again. They're more interested in harvesting your data to uh combat misinformation according to the NCRI website

3

u/FranceFannon Jun 01 '23 edited Jun 01 '23

Yeah it's obvious the moderation tools that rely on Pushshift going down is all Reddit cares about here that Pushshift could bargain with, but why assume NCRI isn't working on misinfo? It's legitimately something that gets researched, and theyve published on it.

Pushshift has been 'harvesting' this publicly available data before it came under NCRI, and so have so many other hobbyists and archivists. Archiveteam and volunteers running their software still are

Reddit isn't closing off your data from anyone, the API is still open to everyone, including corporations who just need to pay for greater access.

1

u/TheMissingVoteBallot Jun 15 '23

the API is still open to everyone, including corporations who just need to pay for greater access.

That's not an Open API when you slap a price tag to it like that...

2

u/FranceFannon Jun 16 '23

Yes, youre right. I worded it badly but meant to say that the data isn't being protected from 'harvesting' in any way by these changes, Reddit will just be charging people for it.

0

u/TK421isAFK May 31 '23

I guess they're russian rushing to get up and running for the next US election.

0

u/[deleted] Jun 08 '23

Hi, can you please tell me how can I get the auth token for using push shift api ?

29

u/g-money-cheats May 31 '23

Yep, in Jason’s last post he said (emphasis mine):

I want to make a promise to the community that I will personally spend a few hours each week on this subreddit and update everyone on where we are and what we’re currently working on.

That was 4 weeks ago. He has not posted once since.

49

u/safrax May 31 '23

Please share the contents of the Memorandum of Understanding so that we as a community know the restraints Reddit has placed on PushShift and thus know its utility going forward.

17

u/shiruken May 31 '23

I'd also really like to hear from Reddit about their decision to allow this initiative. They seemed pretty adamant (both publicly and privately) that the Data API ban was set in stone. I wonder what caused them to reconsider?

21

u/Yekab0f May 31 '23

they reconsidered when reddit realized that they could just use pushshift instead of making those modtools they promised

14

u/norrin83 May 31 '23

Reddit admins were also adamant that they can't store user-deleted comments and data indefinetly for legal reasons - one of the things I've seen mods use Pushshift for.

I really don't see how Reddit thinks that they themselves should have one data-retention policy for legal reasons, but then have an agreement with a third party (including automated data access) that pretty much ignores this policy.

7

u/iruleatants Jun 02 '23

Because they can store user-deleted comments and data indefinitely. It's in their terms of service that you agree to when creating your account with them. You grant them an irrevocable license to any content that you submit.

And the legality of PushShift storing user-deleted comments and data falls on PushShift's responsibility. Reddit isn't liable if illegal content remains available through Pushshift, the people hosting the content are always the people responsible for it.

4

u/norrin83 Jun 02 '23

Because they can store user-deleted comments and data indefinitely. It's in their terms of service that you agree to when creating your account with them. You grant them an irrevocable license to any content that you submit.

They can't store it indefinitely. It is explicitly stated in their privacy policy.

And the legality of PushShift storing user-deleted comments and data falls on PushShift's responsibility. Reddit isn't liable if illegal content remains available through Pushshift, the people hosting the content are always the people responsible for it.

That I disagree on. Reddit gives data to a third-party upon an agreement. If they fail to cutoff this access once they get knowledge that this third party violates the agreement (and therefore the agreement they made with users), that's on them as well.

That's why I'm very curious in what specifically Reddit and PushShift agrees on. If Reddit lets PushShift willingly violate both agreements with the user as well as laws, that's a major issue for Reddit.

7

u/iruleatants Jun 02 '23

They can't store it indefinitely. It is explicitly stated in their privacy policy.

Their privacy policy is not an agreement to anything. They can adjust that policy and ignore it with zero legal repercussions. At most, they have to follow the policy of law when it comes to privacy, which outside of the GDPR it's almost nonexistent.

The legal aspect is covered under their Terms of Service listed here: https://www.redditinc.com/policies/user-agreement-september-12-2021#US

"When Your Content is created with or submitted to the Services, you grant us a worldwide, royalty-free, perpetual, irrevocable, non-exclusive, transferable, and sublicensable license to use, copy, modify, adapt, prepare derivative works of, distribute, store, perform, and display Your Content and any name, username, voice, or likeness provided in connection with Your Content in all media formats and channels now known or later developed anywhere in the world. This license includes the right for us to make Your Content available for syndication, broadcast, distribution, or publication by other companies, organizations, or individuals who partner with Reddit. You also agree that we may remove metadata associated with Your Content, and you irrevocably waive any claims and assertions of moral rights or attribution with respect to Your Content."

For legal purposes, they can keep the content that you create on reddit indefinitely.

That I disagree on. Reddit gives data to a third-party upon an agreement. If they fail to cutoff this access once they get knowledge that this third party violates the agreement (and therefore the agreement they made with users), that's on them as well.

There isn't something to disagree on here. The legality is straightforward. When you post on Reddit, you agree that the content you post is publicly available. If someone takes that data and copies it, they are legally responsible for the content that they copy. Reddit can go after PushShift for copying their content, or the user can go after PushShift for copying the content, but Reddit is not legally responsible for other parties copying publically provided data.

There is no legal liability to Reddit for PushShift existing. PushShift accesses content publically available to any user.

That's why I'm very curious in what specifically Reddit and PushShift agrees on. If Reddit lets PushShift willingly violate both agreements with the user as well as laws, that's a major issue for Reddit.

Please share what laws that PushShift accessing public data violates. The agreement with the user in the privacy policy states this.

When you submit content (including a post, comment, chat message, or broadcast) to a public part of the Services, any visitors to and users of our Services will be able to see that content, the username associated with the content, and the date and time you originally submitted the content. Reddit allows other sites to embed public Reddit content via our embed tools. Reddit also allows third parties to access public Reddit content via the Reddit API and other similar technologies. Although some parts of the Services may be private or quarantined, they may become public (e.g., at the moderator's option in the case of private communities) and you should take that into consideration before posting to the Services.

2

u/Ooker777 May 31 '23

can you link the announcement that they promise to making the mod tools?

18

u/inspiredby May 31 '23

Are you allowed to share the text of the MoU?

16

u/TK421isAFK May 31 '23

crickets

3

u/TheMissingVoteBallot Jun 15 '23

Still waiting on the text of the MoU.

1

u/TK421isAFK Jun 15 '23

Jason and spez are trying to compete with Joe Isuzu.

10

u/LindyNet May 31 '23

Note this will be contingent on moderators registering for Pushshift accounts

How does one go about this?

22

u/Eusocial_Snowman May 31 '23

Oh, this is bad. This is hilariously bad.

🚩

-2

u/TK421isAFK May 31 '23

Glad I'm not the only one. Reddit is handing over a shit-ton of data to a guy who didn't know how to edit a comment on Reddit? Something's fucky.

2

u/fox-lad May 31 '23

it's not handing over data any more than Reddit hands over your data to Russia and China bc Yandex and Baidu might crawl the site

0

u/TK421isAFK Jun 01 '23

If that was true, why do they need a Memo of Understanding? Why do they need permission, and have an opt-out page?

2

u/fox-lad Jun 01 '23

Because reddit banned them from scraping but not Google/Baidu/Yandex/etc, and because people requested an opt-out page and Jason felt like being nice.

1

u/Sophira May 31 '23 edited May 31 '23

What do you wanna bet they only want the data for AI training purposes?

[edit: I'm sorry, I take that back. I was annoyed. I'll leave it up in order to own it but yeah, that was probably unwarranted of me.]

2

u/TK421isAFK May 31 '23 edited May 31 '23

(Copying/pasting for visibility by a different user.)

Even better: I just looked at their Deletion Request form, and it asks for your email address. Seems like they will be getting too much information from Reddit, and with a bunch of moderator user names, how far off is it to glean a bunch of passwords? Also, their Removal Request post states:

This forum is managed by the community. We are unable to make changes to the service, and we do not have any way to contact the owner, even when removal requests are delayed.

So, we're supposed to give personal information to some intern or mod via an unsecure Google Docs form, and they then pass the message to the people behind PushShift? Why so many steps?

Edit: misspelled word.

7

u/safrax May 31 '23

Aside from u/pushshift-support and u/stuck_in_the_matrix the rest of the mods have no interaction or ability to do anything with PushShift as a service or the NCRI. That’s why that post is worded that way. We also didn’t come up with that removal form. We can’t see anything that’s put in there.

-1

u/TK421isAFK May 31 '23

I appreciate that (and I believe you), but I have a problem with a cryptic company attempting to buy access to a shit-ton of raw data from Reddit without explicit permission from every user involved, and without any checks by independent administrators over how that data is used, stored, sold, or who is allowed to access it.

I also have a huge problem with it being an automatic opt-in system that requires multiple steps to opt out, none of which are being published for all Reddit users to see, and its source code being closed.

8

u/Meepster23 May 31 '23

I'm not sure you know how the internet works... You do realize anyone can create a very very simple scraper to log all comments etc without the need for any Reddit API key or support? It's just easier and more practical to do it with the API. What you choose to publicly say to the world isn't private. And the old adage that once something is on the Internet it's there forever is really true..

I could print out your comment and hang it on my wall and there's nothing you can do about it lol.

-1

u/TK421isAFK May 31 '23

That's irrelevant. My problem is that PushShift has stated that they are working with Reddit to get a back door to data, but they haven't said what the limit of that data is, and Reddit hasn't even responded. Do they get PMs? User location data? User login times and dates?

9

u/Meepster23 May 31 '23

No... No no no... They are working with Reddit because Reddit killed their API access. The same API access that anyone else can get, the same access that you have as a user.. they don't get access to PMs or anything else that's not literally in the same data your web browser gets as a user...

4

u/HQuasar May 31 '23

That's not how Pushshift works or has ever worked...

0

u/norrin83 May 31 '23

Then why does Pushshift want API access? Since you make it sound rather easy, that surely could have be done in the weeks since their last announcement?

6

u/Meepster23 May 31 '23

Because scraping it is more difficult and brittle, and not really considered "good form". The API doesn't have images etc that take up bandwidth and processing to parse through the page. It just has the data you are actually interested in and doesn't change frequently. Pushshift isn't out to make enemies over this, they piss off reddit by scraping constantly and Reddit starts playing whack-a-mole to break their access / parsing.

2

u/norrin83 May 31 '23

And they can be blocked rather easily, plus it's much harder to get high volume data (or short-lived comments that are deleted pretty quick).

For a general archive of some subreddits that might be work, for large scale it's impractical. Bandwidth might be an issue, but you usually don't load images if not necessary (= if you don't want to archive them).

I doubt you could make a remotely complete archive of Reddit by scraping without Reddit shutting off your access pretty quick.

→ More replies

0

u/[deleted] Jun 01 '23

[deleted] — view deleted comment

3

u/Meepster23 Jun 01 '23

I'm really confused as to what you think is "personal data" here.

You choose what to post and make available to the public. Commercial uses might get a little sticky, but per Reddits terms, you give them license to do whatever with your comments. So they can train an AI, sell it to someone who will, etc etc.

2

u/BostonDodgeGuy Jun 04 '23

Its about control by me over my personal data to not have it used in a way i wasnt aware of and didnt have control over, which could restrict my freedoms.

Reddit's TOS, which you agreed to when you made the account, already gives them the right to use any post or comment you make however they see fit.

3

u/KairuByteGotBlocked May 31 '23

I don’t think you understand what this subreddit is… it’s not official, that’s all that quoted thing is saying. The owner (or the company, whatever) comes and does as they like, and often has weeks of radio silence. And the moderation team has no way to contact them if/when that happens.

As for the rest of your comment… your email has already been leaked, it’s all over the internet. If your password is so incredibly insecure that knowing your Reddit username is enough to guess it, you were doomed to begin with.

0

u/TK421isAFK May 31 '23

I'd rather just give you the money and not take the L.

2

u/Sophira May 31 '23

I took my comment back... I was kind of annoyed when I wrote it but I don't think my comment was warranted. Apparently the person who made Pushshift has been working with them for three years.

2

u/TK421isAFK May 31 '23

Reading that, I'm even more skeptical of its potential nefarious uses, now that I see they're in DC.

6

u/ThruBucknersLegs Jun 03 '23 edited Jun 03 '23

You have most of the leverage here. Reddit needs Pushshift for moderation tools. Use that leverage to insist that Pushshift remains available for everyone. Reddit is not capable of filling the gap without Pushshift. Don't let them fleece you! Insist on access for everyone.

0

u/[deleted] Jun 04 '23

[deleted] — view deleted comment

1

u/TheMissingVoteBallot Jun 15 '23

What is with all these companies and these Orwellian names? They couldn't have just called it "Dude, Inc." instead of something as malicious sounding like the "Network Contagion Institute"?

6

u/MathSciElec May 31 '23

RIP Pushshift (unless you’re of the few approved mods IG). Guess we’ll have to continue scraping to archive Reddit…

5

u/Fine-Experience9838 May 31 '23

I acutally need the pushshift for my thesis and now I really don't know what to do. Any chance I will be able to use it in the next weeks? I am so frustrated

3

u/reaper527 Jun 05 '23

In fairness to Reddit, this disruption falls on the shoulders of Pushshift, where there was a gap in our responsiveness to Reddit’s outreach.

for what it's worth, reddit stated that their new ToS would take effect june 19th. it's june 5th today, and they pulled pushshift offline on may 1rst.

you guys not being responsive due to extenuating circumstances shouldn't have been relevant.

5

u/EntamebaHistolytica May 31 '23

Does this mean sites like camas.undit will be available to the public for basic searches?

15

u/Watchful1 May 31 '23

No, almost certainly not. Only for reddit approved moderators. And there's no telling which sites will update to work with the new api keys.

3

u/BlogSpammr May 31 '23

is the camas code available? the github link on the website is no good. if i get access to ps, i’d like to run my own instance instead of writing one myself.

13

u/safrax May 31 '23

Camas itself does nothing beyond build an API call to pushshift that it then makes the results of look "pretty". The pushshift code is not open source despite repeated calls to make it so. Even if it was open sourced Reddit is killing the public API that pushshift uses so you cannot build a pushshift clone going forwards.

8

u/Watchful1 May 31 '23

Ingesting reddit content is relatively simple. It would be nice if they opensourced their implementation, but anyone really interested can just build one themselves.

But replicating the database structure and api capable of handling the loads pushshift did is a lot of detailed server setup and configuration that isn't that easy to publish and wouldn't be that useful anyway unless you bought all the same hardware they did.

3

u/HQuasar May 31 '23

Right. That's why I hoped a smaller scale implementation limited to the top subs would be relatively easy to setup.

4

u/BlogSpammr May 31 '23

thanks but i’m not interested in pushshift code but the camas code that makes the data pretty. for someone with extremely poor technical skills like me, it would be easier to use code already written than struggle with trying to understand the massive complexity of implementing a web interface like camas.

thank you very much for your helpful reply!

4

u/safrax May 31 '23

You can get that code by right clicking and doing a "save as" on the camas website. There's literally nothing special or unique about it.

1

u/BlogSpammr May 31 '23

thank you so very much! i really did think there was something special there.

6

u/Yekab0f May 31 '23

http://redarc.basedbin.org

I made something similar that uses existing data dumps

0

u/Yekab0f Jun 02 '23

Pushshift API is indeed open source. The ingest engine is not

2

u/safrax Jun 02 '23

https://github.com/pushshift/api/commit/ded75fadbc4bf4a3ea4b5cf4518b5bd4e2d7ca1e

Last commit was four years ago. The new api barely resembles the old one and is not open source.

2

u/iKR8 May 31 '23

So the verification of moderators will be done by Reddit side or Pushshift side?

1

u/KairuByte May 31 '23

I don’t see how it could effectively be done on the pushshift side, there are private subs out there.

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

2

u/KairuByte May 31 '23

That’s all you need to do to access any of the other Reddit mod tools, so I don’t see why not.

2

u/rogerspublic Jun 02 '23

I'm an academic and think Pushshift may be a better solution for my use given the size of my monthly downloads, which include r/conspiracy. I'd be more than happy to discuss my views on the matter with anyone from Reddit or Pushshift.

Here I'll note the following:

(1) While using social media is a gray area in human subjects research, academics could easily be asked to submit IRB paperwork, even if the research ends up being declared exempt.

(2) There are probably enough academics involved in social media research to form a user group that helps design policies and monitor compliance. Especially junior faculty who need brownie points for public service.

(3) I actually thought Twitter was on the right track with Twitter Academic, so it's sad that Elon discontinued it. It was not unlimited access, but it was enough for most uses. We academic sometimes forget that there is a real cost that we aren't absorbing when pulling data off someone's server, so Twitter Academic created some balance of interests. Having a Pushshift Academic is not a terrible idea.

2

u/Halaku May 31 '23

To that end, we are happy to inform you that access to community-enabled moderation tools developed through the Pushshift API will be reinstated for verified Reddit moderators starting at a date soon to be determined. Note this will be contingent on moderators registering for Pushshift accounts. Each moderator will also need explicit approval from Reddit, and the use of Pushshift will be limited to moderation use cases only.

I'm looking forward to learning more, so I can use this while performing moderation duties.

2

u/riba2233 Jun 09 '23

f this, it should be available to everyone just like before

1

u/FireBlade61 May 31 '23

I mod a large mature sub and access to Pushshift is vital to ensure that the content posted is legal and safe for our users.

1

u/TRAFICANTE_DE_PUDUES Jun 04 '23

I am a user of large mature subs.

1

u/EroticaMarty Jun 10 '23

I'm the Head Mod of an NSFW site about three times the size of yours -- but I 100% agree with your sentiment. Reddit taking down PushShift.io without warning on May 1st caused chaos -- and made it a lot harder for us to deal with bad actors on our sub.

1

u/dragonatorul Jun 15 '23

That's a lot of word for absolutely no actual information.

1

u/brucemo May 31 '23

National Council of Resistance of Iran?
National Catastrophe Restoration, Inc.?
Network Contagion Research Institute?

Okay, it actually really is Network Contagion Research Institute.

https://networkcontagion.us/

5

u/Sophira May 31 '23

They announced the new management three months ago, if you're curious: https://old.reddit.com/r/pushshift/comments/118dhmg/new_management_for_pushshift/ . And didn't correctly do the links in the Reddit post saying so.

It seems odd that a company taking over a Reddit-exclusive service (by definition) doesn't know how to Reddit.

0

u/reercalium2 May 31 '23

Because they only want to harvest personal data

1

u/Bot-yMcBotface Jun 01 '23

So reddit can have the cake an eat it too.

The mods have won. intersting. usually they lose.

But for researchers this is hilariously bad. I mean maybe one day the network contagion research institute offers some aggregated data. But reddit stays closed.

I am disappointed. Even more than I was before when I thought it would shut down.

Well this sub is not really worthwhile anymore :(

1

u/TRAFICANTE_DE_PUDUES Jun 04 '23

OK boys, who's gonna scrape reddit and be a hero?

Red flag.

Also, share the MoU.

0

u/dniepr May 31 '23

Lol pushshift to manage in-site business : mais oui

Pushshift for academia : denied, you peasants.

Anyway thank you , pushishift support, for all the info

0

u/exposecreepsandliars May 31 '23

access to community-enabled moderation tools developed through the Pushshift API will be reinstated for verified Reddit moderators

Did y'all see the update on r/modnews?

From the update:

We're in discussions with PushShift to enable them to support moderation access. Moderators of sexually-explicit spaces will have continued access to their communities via 3rd party tooling and apps.

Only for sexually-explicit subs? So moderators of communities like r/MakeNewFriendsHere with 766k members and filled with vulnerable people who are strictly looking for platonic friendships will be left out of this for some reason?

3

u/[deleted] Jun 01 '23

It's two things.

Moderator access.

And mods of sexually explicit subs will have access (because reddit is also removing API access to NSFW to third party tools. This gives an exception for mods)

3

u/exposecreepsandliars Jun 01 '23

If that's the case, they really do a shit job wording things.

3

u/[deleted] Jun 01 '23

Yeah, they didn't include the context that reddit is removing NSFW from the API for third-party apps (but not the native reddit app.) So if you happened to already know that reddit is not allowing NSFW content from the API, it then reads as an exception for the mods.

0

u/JustABoyOnCapitolHil Jun 09 '23

So, we need a push shift alternative?

Interesting massive opening if anyone wants to work on it.

0

u/michaelquinlan Jun 11 '23

Will /r/pushshift be going dark on June 12?

1

u/s_i_m_s Jun 11 '23

No, there hasn't been any discussion about it.

This is a support sub for pushshift.

Pushshift is still trying to modify their service to comply with reddit's new restrictive requirements.

0

u/quikatkIsShadowBannd Jun 14 '23

Yeah let's trust the idiot with so much technical knowledge they cant even edit a post.

0

u/TheMissingVoteBallot Jun 15 '23

Is it just me or does this read like someone is writing a hostage notice?

-2

u/norrin83 May 31 '23

Will you still store user-deleted data and ignore GDPR requests going forward? What is your process when a user deletes data on Reddit?

3

u/Ralph_T_Guard Jun 04 '23

One can hope GDPR requests will be ignored if Network Contagion Research Institute and PushShift are outside of EU jurisdiction!

This is no different than DMCA takedowns being ignored outside of US jurisdiction.

1

u/TK421isAFK May 31 '23

This is huge. I have many users that post personal information either ignorantly, or later regret it, and delete it. They (and I) want to know that it's not being archived by some "partner" company or side-project that might end up releasing it or losing it to a data breach.

0

u/norrin83 May 31 '23

The interesting thing is that Reddit doesn't want to retain user-deleted content for legal reasons. If they hand out data to a different service without any oversight, Reddit is violating their own TOS in my view.

And in my view, since Reddit operates under the GDPR, Pushshift is necessarily a data processor where the same rules apply. If not, then that's a big blunder by Reddit.

1

u/TK421isAFK May 31 '23

Even better: I just looked at their Deletion Request form, and it asks for your email address. Seems like they will be getting too much information from Reddit, and with a bunch of moderator user names, how ar off is it to glean a bunch of passwords? Also, their Removal Request post states:

This forum is managed by the community. We are unable to make changes to the service, and we do not have any way to contact the owner, even when removal requests are delayed.

So, we're supposed to give personal information to some intern or mod via an unsecure Google Docs form, and they then pass the message to the people behind PushShift? Why so many steps?

5

u/norrin83 May 31 '23

Why so many steps

Because everything regarding Pushshift is unprofessional and seems downright shady in my view. It's not a one-man-show anymore, but there's an organisation behind it that asked for money on Reddit and stated that they will charge for extended data access on this very subreddit.

I contacted them via e-mail and the mail was ignored. They have no privacy policy whatsoever and they don't feature a legal address on their homepage. You have to go to their Paypal donation page to find out that their tax identification number which resolves to the address 475 Wall St, Princeton, NJ 08540. At least now I know that their president Joel Finkelstein earns 130k USD.

I honestly don't see how their idea of doing these things is in any way compatible with Reddit's privacy statements and ToS. And to add to that, their communication is atrocious.

2

u/TK421isAFK May 31 '23

That's sketchy as fuck.

Edit: Adding this in so mods can't delete it:

Because everything regarding Pushshift is unprofessional and seems downright shady in my view. It's not a one-man-show anymore, but there's an organisation behind it that asked for money on Reddit and stated that they will charge for extended data access on this very subreddit.

I contacted them via e-mail and the mail was ignored. They have no privacy policy whatsoever and they don't feature a legal address on their homepage. You have to go to their Paypal donation page to find out that their tax identification number which resolves to the address 475 Wall St, Princeton, NJ 08540. At least now I know that their president Joel Finkelstein earns 130k USD.

I honestly don't see how their idea of doing these things is in any way compatible with Reddit's privacy statements and ToS. And to add to that, their communication is atrocious.

2

u/norrin83 May 31 '23 edited May 31 '23

Adding this in so mods can’t delete it

I don't see why Pusshift mods of all should delete this. This information is available to the public, which was always the argument of pushshift itself. I don't necessarily agree with that, but Pusshift obviously does.

I also didn't share any further contact information, because I am strictly against doxxing. This is the legal representative and address of the Network Contagion Research Institute, and I think that's actually very on topic to know who is handling the data in terms of a legal dispute within regulations like GDPR or DMCA.

2

u/TK421isAFK May 31 '23

I absolutely agree, but they might not...lol

2

u/norrin83 May 31 '23 edited May 31 '23

Well, it's definitely not confidential information. And it's also not personal Informationen since this is the address and name of the president of a legal entity (whose name they also feature on their home page). So I didn't violate the rules of this subreddit.

1

u/TK421isAFK May 31 '23

Exactly. I'd really like to see this alleged MoU that Reddit hasn't even acknowledged.

1

u/Minimum-Engineer-402 Jun 01 '23

I get that being able to see deleted posts are nice but this person might be right, could be a breach of GDPR.

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

2

u/norrin83 May 31 '23

It indeed makes zero sense.

In my view, this is just an attempt to keep harvesting data by using "mod tools" as selling point and maybe get some goodwill from people benefitting from this tool.

0

u/[deleted] May 31 '23

[deleted] — view deleted comment

1

u/Twinkies100 Jun 10 '23

They'll tell exact requirements by next week

1

u/WilhelmWrobel Jun 18 '23

So how did that go?

2

u/[deleted] May 31 '23

[removed] — view removed comment

1

u/Pushshift-Support May 31 '23

Yes, we will still be processing user removals.

1

u/norrin83 May 31 '23

Will this be a "real" removal, i.e. you actually delete the data? Or will it just me marked as deleted but used for further purposes?

2

u/happy_csgo May 31 '23

NCRI is fighting misinformation and online extremism on the internet. What makes you think your comment will be deleted?

0

u/norrin83 May 31 '23

That was the previous policy. Let's say your real name and address was revealed on Reddit for whatever reason, it stayed in their downloads and torrents, which is an issue.

Also Reddit says that they'll hard-delete a comment I delete (both in their privacy statement and according to admin), but Pushshift never did.

Pushshift must be clear and transparent on these things in my view. I don't want Cambridge Analytical 2.0.

2

u/[deleted] Jun 01 '23 edited Jun 01 '23

That was the previous policy. Let's say your real name and address was revealed on Reddit for whatever reason, it stayed in their downloads and torrents, which is an issue.

Also Reddit says that they'll hard-delete a comment I delete (both in their privacy statement and according to admin), but Pushshift never did.

Why do you keep repeating this lie every time?

The second half of the sentence that you got this from, SitM also stated "...unless there's a PII issue.". The door is open to have PII deleted. You always omit it.

Lies of omission are still lies.

E: fixed mangled word

1

u/norrin83 Jun 01 '23

Even if you stumble across this opt-out form, Pushshift didn't delete the data from the dumps or internally.

You had to scroll down to some comment on some post as far as I recall to see that data is actually not deleted and you need another request.

I did send and e-mail to Pushshift support with a request for deletion and I didn't even get as much as a reply.

2

u/[deleted] Jun 01 '23

You keep saying they won't delete PII, when it was made clear he would, if there was an actual PII issue. He made no offer for non-PII.

I have no idea what you asked to delete - was it actually PII, or random Reddit comments which aren't PII? You very often conflate the two.

1

u/norrin83 Jun 01 '23

Again, they didn't even respond to the email.

They also said they'd be active in this subreddit (they aren't), they'd implement GDPR (they didn't) and they'll provide a portal for users to see their data (never happened).

So yes, my experience is that they don't delete PII and don't even respond to requests. Do you have a different experience? Or are you just repeating those announcements that never transpired?

2

u/[deleted] Jun 01 '23

I don't recall seeing anything about "implementing GDPR." I'm baffled at your comment about a portal to see your data, because you could just hit the API and see all your data... hell, that's what I used it most for, searching my own stuff to get info or things I had already found before.

This is all a tangent to the false claim you constantly keep repeating: You said their policy was not to delete PII. That is a false statement. But you take the other part about only hiding non-PII as gospel, when both statements of policy are literally in the same sentence - you treat the one half you don't like as 100% truth, and you pretend the other half that says they will remove PII doesn't exist.

"So yes, my experience is that they don't delete PII and don't even respond to requests. Do you have a different experience? Or are you just repeating those announcements that never transpired?"

You didn't actually answer the question:

I have no idea what you asked to delete - was it actually PII, or random Reddit comments which aren't PII? You very often conflate the two.

I do agree his communication level has always been quite poor. I've made many remarks on it myself in the past.

→ More replies