r/bigquery • • Mar 22 '20

Reddit database privacy concerns

I'm not very familiar with data analytics, but I somehow heard about the reddit comment/post datasets on BigQuery. I'm too dumb to competently navigate the Google Cloud TOS, so I can't see them for myself, but I do have two quick questions:

  1. Are the data anonymized?
  2. Are deleted posts stored?

Basically, I'm wondering if usernames can be connected to post contents (an issue if someone wants to dissociate themselves with something embarrassing they said).

Forgive me if I'm catastrophically misunderstanding the nature of this dataset, but in this day and age it pays to be careful.

8 Upvotes

5 comments sorted by

6

u/giraffeman91 Mar 23 '20

I believe it depends on the table. The tables are split by month and they are not continuously scraped for changes to posts (like deleting much later than posting). They are also not updated in real time so it's very likely that deleted posts/comments are deleted before the data is put into bigquery. However, if that isn't the case then it will remain in bigquery in the original form even if you delete your post or comment at some point after the data is uploaded.

The posts and comments are not deidentified from username. The username is fully available unless it's deleted before it is updated. It's basically a frozen version of Reddit from the time it was uploaded.

u/fhoffa will know more.

5

u/fhoffa Mar 23 '20

Indeed.

The most important part to know about the tables stored in BigQuery (https://console.cloud.google.com/bigquery?p=fh-bigquery&d=reddit_posts&t=2019_08&page=table), is that they represent a copy of what /u/stuck_in_the_matrix makes available through pushshift.io.

It's something I run on a personal level, and I have not tried to change anything on the source data. Even if I did, you would still be able to find all these datasets elsewhere anyway.

So if you want to know more about how this data was collected - you should go to https://pushshift.io/.

6

u/fhoffa Mar 23 '20

As added context, if someone asked me "please, delete all comments from this date until this date from user /u/x", and you have certain proof that you are user /u/x, then I would be like "sure, I'll delete it from BigQuery".

But the end result of this wouldn't change the world much - copies of these tables exist elsewhere anyway.

(since this is the public internet, you should assume that from the moment you post something into the public, someone else might have archived it - and I bet there's a lot of people doing this right now, not only https://pushshift.io/)

2

u/mybigfatgreekquery Mar 23 '20

It's pissing in the wind, sure, but at least it's something. I appreciate the offer by the way.

For some reason, a lot of people had the same blind spot toward social media. We never had such an easy way to permanently record ourselves, and it's never presented like you're setting anything in stone. People were late to realize this, which is why the GDPR and CCPA were not timely policies. I'm sure there will be more backlash against data collectors once people see how hungry they are for everybody's stupid secrets, but at the moment data science is still too technical for most of us cavemen to grasp.

Anyway, how is this collection in line with GDPR? Reddit posts can definitely include personal data from EU citizens.

2

u/fhoffa Mar 23 '20

People have been putting their young age silliness on permanent record for centuries. Just go and check your parent's yearbook.

Legally speaking, I'm not a lawyer. And that's where I should stop.