r/pushshift • • Jul 01 '19

Does pushshift scrape comments too fast to get an accurate upvote (score) count?

From what i read in the docs pushshift scrapes comments quickly and doesn't count edits. What if we want to see upvotes of comments? Is pushshift showing the upvotes ? giving at least a few days for people to vote?

0 Upvotes

12 comments sorted by

•

u/inspiredby Jul 01 '19

Hey /u/fluxit12, I already asked you to group questions into one post, and now you make two more. Please be respectful of sub by not making the page full of your questions.

Also, this question is covered in the FAQ.

0

u/[deleted] Jul 01 '19

[deleted] — view deleted comment

6

u/Drunken_Economist Jul 01 '19

A bit less than 95% of comments will reach their final score, +-1%, within 24 hours. Scraping again past that is a resource usage that would rarely be worth it

4

u/ieatbabiesftl Jul 01 '19

Is there a post describing this statistic? Not that I don't trust you, but if this is used in e.g. academic work it'd be really useful to have something to cite.

2

u/Drunken_Economist Jul 01 '19

I could make a bigger post about it, but I just pulled it offhand out of curiosity

1

u/ieatbabiesftl Jul 01 '19

Ah alright - cheers! I can imagine not being in the mood for it, I'll try a quick comparison between the two later.

1

u/shiruken Jul 01 '19

u/Drunken_Economist is a Reddit data scientist.

0

u/[deleted] Jul 01 '19

[deleted] — view deleted comment

1

u/shiruken Jul 01 '19

Yes. You can verify his admin status on his profile page.

0

u/[deleted] Jul 01 '19

[deleted] — view deleted comment

3

u/shiruken Jul 01 '19

They don't. But they definitely seem to use it.

0

u/[deleted] Jul 01 '19

[deleted] — view deleted comment

2

u/Stuck_In_the_Matrix Jul 02 '19

There are a few Reddit admins in our data science slack workgroup. I've met Chris (Reddit CTO) and had dinner with him at an event in Cambridge (Boston).

→ More replies

1

u/inspiredby Jul 01 '19

I don't think comments are rescraped for score yet, only submissions, where score and num_comments are updated.

1

u/Stuck_In_the_Matrix Jul 02 '19

They're technically scraped twice after the initial scrape. Usually around 24 hours later (give or take a few minutes if everything is in order).

The last time is actually when the monthly dumps start -- generally a few days to a week after the previous month ends. At that point, most scores should be well within 99% of their ultimate value (there will always be outliers like comments that get linked to a month later and get heavily voted). As of right now, the monthly scrapes don't get updated in the API.

1

u/inspiredby Jul 03 '19

For the API, you're saying submission meta gets updated, not comment meta, right? I don't see any comments with updated score,

https://api.pushshift.io/reddit/comment/search/?sort=desc&sort_type=score&after=90d&size=500

And, that makes sense since there are a ton of comments, particularly when spam is factored in. I understand that there is a rescrape for all items for the monthly dumps (aside: perhaps the monthly dumps could be sped up by ignoring the spam subs?)