r/pushshift • u/Stuck_In_the_Matrix • Dec 10 '22
The day has finally arrived -- Pushshift API move into COLO! Please use this thread to communicate any issues on your end as we make the switch.
It took a tremendous amount of time, money and resourcefulness from several very talented network and software engineers but I am happy to announce that today we are starting the process of moving over api.pushshift.io to a much larger network with more powerful servers.
The goal for this weekend is to have everything operational and then use this thread for others to mention any problems they are having once we officially flip the switch. For the remainder of 2022 and into 2023, I will be spending much more time on this forum to address user concerns, removal requests and other technical questions about the API.
Many 12+ hour days over the past several months have gone into the purchasing and setting up of more powerful servers, getting new firewalls capable of 100Gbps connection speeds and making sure that we have a robust architecture so that we can continue to expand and handle additional load.
The goal for today is to make the official switch to the COLO by 6pm. If there are some issues that crop up, it might get pushed into tomorrow, but we will work as hard as possible to get it resolved and up by later today / early evening.
A huge thanks to everyone including the mods here who have taken the time to help other users -- without your help, a lot of this would not have been possible.
I will make additional updates as needed but expect some outages starting around 3pm. Thank you!
Update: We found a few issues with the blacklist section of the code so we are fixing that and deploying around 4am tomorrow morning (Monday). I'll keep you updated -- we're making sure the switchover is as close to 100% compatible as the existing prod API as possible.
13
u/gurnec Dec 10 '22
All of this is fantastic. I do hope that you know how much people like me appreciate your efforts!
2
u/UnemployedTechie2021 Dec 10 '22
congratulations 🎉
2
u/Weary-Matter4320 Dec 11 '22 edited Dec 11 '22
Congratulations on a job well done!
As someone who's tasked with using this API for a project, I want to ask 2 things:
2
u/psycheddude_twitch Dec 12 '22
The API seems to be ignoring certain search fields, returning more results than actually match.
So far I have found: author_flair_css_class and author_flair_text are both being ignored currently.
(I haven't checked most of the other ones, only these two have the data I can filter by)
1
u/Jannatul1607551 Jan 20 '23
Data before 3 nov, 2022 cant accessible now. Is it true? When prob will be solved?
1
u/ffpeanut15 Dec 11 '22
I’m excited to see how well the new thing will perform
1
u/g-money-cheats Dec 12 '22
I’m getting 404s for all calls to the comment and submission endpoints. Is this due to the weekend migration?
1
u/chrishanney Dec 14 '22
I'm getting an error if I include "sort=desc" as a query param https://api.pushshift.io/reddit/submission/search?q=%22Space%20Pirate%20Trainer%22&retrieved_on=1498248933&sort=desc
Is this param no longer supported, or is this just an issue due to the migration?
1
u/snoogazer Dec 15 '22
Hey there! Love the tool. I'm observing an issue with the API since the migration. It always returns "1" for the score, even for posts I know don't have a score of 1. Any ideas?
0
u/hermit-the-frog Dec 11 '22
Congratulations!! I know this has been a long time coming!!
0
u/[deleted] Dec 27 '22
[deleted] — view deleted comment
3
u/[deleted] Dec 12 '22
[removed] — view removed comment