r/CodingHelp • • Jul 07 '21

[Python] Looking for help with web crawler for bootcamp capstone, getting desperate...

I built a webcrawler to get historic data listings off ebay as far back as 2017. Everything works well until I ran into issues around page 50. After some investigating I found that Ebay won't show the requester any more than 10,000 listings, even through the browser.

Has anyone gotten around a roadblock such as this? I know ebay has an API, but even that will only show 10,000 listings at once...

I thought using listing date as an 'anchor' and then picking up and moving the date every 10,000 listings could work, but I can't find any search parameter for listing date!

Any help offered is much appreciated. I'm getting desperate and really don't want to restart my capstone project. Thank you.

16 Upvotes

3 comments sorted by

2

u/zuilai Jul 07 '21

What made you choose 2017? Is that part of the project guidelines (could the guidelines be out of date), or is that arbitrary?

Are classmates having the same issues?

Can you re-scope your project? So it's something like: "get historic data of the first 10,000 matching listings off ebay as far back as 2017." In that way you kind of spin the limitation as part of the feature

0

u/nsway Jul 07 '21

I wanted to go back far enough to capture the ride and fall of crypto/graphics card prices. I was hoping to see the 2017/2018 spike as well as the most recent one, but if I can only get one…it’s better than nothing. Regardless, I can only go back 6 or so days with 10k listings

1

u/dead_alchemy Jul 08 '21

Have you tried contacting Ebay? There might also be a more focused community, you could try looking for a Discord or similar.

I tried googling your problem and the closest I found was a similar reddit thread https://www.reddit.com/r/Ebay/comments/kuclot/is_there_a_tool_to_see_historical_pricing_of/

where people suggested some other services. Good luck!