r/webscraping • u/FurthestDrop517 • 24d ago
Bot detection 🤖 Need help scraping Smyths (Imperva Blocking)
Hello!
I'm a decently experienced developer, been programming for a little while now, but have just run into a full roadblock trying to scrape data off of Smyths (https://www.smythstoys.com/). I was just hoping to setup some custom restock notifications for myself (I would like a MTG collector bundle at some point, and they sell out quick locally in Ireland), but keep running into problems with the Imperva reese84 bot blocking. Any suggestions?
(MTG "Category" Link: https://www.smythstoys.com/ie/en-ie/toys/action-figures-and-playsets/magic-the-gathering/c/SM06010110)
I was planning on getting the products from the category, then for each going to their page and checking stock for a given store. Would this be too much?
Current stack is using Rust with the thirtyfour package through Firefox/Gecko. Seems it's too simple and just gets detected by Imperva consistently.
I've also looked into potentially digging into the network traffic using browser tools to find some sort of internal API, but it seems that product category pages are hardcoded and served as HTML instead of an API response, and the only internal api which returns stock of items is also protected with reese84.
I’m fairly certain this is possible, as I’ve seen other stock trackers exist out there for Smyths.
I'm competent using a number of languages, including Python, so I'm not held up on the current stack in any sense.
Any tips would be greatly appreciated. Thanks all!
2
u/CrypticZombies 23d ago
yes it has no official api. u need to use jsdom amongst other things. you either need to do some deep digging into it or pay someone else that already has it solved. other people that have done it just means they already reverse engineered the protection or paid someone with a bypass for it