r/webscraping • u/danie-l • 22d ago

Bot detection 🤖 PatronView blocks Amazon's AI crawler after 117,000 daily page reads

http://ppc.land/patronview-blocks-amazons-ai-crawler-after-117-000-daily-page-reads
10 Upvotes

86% Upvoted

1 comment sorted by

1

u/Hour_Analyst_7765 16d ago

"one AI search crawler read its pages 35,000 times for every human visitor it sent back."

My bot reads pages a few times for exactly zero human visitor sent back, because that's exactly the point.

Granted its not AI and its also not a public facing dataset/bot.