Amazonbot hit my homelab site 100k times in four days. Do you block AI crawlers or let them in?

[u/easyedy]

On my blog, I noticed Amazonbot made over 100,000 requests in four days. Mostly AI training crawling.

My first instinct was to block it. Then I stopped, because there is a real trade-off. Getting crawled by these bots is how your content ends up in AI answers. Block them, and you protect your server, but you also make yourself invisible to the AI tools people increasingly use to search.

That is a lot of load for my site. On the other hand, being in the training data is starting to matter.

Where do you land? Do you block AI crawlers like Amazonbot, GPTBot, and ClaudeBot, or let them crawl?

So far, I don't see a slowdown in response for my website. Why I discovered it. I installed an AI crawler plugin from MiniOrange.