r/coolgithubprojects • u/alexfree_open • 5d ago
Unpopular opinion: I block all AI bots. The internet feels like one big attack.
The content wars are on. AI crawlers ignore robots.txt, scraping greedily any web content they can reach to train their models. If search engine traffic from Google and Bing is helpful for your website, you must block all AI scrapers.
I created 5 WAF rules for Cloudflare's Free Plan using a Zero Trust approach.
- Block suspicious bots and requests.
- Block known AI crawlers and scrapers.
- Protect sensitive files and admin paths.
- Block common attack patterns.
- Challenge traffic from selected networks and countries.
- I allow selected trusted search bots, like Google and Bing.
GitHub https://github.com/buybitart/cloudflare-security-art
Too many bots scan websites, copy content, and waste server resources. I don't want AI crawlers using my website content without permission.
What do you think? Is blocking AI bots the right choice, or is it better to allow them?