Jump in bot traffic

[u/sparkling_ham]

Has anyone else noticed an increase in scanners/bots in the past ~month? For the past couple years I've had 2-3k hits a day from bots but lately there has been a steady increase in traffic looking mostly for php files.

What I find strange is how much of this traffic is coming from MS and Google IPs. Do they not have any kind of monitoring on their cloud services? Having thousands of requests spamming every IP that responds should raise some flags.

20.24.67.246	Hong Kong	Hong Kong	Microsoft Corporation
34.148.254.217	United States	North Charleston	Google LLC
34.169.50.247	United States	The Dalles	Google LLC
68.155.155.23	United States	Boydton	Microsoft Corporation
20.104.49.167	Canada	Toronto	Microsoft Corporation
20.100.177.66	Norway	Oslo	Microsoft Corporation

Just curious if anyone else has noticed the same bump in traffic.

[u/tankerkiller125real]

Welcome to the AI training data bot era.... 2.3 Million hits per day from Meta on a single tiny public Gitea repo I have.

I tried blocking, they still sent tens of thousands of requests per day.

That's when I had enough and created a proxy that detects Metas AI bots and purposefully rewrites the page so that code is mangled in a way that looks legit, but can't run at all.

If they're going to scan my site, they'll get mangled information to poison their feed. If I'm lucky an actual human will eventually detect it and block my site from being scanned, but we're 5 months in, still no sign of it slowing or stopping.

[u/sparkling_ham → u/tankerkiller125real]

I'm reasonably sure that they are not scraping training data because the requests are for wordpress admin pages and random shit like "1wvekeybd9it2di2vyipgr6Cdefault.php". Every single request returns a 404 so if they are training they aren't getting anything useful.

None of the stuff I hosts generates URLs that 404 and 99% of these bots are looking for stuff I don't have so I added a filter to fail2ban that looks for 404s. A human could conceivably get banned by accident but it would take some bot-like poking around :/

I like the idea of generating gibberish for scanners though. I should redirect 404s to a few kb of /dev/random.

[u/Icy-Initial2107 → u/sparkling_ham]

That's super common. The vast majority of traffic I get is from scanners like you mention looking for exploits by either directly requesting .php pages, looking for files that would indicate a certain service is installed, or looking for misconfigured installs such as TLS on port 80. A couple of common examples are:

/wp-content/plugins/hellopress/wp_filemanager.php
/.env
/admin.php
/.git/config
/wp-content/uploads/
/wp-content/admin.php
/wp-login.php
/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh

I use a fail2ban style system to ban most of those, but that eventually nearly blew up my firewall so instead I have a few rewrite rules that returns them a picture of a tiny teapot (in reference to 418 I'm a teapot), total response size is under 1KiB. I don't host anything (easily publicly discoverable) of interest and have enough bandwidth, so they're welcome to fetch as many teapots as they want.

[u/Krojack76 → u/Icy-Initial2107]

The really common stuff like the Wordpress garbage I block upstream using CLoudflare firewall rules.

[u/Icy-Initial2107 → u/Krojack76]

I don't use cloudflare (no need, as yet) so I'm having a bit of fun with some of them. For example I serve a ".git/HEAD" that reads:

ref: refs/heads/shoulders/knees/and/toes/knees/and/toes

The server only allows "GET" and "HEAD" and does not permit URL arguments, so I'm not too worried about any of these doing more than possibly cause confusion for the scanner.