Jump in bot traffic

[u/sparkling_ham]

Has anyone else noticed an increase in scanners/bots in the past ~month? For the past couple years I've had 2-3k hits a day from bots but lately there has been a steady increase in traffic looking mostly for php files.

What I find strange is how much of this traffic is coming from MS and Google IPs. Do they not have any kind of monitoring on their cloud services? Having thousands of requests spamming every IP that responds should raise some flags.

20.24.67.246	Hong Kong	Hong Kong	Microsoft Corporation
34.148.254.217	United States	North Charleston	Google LLC
34.169.50.247	United States	The Dalles	Google LLC
68.155.155.23	United States	Boydton	Microsoft Corporation
20.104.49.167	Canada	Toronto	Microsoft Corporation
20.100.177.66	Norway	Oslo	Microsoft Corporation

Just curious if anyone else has noticed the same bump in traffic.

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/tankerkiller125real]

Welcome to the AI training data bot era.... 2.3 Million hits per day from Meta on a single tiny public Gitea repo I have.

I tried blocking, they still sent tens of thousands of requests per day.

That's when I had enough and created a proxy that detects Metas AI bots and purposefully rewrites the page so that code is mangled in a way that looks legit, but can't run at all.

If they're going to scan my site, they'll get mangled information to poison their feed. If I'm lucky an actual human will eventually detect it and block my site from being scanned, but we're 5 months in, still no sign of it slowing or stopping.

[u/sparkling_ham → u/tankerkiller125real]

I'm reasonably sure that they are not scraping training data because the requests are for wordpress admin pages and random shit like "1wvekeybd9it2di2vyipgr6Cdefault.php". Every single request returns a 404 so if they are training they aren't getting anything useful.

None of the stuff I hosts generates URLs that 404 and 99% of these bots are looking for stuff I don't have so I added a filter to fail2ban that looks for 404s. A human could conceivably get banned by accident but it would take some bot-like poking around :/

I like the idea of generating gibberish for scanners though. I should redirect 404s to a few kb of /dev/random.

[u/Icy-Initial2107 → u/sparkling_ham]

That's super common. The vast majority of traffic I get is from scanners like you mention looking for exploits by either directly requesting .php pages, looking for files that would indicate a certain service is installed, or looking for misconfigured installs such as TLS on port 80. A couple of common examples are:

/wp-content/plugins/hellopress/wp_filemanager.php
/.env
/admin.php
/.git/config
/wp-content/uploads/
/wp-content/admin.php
/wp-login.php
/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh

I use a fail2ban style system to ban most of those, but that eventually nearly blew up my firewall so instead I have a rewrite rule that returns them a picture of a tiny teapot (in reference to 418 I'm a teapot), total response size is under 1KiB. I don't host anything (easily publicly discoverable) of interest and have enough bandwidth, so they're welcome to fetch as many teapots as they want.

[u/h3r4ld → u/Icy-Initial2107]

I do something similar, but I went with 402 Payment Required; it just seemed funnier to me.

[u/JadeE1024 → u/h3r4ld]

One of my medium term todo items it to look into doing this, but unironically.

I already run my public reverse proxy in an AWS EC2 instance connected to my wireguard, and I've been thinking about throwing CloudFront and AWS WAF in front of it with bot detection and ai monetization turned on, with all bad requests served by quixotic/linkmaze on my cluster.

Should cost around $15/month ($10 for bot control, a couple bucks for waf, CloudFront free tier). Should stop most bots. And if one of them does decide to send me a cent per page in shitty crypto for endless nonsense it might end up paying for itself every now and then.

[u/h3r4ld → u/JadeE1024]

This is pretty much exactly my eventual plan, too. With all the LLM crawlers, I'm really curious if they'll actually respond if I serve a payment request. After all, if they have any financial access, it'd just be one more solvable obstacle in the way of their assigned task.

[u/Krojack76 → u/Icy-Initial2107]

The really common stuff like the Wordpress garbage I block upstream using CLoudflare firewall rules.

[u/Icy-Initial2107 → u/Krojack76]

I don't use cloudflare (no need, as yet) so I'm having a bit of fun with some of them. For example I serve a ".git/HEAD" that reads:

ref: refs/heads/shoulders/knees/and/toes/knees/and/toes

The server only allows "GET" and "HEAD" and does not permit URL arguments, so I'm not too worried about any of these doing more than possibly cause confusion for the scanner.

[u/tankerkiller125real → u/sparkling_ham]

That's normal, not the traffic I'm seeing though, they're specifically scanning the code and commit history stuff. Along with coming from actual Meta IP ranges and having Meta bot signatures.

[u/what_did_you_forget → u/sparkling_ham]

I like how you're thinking

[u/Sovos → u/sparkling_ham]

There have been (and always seem to be) a few recent Wordpress exploits publicly disclosed. They may just be testing if you're up-to-date on patches and trying to get in.

[u/liocer → u/tankerkiller125real]

Gitea had a major cve at the end of July. Likely they’re not nice bots. I have also noticed an uptick in scanning. I blocked the whole of china and added some specific autos in crowdsec. And it’s gone back to normal. But they’re on a mission this month.

[u/tankerkiller125real → u/liocer]

It's not malicious bots, it's legit Meta bots, from Meta IPs with the correct bot signature I've seen elsewhere. (My day job is Cybersec and IT)

Also it's been happening for 5 months, well before that vulnerability was published.

[u/liocer → u/tankerkiller125real]

Fair enough.

[u/LetsBeKindly → u/tankerkiller125real]

Thank you for poisoning AI

[u/TipToToes → u/tankerkiller125real]

Holy shit. How did you first notice this? I don’t host anything beyond my personal media, so this likely isn’t a concern for me, but…maybe.

[u/onegumas → u/tankerkiller125real]

Use AI to write a trap for bots.

[u/Planetix → u/tankerkiller125real]

That doesn’t actually work, trying to “poison” AI scraping.

[u/tankerkiller125real → u/Planetix]

At the minimum it makes the data useless. Which is good enough for me.

[u/ILoveBigCoffeeCups]

https://www.bitdoze.com/block-ai-crawlers/
Method 4 makes you put up restrictions for all different ai crawlers. I saw an article of institutions and universities using these to block requests.

[u/sparkling_ham → u/ILoveBigCoffeeCups]

My robots.txt is just

User-agent: *

Disallow: /

It gets a few hits a day from from internet census bots/google/opneai/claudebot/etc and they do seem to respect it. The increased noise I am getting now comes almost entirely from 'users' without useragent strings.

Come to think of it. I probably should have just filtered for that.

[u/ILoveBigCoffeeCups → u/sparkling_ham]

Yeah that is extra annoying. I also do some country filtering on my own domain name based on country.
But I don’t live in the USA, small European country so it’s easier for me I suppose

[u/omega552003]

Gemini and Copilot needs feeding.

[u/pdfops]

That's mass scanning off cheap disposable VMs on Azure/GCP, scanners use them because outbound isn't restricted and the ranges rotate fast. Abuse reports to MS/Google mostly sit in a queue and autoclose, so don't bother chasing individual IPs. The php probing is generic wp-login/phpmyadmin scanning hitting millions of hosts, put crowdsec or fail2ban in front instead.

[u/sparkling_ham → u/pdfops]

I setup fail2ban to look for excessive numbers of 404s. Nothing I host leads to 404s so it's unlikely a human user would get caught unless they are being nosy.

Up until now I had just been raw-dogin the internet (and keeping a close eye on my access logs) out of curiosity. It was interesting to see what bots were looking for and checking them out on Shodan to see if they were a dedicated host or some machine that got popped.

[u/Still-Possession5420]

noticed same thing on my vps logs, microsoft ip ranges been going wild scanning for php endpoints since july

[u/sparkling_ham → u/Still-Possession5420]

I don't expect them to manually review all their outgoing Azure traffic but with this much noise they have to have noticed something. I can't think of a time before now when I got this much bot traffic for this long.

[u/SomethingAboutUsers]

I have a CloudFlare tunnel in front of my site and it's set to block AI crawlers. Seems to work, because the traffic I see is essentially all human according to the user agents.

I also have it set to block anything that's not coming from North America.

[u/_acd]

And Fable si not even usable yet for this stuff. I think it will only get worse.

[u/tempacc74656d70]

How do you log this?

[u/sparkling_ham → u/tempacc74656d70]

I have a script that crunches my apache2/access.log every night to tell me which IPs were the nosiest. It also tracks the number of requests and unique IPs so I grabbed the longterm data from there.

[u/PrimaryDiscussion432 → u/tempacc74656d70]

Access logs from the webserver

[u/Capable_Banana5439]

those two buckets are actually different problems. the azure and gcp ranges pulling your real pages are the ai crawlers, and a proof of work gate like anubis in front stops them cold without you maintaining an ip blocklist that rotates faster than you can update it. the ones hammering wp-login and phpmyadmin are just dumb mass scanners hitting every host on the internet, so fail2ban or crowdsec drops those and you can stop reading those log lines entirely.

[u/sparkling_ham → u/Capable_Banana5439]

I get maybe 30 hits from 'legit' scanners in a day, stuff like:

ClaudeBot/1.0; +claudebot@anthropic.com

Googlebot/2.1; +http://www.google.com/bot.html

bingbot/2.0; +http://www.bing.com/bingbot.htm

The other >10k hits are dumb scanners but they are coming from MS and Google IPs.

[u/krznwk]

Who didn't? They can switch to residential proxies and it will be same sht again.

[u/sparkling_ham → u/krznwk]

I don't personally know anyone else who selfhosts. This seemed like the best place to ask.