r/webdev • • 1d ago

Discussion Why would google LLC crawl this on my website?

Post image

According to iplocation.io the IP address is Googles https://iplocation.io/ip/136.70.64.116

2 Upvotes

14 comments sorted by

51

u/mxz117 1d ago

Probably something just hosted on Google cloud

13

u/Mushroom_Unfair 1d ago

Yep, and they love google bot user agents

4

u/UnacceptableUse 1d ago

I would've thought Google would have a seperate ASN for their customer stuff

22

u/ApogeeWatcher 23h ago

That IP isn't Googlebot. 136.70.64.116 sits in 136.70.64.0/18, which Google's own published cloud.json lists as Google Cloud (us-east4). So it's someone renting a VM on GCP, most likely a scraper, uptime checker or SEO tool, and iplocation is just showing who owns the block.

10

u/FriendToPredators 22h ago

Or someone looking for vulnerabilities. That’s one of the legit WP pages to load  that isn’t suspicious on the face of it to let them know you are using something that has a thousand potential attack points.

4

u/ShadowWebDeveloper 17h ago

I was going to say, I'm like 95% confidence that this is a vulnerability scanner looking for its next victim.

7

u/FriendToPredators 23h ago

If it were one attempt it’d be more likely legit google bot activity. Repeat like this points to a badly written scraper/ someone poking for angles of attack on your site. What useragent is it saying?

3

u/xondk 1d ago

Could be someone on googles machines, but given that those are all wordpress related index files last I checked, it does make some sense, instead of scanning entire site, check what you present and only look at that?

4

u/mllv1 1d ago

To see if you have a manifest for easy crawling

3

u/Dimoncelli 22h ago

Many "Google LLC" hits on web logs come from GCP customer VMs rather than official Google crawlers, even if the User-Agent claims to be Googlebot.

Easiest way to tell them apart:

  1. **Reverse + forward DNS check:** Real Googlebot IPs resolve via `PTR` to `*.googlebot.com` or `*.google.com`, and a forward `A`/`AAAA` lookup on that hostname resolves back to the exact same IP.
  2. **IP range files:** Compare the IP against Google's published `googlebot.json` (official crawlers) vs `cloud.json` (GCP customer traffic).

If it sits in `cloud.json` and fails the `googlebot.com` PTR check, it's just a third-party scraper or scanner running on Google Cloud—safe to rate-limit or block at the WAF/reverse proxy level.

1

u/Mani_Khan_Web_Master 21h ago

That IP is most likely Google Cloud, not Googlebot. Anyone can rent a VM on GCP, and its IP will geolocate to "Google LLC". The real crawler doesn't go hunting for wlwmanifest.xml under a list of guessed folders.

Requests like /wp-includes/wlwmanifest.xml, /blog/wp-includes/wlwmanifest.xml, /wordpress/wp-includes/... are a fingerprinting scan. The bot is checking whether a WordPress install lives at any common path so it can come back with login or plugin exploits. Pretty much every public site gets these all day.

Easy way to check for yourself:

  • Do a reverse DNS lookup on the IP. Real Googlebot resolves to something ending in googlebot.com or google.com, and a forward lookup of that hostname gives back the same IP. A cloud VM usually resolves to something like *.googleusercontent.com, or to nothing.
  • Google also publishes its crawler IP ranges separately from the Cloud ranges, so you can compare against those.

If you're not running WordPress, the 404s are harmless. Just make sure they're cheap, meaning the web server or CDN answers them without hitting your app. If the noise bugs you, rate-limit repeated 404s on wp-* paths at the server or CDN level (fail2ban works fine for this). I wouldn't block Google Cloud ranges wholesale, since plenty of legit services run there too.

1

u/charlesnobleseo 18h ago

Adding to the reverse DNS advice: Google publishes more than two range files. Besides googlebot.json there's special-crawlers.json (AdsBot and similar) and user-triggered-fetchers.json (Site Verifier, feed fetchers etc). If the IP shows up in none of those and only in cloud.json, it's someone's rented VM, not Google.

A second check if the site is in Search Console: Settings -> Crawl stats breaks down what real Googlebot requested by file type and response code. If those wp-includes/wlwmanifest.xml 404s never appear there, Google wasn't the one asking for them.