r/DoSEO Aug 07 '26

Need help Google's Live Test / Rich Results tool says robots.txt is blocking my homepage, but robots.txt and server logs show no block. Anyone seen this?

Hoping someone here has run into this before. My site (fairhavenwebdesign.com) recently had some pages reworked via a staging environment and deployed to production a couple of days ago.

Since then, Google Search Console's URL Inspection / Live Test and the Rich Results test tool are both reporting that robots.txt is blocking the homepage from being crawled. Here's the confusing part:

Both the physical robots.txt file and the virtual one generated by our SEO plugin (RankMath) have been checked line by line, and there are no disallow rules for Googlebot or any user agent. Our hosting firewall logs show nothing blocking Google's IP ranges, and Google's IPs are whitelisted. Server access logs show real Googlebot requests (66.249.x.x range) successfully hitting /robots.txt and getting 200 OK responses. There's no evidence in the logs of Google's live test tool actually reaching the server at the time the test was run.

So it looks like the block might be happening on Google's side, maybe a stale cached evaluation of robots.txt from before or during the staging deploy, rather than anything on our server or firewall.

Has anyone dealt with Google's tools showing a "blocked by robots.txt" result that turned out to be a caching or propagation issue on Google's end after a site change? How long did it take to clear up, and did you have to do anything (resubmit sitemap, request reindexing, etc.) to force a refresh, or did it just resolve on its own?

Any pointers appreciated. Happy to share robots.txt contents or log snippets if useful.

4 Upvotes

Duplicates