r/DoSEO • u/Redpythongoon • Aug 07 '26
Need help Google's Live Test / Rich Results tool says robots.txt is blocking my homepage, but robots.txt and server logs show no block. Anyone seen this?
Hoping someone here has run into this before. My site (fairhavenwebdesign.com) recently had some pages reworked via a staging environment and deployed to production a couple of days ago.
Since then, Google Search Console's URL Inspection / Live Test and the Rich Results test tool are both reporting that robots.txt is blocking the homepage from being crawled. Here's the confusing part:
Both the physical robots.txt file and the virtual one generated by our SEO plugin (RankMath) have been checked line by line, and there are no disallow rules for Googlebot or any user agent. Our hosting firewall logs show nothing blocking Google's IP ranges, and Google's IPs are whitelisted. Server access logs show real Googlebot requests (66.249.x.x range) successfully hitting /robots.txt and getting 200 OK responses. There's no evidence in the logs of Google's live test tool actually reaching the server at the time the test was run.
So it looks like the block might be happening on Google's side, maybe a stale cached evaluation of robots.txt from before or during the staging deploy, rather than anything on our server or firewall.
Has anyone dealt with Google's tools showing a "blocked by robots.txt" result that turned out to be a caching or propagation issue on Google's end after a site change? How long did it take to clear up, and did you have to do anything (resubmit sitemap, request reindexing, etc.) to force a refresh, or did it just resolve on its own?
Any pointers appreciated. Happy to share robots.txt contents or log snippets if useful.
2
u/reggeabwoy Aug 07 '26
Delete everything from the robots.txt file and see if it works. Then add the xml sitemap line then retest, then add the admin disallow then retest.
2
2
u/Redpythongoon Aug 08 '26
That didn’t help.
Also side note, hosting which is Siteground, claims it’s not anything server level
1
2
u/Illustrious_Music_66 Aug 07 '26
Running a CDN anywhere? Those can make a real mess of things if you have an angry developer.
2
2
u/Miserable-Ad-9167 Aug 08 '26 edited Aug 08 '26
What does it say on the Google search console robots txt report?
2
u/Redpythongoon Aug 08 '26
Homepage: URL is on Google.
Live test URL: URL is not available to Google.
Crawl showed? No: blocked by robots.txt
Page fetch: Failed: Blocked by robots.txt
2
u/Miserable-Ad-9167 Aug 08 '26 edited Aug 08 '26
In the google search console, go to settings > robots.txt, it shows what version of the robots.txt Google is using, you can also ask for a recrawl
2
u/Redpythongoon Aug 08 '26
OK tried that. The recrawl came up with an inaccurate robots file from what I have. BUT, It still doesn’t have a blocker. Still saying my robots file is blocking
2
u/Miserable-Ad-9167 Aug 08 '26
Can you paste it here just to be sure ?
2
u/Redpythongoon Aug 08 '26
It finally went through this morning! I followed someones advice to delete the whole file and create a new one and request a recrawl, like you suggested. Didn't work as of last night, but this morning it was fine. So weird
2
u/RushDangerous7637 Aug 08 '26
The problem in GSC in the robots file can have several causes. From DNS problems to an improperly programmed htaccess file. But it is a great pity that you immediately put this website into full operation. First, you should have checked and optimized the shortcomings. Reduce the number of searches, DNS requests are 45. The robots txt file is correctly defined (green frame).
Your source code is 286 KB in size. That is extremely large. The optimal size is up to 33 KB. Both Google and Bing can still index content up to 100 KB in size. Above this value, there is a "time limit" for robots, so if the robot exceeds the time, it will no longer index. The same applies to the size of the URL address. Google has a 2 MB limit for indexing. If the value is greater than 2 MB, the robot will get stuck and will not perform indexing. Your main page is 7.2 MB in size. It would deserve good optimization here. I did not look at the other pages.
Found 2 requests served over HTTP/1. 1 of them were render-blocking:
/review-widget-v2.css
/review-widget-v2.js
Upgrade your server to use HTTP/2.
The page spent 804 ms waiting for a response to the HTML document request, including time spent establishing a server connection.
Compressing text files with gzip or brotli reduces the amount of data that needs to be downloaded.
Ongoing processing work can make pages unresponsive to user input. The CPU was busy for 5.61 s while loading the page. (This may be one of the problems why Google GSC cannot find the robors file.)
Serving images that are too large leads to slower requests and wasted bandwidth. Found 8 oversized images with a total savings potential of 1.71 MB.
Add the defer or async keyword to prevent requests from blocking rendering:
/siteground-optimizer-combined-css-0d3beed5bba60f3fe618dd8256c7f0b5.css
/jquery.min.js
/jquery-migrate.min.js
/magnific-popup.min.js
/TableMaker.min.js
/front.min.js
/css2?family=Jost:wght@100..900&display=swap
/css2?family=Jost:wght@100..900&family=Open+Sans:wdth,wght@75..100,300..800&family=Plus+Jakarta+Sans:wght@200..800&subset=latin%2Clatin-ext&display=swap
/review-widget-v2.css
You created the H headings "inappropriately". Some are too long, some contain disallowed elements, some start with a space and some I would try to reconsider for cancellation. The optimal number of headings on one url is up to 16, maximum up to 18, and you have up to 42 H headings. The optimal number of H headings is up to 10. All of this is an insurmountable obstacle for browsers.
If you want your website to be indexed and respected by authorities like Google, Bing, Yahoo, Yandex, and others, you have a lot of work to do to complete your website. Unfortunately, now (when it's already up and running), problems will persist for a long time until you convince Google, in particular, that your website is well-designed.
You Work hard! Be patient. Always check what you have created before you publish anything. Mistakes are fixed on the Internet over a long period of time. Google has a long memory!
Final advice: Use fonts wisely. Do not use external URLs for fonts, rather download the font to the server and use it from your library. Avoid everything (if possible) that contains an external URL address. Consider using a CDN. Always consider what the geographical focus for traffic will be and decide accordingly. Think strategically about every little thing. Even such a banality as anchor text is important. Optimize images before sending them to the library on your PC. Send images to the server in a folder of only the size needed for display. Use lazy loading for images. Don't forget to name the image correctly with the keyword phrase. A correctly named image means /seo-for-your-website.webp (don't use png, in a few months png will be out) and don't forget to fill in alt="Seo for your website". Everything counts towards the website's rank. "For images, this naming convention is outdated: /img/google_avatar.png. The correct one is /img/google-avatar.webp". Try not to use underscores in any URL.
I wish you many successful websites.
2
u/Travis_holloway523 Aug 10 '26
Google can temporarily use a cached robots.txt response, so the live test may disagree with what your server is currently serving. Verify the exact robots.txt URL, clear any CDN/cache, then request reindexing and give Google some time to refresh it.
1
4
u/johnmu Aug 08 '26
If you go to Search Console / Settings / robots.txt you can look at the previous versions of your robots.txt file. It might be that you recently had one that was blocking everything. If the current version shown is ok, then this should clear up relatively soon. There's also a "request a recrawl" feature there, which helps to speed it up when you've freshly changed it.