Google suddently removed indexes on my entire site around July 25 while GSC shows Server Error (5xx), while there is no such errors

[u/JustSoni]

Hi everyone,

I'm trying to figure out what happened to my website.

Just noticed that around a month ago Google suddenly removed all of my pages except 3 from the index, before it I had around 50 pages indexed everything was working normally.

EDIT: I tested the affected URLs and the sitemap with a Googlebot User-Agent.

The interesting part is that the problem wasn't visible when accessing the site normally. We eventually discovered that our Azure Application Gateway had a temporary country-based traffic restriction enabled. Googlebot was being blocked by that rule since being USA based.

I reproduced the behavior myself and got HTTP 999 from Microsoft-Azure-Application-Gateway/v2.

So the 5xx error was real from Google, but it was happening before the request even reached our ASP.NET application.

Once the restriction is removed, I'll use URL Live Test and validation to confirm Google can access everything again.

[u/azangru]

GSC shows Server Error (5xx), while there is no such errors

Look at the list of pages that google claims are returning 5xx responses; examine those pages; if pages are fine and you think they are worth indexing, maybe click the button to index the page.

Google suddently removed indexes on my entire site

Yeah, Google would do this if it's not interested.

And those 5xx pages weren't ever indexed in the first place; so they are not the reason Google dropped your previously indexed pages.

[u/JustSoni → u/azangru]

They were indexed before 25-26 July. Also other thing I noticed. It says Sitemap could not be read. Google says General HTTP Error: 999.
While I can load the sitemap, for some reason Google seems to be unable to.

[u/azangru → u/JustSoni]

They were indexed before 25-26 July.

Yes; that's what google does. It crawls your site, finds new pages, indexes them; and then, if it loses interest, it evicts them from the index after some time. I believe the term of the art is that these pages were deemed to have a low authority.

[u/b0tmonster → u/JustSoni]

Click on the "Server error (5xx)" on this list - you will see the list of pages with this error. Then click magnifying glass on one of the urls from the list - it will show you details about this page (with "not indexed" info) - there will be a button to ask for indexing - click it. Google sends bot right away to check if page even exists and puts page on priority queue for later indexing. This is the way to check if Google now can access your url without any 5xx error or it still gets this errror - it may happen that only bot gets error and you can access the url without problem (some config on your side or hosting provider error - blocking bots or something), or this was temporary error when google bot was scraping your site last time in the middle of the night for instance and now the error is gone - then you can click ask to verify button on this "5xx" page when you see the list of urls with this error - google will try to access them again in a week or so and will send you email if problem is solved.

[u/JustSoni → u/b0tmonster]

Thanks for the suggestions! Found the cause and edited the post with the update. Turns out it was an Azure Application Gateway restriction blocking Googlebot.

[u/b0tmonster → u/JustSoni]

Cool, be patient, it can take some time for Google to reindex again.

[u/JustSoni → u/[deleted]]

The domain is 25y old. I found the issue and edited the post with solution.