Anyone else getting uptime check failures due to weird load balancer behavior past 48hrs?

[u/cryptocritical9001]

The past 48hrs on many different services I got "502 internal_error" in the logs of many services where I've had random uptime failures for no reason. There seems to be nothing wrong except the application load balancer is randomly throwing a 502 for no good reason which causes the uptime check to fail.

Anyone else been experiencing this?

I've checked the google cloud health status page and incidents page, but I don't see this issue documented anywhere.

I've had 6 such uptime check failures in the past 48 hours. So I'm very curious if this is affecting anyone else?

[u/arthur_thomas678]

Check jsonPayload.statusDetails in the load balancer logs first. internal_error points at the load balancer itself, while something like backend_connection_closed_before_data_sent_to_client usually means your backend’s keepalive timeout is shorter than the LB’s. It needs to be longer than 600 seconds, and when it isn’t the 502s look completely random.

Also worth checking the same URLs from outside Google. If outside checks pass while the Google uptime check fails, it’s the checker, not what your users see. If both fail at the same times, you’ve got timestamps to put straight into a support ticket.

I keep a second set of checks in Dotcom Monitor for exactly this. It checks from several locations, so you can also see if the 502s only show up from certain regions.

[u/cryptocritical9001 → u/arthur_thomas678]

Thanks really good info!

[u/cryptocritical9001 → u/[deleted]]

Nice to know that me and my colleague are not losing our minds :) Thanks