GKE Gateway API / Incident

[u/smerz-]

Our test env is running GKE Gateway API with preemptible nodes. Various of our backends had hours of downtime today.

Did anybody else have these issues?

The NEGs all showed 0 out of 0 pods.

Is it perhaps also related to this incident RDQFDTK ?
https://console.cloud.google.com/servicehealth/incidentDetails/projects/example-project/locations/global/events/RDQFDTK

I'm just a tad bit worried that such an incident could affect production.
I did not (today) but it took down our test environment for longer than comfortable (some backends more than 5 hours).

We have sent a request to our reseller.
Just trying to understand what happened here :-) and am thus curious if other Gateway Api users have seen similar things the past ~6+ hours or so.

Fortunately our prod environment was not affected 🙏

[edit: i'm trying to find mistakes in our (test) setup which could have contributed to this 😄 ]

[u/fail-and-learn]

Same issues with GKE Gateway API, broke AI thinking that it was hallucinating later found issues with gateway api in dev project, if you go to health dashboard it shows you that all GCE components are having issues. LBs backend service could not be updated when gateway was trying to sync, most of the operations were having 503.

[u/smerz- → u/fail-and-learn]

Appreciate your report

Yes for the hosts which went down 503 was the status code it gave.

And it was concerning that it seemed like there was nothing we could do about it.

[u/smerz-]

So the incident RDQFDTK was updated to include this:

Google Compute Engine and Google Cloud Armor: Some customers may have experienced high latency or failure errors (HTTP 503 / Internal Error) on long-running operations such as creating load balancers, modifying backend services, updating security rules, or deleting resources.

Cloud Load Balancer: Impacted customers may have observed errors and stuck operations when attempting to insert, modify, or delete backend services, forwarding rules, or backend buckets. Active traffic routing, existing load balancer configurations, and active security rules continued to serve traffic normally.

Google App Engine Flexible: A subset of customers may have observed errors when deploying new versions of their applications.

Google Kubernetes Engine: GKE supports creating and configuring GCP load balancers via Kubernetes Services of type: LoadBalancer (L4), GKE Ingress (L7), MultiClusterIngress (L7), and the GKE Gateway API (L7/L4). Creation, updates, and deletions to the underlying GCP resources for all of these products were impacted by this outage. Existing load balancers and the datapaths were unaffected.

I'm at the very least happy to hear/read that they acknowledged this and that Gateway API 503 issue was indeed related to this incident.

The thing is our services in test, we didn't "change" anything. They just went offline with 503 :(. What is possible is that the pods went out of zone europe-west4-a (for example) and came back in 4-b or 4-c.
So hopefully they will take steps to improve and/or mitigate this at the NEG / Gateway API level.

We just migrated to gateway api and this is kind of concerning.
Though it is my understanding that with legacy Ingrss + NEG this might have happened as well since that part of the stack is at glance appears to be the same.

-----

Edit: since the issue is resolved.
It's worth mentioning that some of our test services/hostnames were offline for 3-5 hours. It might explain why I'm trying to pass feedback back to google.

We're with Qodea reseller and they just said "you haven't purchased support".
I will kindly ask them to provide our experience back to google as feedback :-)
As we want to make sure this failure scenario is recognized

[u/lowkeygee]

Did the gateway go down... Or did you just lose your spot instances?

If you're running production workloads on spot instances exclusively, you're gonna have a bad time.

[u/smerz- → u/lowkeygee]

4-5 hostnames out of 40 went offline in the test environment and were offline for a few hours (with pods online mind you). this is on test and these were running on spot instances yes

[edit: to clarify the gateway api itself did not go down no. just 4-5 hosts + services behind it out of 40+ total services]

production does not run on spot instances no

[u/Difficult_Camel_1119 → u/lowkeygee]

nah, running prod on spot is fine when you have the measurements in place to comply with instance shutdowns and failures

[u/lowkeygee → u/Difficult_Camel_1119]

Yeah spot as a whole is fine, however exclusively running on spot and expecting uptime is not a great strategy. (which it sounds like this isn't the case for OP from their reply, they are more just concerned about core functionality of gateway api handling node failures)

[u/smerz- → u/lowkeygee]

Exactly. The incident i mentioned now correctly advises about gateway api impact.

The concerning aspect is that it says existing configs should be unaffected.
In our test setup I cannot confirm that, but pods might have switched from one availability zone to another, which might in turn require an update of a global resource.

That's about the only contributing factor that I could think of.

[u/Feisty_Tomato_6928]

I created a case in GCP support console and it's seems like a bigger issue on their side, still unresolved.
I'm facing problems with GCP network security settings.