For example, right now we're facing an issue where one of our GCP HTTPS load balancers is suddenly failing to connect to any backend service for reasons unknown. Not sure if there was any correlation to the eight reported service disruptions on the status dashboard.
I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.
What's the actual reason listed for the 502s in Stackdriver? That's a great first step.
We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag)
We are, however, able to directly connect to the GCE instances within the backend service via external IP.
Have not heard much back from Google support.
This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.