Hacker News Reader

Comment by frostyj | Hacker News Reader

At the mean time, there are 7 GCP services currently under outage. --A frustrated engineer

I feel better, because I have only one. wonder why gcloud is so bad? not even as good as digital ocean from a user perspective.

Digital ocean has ~300 employees (circa 2017 according to wikipedia). Google has far more than that working on Cloud. Google Cloud is also largely layered on top of their existing internal infrastructure. Google cloud also has more services, more regions, and more zones. My opinion is that there is just way more that can go wrong.

I move people from AWS to GCP for cost, simplicity, and developer experience.

What do you find so bad about GCP?

To be fair, these were not outages, rather degraded services where a feature or two was not working. None of my applications had any errors, nor were my users impacted.

You can find the details here: https://status.cloud.google.com/

You also have to account for the incidents that customers face that don't show up in the status dashboard until a certain influx of support tickets raise the flag.

For example, right now we're facing an issue where one of our GCP HTTPS load balancers is suddenly failing to connect to any backend service for reasons unknown. Not sure if there was any correlation to the eight reported service disruptions on the status dashboard.

Yeah, there are still per customer issues that don't make the status page. GCP does a good job with their RCAs, in-depth with future mitigation strategies. They are also really good about keeping the status updated.

I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.

Sounds like something GCP support would love to help you troubleshoot. The disruptions I'm aware of have no correlation to load balancer traffic paths.

What's the actual reason listed for the 502s in Stackdriver? That's a great first step.

All Stackdriver logs for the load balancer have the following for each request as the status details: failed_to_connect_to_backend

We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag)

We are, however, able to directly connect to the GCE instances within the backend service via external IP.

Have not heard much back from Google support.

Update

This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.

SNI is supported by GCP load balancers, see https://cloud.google.com/load-balancing/docs/ssl-certificate...

Reply on news.ycombinator.com