Hacker News Reader

Comment by boulos | Hacker News Reader

Disclosure: I work for Google Cloud (and on preemptible VMs).

There isn’t anything about preemptible to “cause networking issues”. We may have had a general networking outage, if that’s what you experienced, but we don’t additionally make networking worse for preemptible. You’re likely to get shot, but we don’t adjust throttles or anything.

View on HN

Thanks for letting me know. Overall, we use them for high availability and non mission critical tasks anyways but now that you say this, it gives me more confidence in recommending it. It was strange, it only affected one of the two GKE cluster (both had preemptive nodes). I tried debugging and found that it wasn’t able to even ping any external ips (which is why I thought maybe preemtible nodes also might have networking issues). There weren’t any notices about general networking outages within the last +\- 48 hours (I subscribe to the RSS GCP outages via slack). I ended up manually rebuilding the nodes (non preemtible nodes temp) and it went away. As soon as healthchecks were failing, I created a debug container and kubectl exec -it bashed in and saw that it wasn’t able to connect to the internet. It could ping/telnet internal IP databases just fine. It does warn me that it’s in beta so, again maybe you’re working through some things but thought you should know in case other people have experienced this.

Did your instances have external IPs? We had connectivity issues when using NAT due to it dropping SYN packets. We just gave everything that needed to talk to external IPs their own external IPs.

Glad you found a workaround, but please file a support ticket for this. If you don’t have support: (a) consider it and/or (b) send me an email with your info (contact in profile) and I’ll file a bug.

We actually did have a dedicated Googler who oversaw our move from AWS to GCP. I asked him about it but never got a reply.

Anyway, yes. It caused us problems. Attempts to connect to machines outside of GCP would fail repeatedly due to the NA dropping the SYN. This would mean our connect() call would timeout repeatedly. There was an ICMP response I could see with tcpdump:

ICMP host aaa.bbb.ccc.ddd unreachable - admin prohibited filter, length 68

Reply on news.ycombinator.com