Hacker News Reader

Comment by jerendy92 | Hacker News Reader

Time to dump GCP then. It's not even that the fee is that large, but rather that this is once again Google failing on a long term commitment and shafting those on their platform once again. This was one of the benefits that was pushed by their sales team when they called us up to market GCP over AWS and their EKS offering. Doesn't matter that they are price matching, Google's inability to actually commit to long term support, servicing, pricing or features across any of their products is tiresome. Time to move business elsewhere to AWS or Azure. They may be more expensive, but at least we know what we are paying for, and that it's going to stay that way for a significant length of time.

View on HN

If the main value of GKE over DIY is $73, you should totally DIY.

I mostly try not to be too Google-focused here, but I have to say...

I'm pretty proud of GKE, and I think it offers a lot of value other than just being cheap. Managing clusters is not always easy. GKE handles all of that for you - including integrations, qualifications, upgrades, and patching clusters transparently BEFORE public security disclosures happen.

We have a large team of people who deal with making GKE the industry-leading Kubernetes experience that it is. They are on-call and active in every stage of the GKE product lifecycle, adding value that you maybe can't see every day, but I promise you is there. When things go sideways, there isn't a better team on the planet to field the tickets.

I don't understand the anger here - you're literally saying you'd rather pay more for a service of lower quality because... why? Because they will continue to charge you more? Does not compute.

For those people who use a large numbers of small clusters, I understand this may make you reconsider how you operate. As a Kubernetes maintainer, I WANT to say that a smaller number of larger clusters is generally a better answer. I know it's not always true, but I want to help make it true. GKE goes beyond pure k8s here, too. Things like NodePools and sandboxes give you even more robust control

GKE is the best managed Kubernetes you can get. And we're always making it better. Those clusters actually DO have overhead for Google, and as we make GKE better, that overhead tends to go up. As someone NOT involved in this decision, it seems reasonable to me that things which are genuinely valuable have a price.

Also, keep in mind that a single (zonal) cluster is free, which covers a notable fraction of people using GKE.

>If the main value of GKE over DIY is $73, you should totally DIY.

It's not the fee itself, it's the worry that GKE will do what Google Maps did and massively increase fees with very little notice, causing people to scramble to migrate.

Google has a really bad reputation right now when it comes to cancelling projects that people have built their businesses upon, or jacking up fees quickly. The $73 is irrelevant on its own - the issue is (a lack of) customer trust.

Google and Google Cloud are largely different businesses, though I understand it's hard to keep that in mind in the context of things like this.

I encourage everyone to always stay nimble and keep your eyes on portability. I also encourage you to try to assess the REAL costs of doing things yourself. It's rarely as cheap as you think it is.

As a Kubernetes maintainer, I am fanatical about portability.

As part of the GKE team, I think we provide tremendous value that people tend to under-estimate.

NOTE: I was NOT involved in this decision, but I understand it, and I want to help other people understand it.

sorry that you have to work with such low-IQ people that made this decision. Followed your work on k8s, thank you, its better because of people like you and others.

I don't think that is at all a fair characterization, you just don't have the same data available to you.

Thanks for the props. It means a lot to me personally.

Google and Google Cloud are not largely different businesses to the world at large. That may be true internally, but what you do reflects on the other.

Aside from that, Google has a reputation for pulling this shit, and now Google Cloud does too.

The value you provide is irrelevant to how you make people feel with decisions like this.

If you brand it as "Google," you can't expect the positive associations to transfer and the negative associations not to transfer. That's not how it works. You get both or you get neither.

Keep in mind that you aren't selling to me, you are selling to middle management who hears "it's different we promise" and then goes home to have nightmares about their rival manager piling on: "GCP canceled a service from under you? Who could possibly have seen that coming? Oh, the salesman told you it wouldn't happen, my mistake. (Everyone laughs at manager's stupidity.)"

GCP needs to give these guys ammunition. AWS burns goodwill like they've got a city to light: surprise bills, abandoned (but technically not canceled) services, poor performance, sticky abstractions, shameless grand announcements of services that upon further investigation only exist in the sense that you can ask support and wait 5 days (rDNS), etc. Broken software and subtle (or overt!) killer caveats abound. Go with AWS, we scale to the moon! Oh, "the moon" is >5GB of data through our time series ML? Lol no. Hard limit. Oh, we let your buddies exceed that limit and we're advertising that you can exceed the limit? Lol -- not our problem. Yet nobody holds them accountable. Nobody gets fired for choosing AWS, because AWS over-promising and under-delivering is not a meme. It's reality, especially by Google standards, so GCP marketing could make it a meme if they had any sense about them, but as far as I can tell they aren't interested.

If GCP stays the course, the results are 100% predictable and frustrating as hell, because AWS is a real steaming pile and I hate to see them continue to win The Game because Google, of all companies, can't figure out how to advertise and manage their reputation.

> As a Kubernetes maintainer, I WANT to say that a smaller number of larger clusters is generally a better answer.

I'm not trying to nitpick here, but that justification is awful. It goes against reliability engineering on a deeply fundamental level, pretty much guarantees to make already not that reliable things even less reliable. Generally the more isolated entities you have and the smaller they are the less they affect each other and the environment when something bad happens, the faster they can be recovered, the fewer end users they affect, etc. If I remember correctly, this is even how some Kubernetes people justified ideas behind Kubernetes itself that you want to drop now.

That is not absolute truth. If it were you would eschew kubernetes altogether and just use VMs.

Everything is a tradeoff. If you want total isolation, you pay for it. If you don't want to pay for it, you make more value-based tradeoffs.

Concretely, Google runs "a handful" of "pretty reliable" services on a relatively small number of clusters.

I believe everything that you say. The value it provides is very good.

If Google Cloud would have charged 73$ from the start (or after beta), i think there wouldn't be so much anger.

The anger comes from, a product was free and now it is not. A lot of people made architectural choices that depended on the price of 0. (You mentioned these cases in your post).

However, i believe the bigger issue is, that Google Cloud broke essentially a promise.

As a customer I need to be able to trust my cloud provider, because I am literally helpless without it.

Can I trust an entity that breaks promises ? No, I can't. I need to worry. Especially, if I cannot follow the reasoning behind it.

If it is true, that Google's overhead went up, because of improvements, then it would have better to have two kinds of clusters (better and paid, old-school and free). You would have not broken the promise. People can choose on their own pace to upgrade if they need to.

Also keep in mind, that you also carry the Google brand. Hence, if other teams of Google break promises (like f.e. Stadia) this will also reflect on the Google Cloud team. Unless you keep a crystal clear track record, i need to assume it can get worse than what you have done right now.

My conclusion is that, I will design the cloud architecture I am responsible for, such that it has minimal dependencies on Google Cloud specifics.

> However, i believe the bigger issue is, that Google Cloud broke essentially a promise.

Which promise?

> I don't understand the anger here - you're literally saying you'd rather pay more for a service of lower quality because... why? Because they will continue to charge you more? Does not compute.

This response, right here, is everything you need to understand about why Google Cloud is failing to sell to the enterprise market.

The enterprise market only really cares about one thing: rock solid stability. It doesn't care about features, and it doesn't (really) care about price. It wants a product that it can forget is there.

What's really sad is, technically, GKE is that product. It just works. It is solid. You do get to forget that it's there. Until you get a random email telling you that you get to explain to your boss that your bill is going up next month and your project might end up running over budget as a result.

If you can understand why a large segment of the market prefers to pay a higher but stable charge over a lower but undependable charge, then you can understand why Google Cloud is failing at selling to enterprise.

In addition, AWS charges only ever go down. I don’t think I’ve ever seen a price increase.

They also make sure to charge enough for services so it's always profitable to them (smart).

Just a data point:

I'm the CTO at a very small company. All our stuff is running on GKE. Our monthly bill tends is a lot less than $10,000/mo. We're currently in the process of splitting our stack into separate projects and clusters, because co-locating projects in a single cluster has gotten messy. We'll probably end up with 4-5. That will increase our bill by $292/mo, worst case, assuming the first cluster is free. For a company our size, it's not a huge expense. But these things add up.

Since moving from DigitalOcean, our Google Cloud setup has more than doubled our monthly bill. We're paying for more compute, but certainly not twice the amount, as we've only gone from 14-15 nodes to around 20; it's just more expensive across the board, both node cost and ingress/egress. We're even cost-cutting by using self-hosted services instead of Google's; for example, we use Postgres instead of Cloud SQL. I ran the numbers earlier today; the equivalent on Cloud SQL would be 3.4 times more expensive.

In short, Google Cloud is expensive, and it's not like the bill is getting smaller over time.

Developments like these factor in my choice of cloud provider for future projects.

For sure, though I will say if you're trying to cut costs, it's my understanding serverless is quite cheap. So if you can turn some of your services into serverless containers/functions. I'd highly recommend it.

Not sure what size a "very small company" size is, but I'm just curious as to why you chose GKE. I make tech decisions for a (probably) much smaller company, and I found things like App Engine Flexible Environment, Cloud Run and Cloud Functions let me do much of the stuff I can do with k8s but with much, much less complexity (at least on my side of things). The main factor is that I don't have a full-time infrastructure expert, and my experience in the past is that k8s essentially requires that.

Less than 15 employees. Several products, two teams, <= 20 nodes.

We migrated our stuff from DigitalOcean around 2018. At the time, we briefly toyed with the notion of self-hosting Kubernetes on DO, but it's complex to manage, and we don't have any dedicated ops staff. GKE is significantly easier to manage.

At that time we migrated, the things you mentioned weren't available/mature, I think. Even today, I'd choose Kubernetes over a complex mishmash of different systems. I like the unified, extensible ops model. In fact, I'd go so far as to say that I wish all of GCP could be managed as Kubernetes objects.

But DigitalOcean has managed K8s now: https://www.digitalocean.com/products/kubernetes/

DigitalOcean did not have Kubernetes then. Are you suggesting we should spend 6-12 man months migrating back?

No my mistake DOKS came out late 2018.

I had been using it since May 2018 but it didn't come out of early access till December.

How about contracting an ops-oriented person for a month that would do the migration for you? Where do those cost functions intersect?

Would never happen. Just the amount of time needed to dedicate to onboarding a temporary contractor would be really disruptive to the developers, not to mention the disruptive effect of the technical migration — databases to move over, persistent volumes to copy, DNS to repoint, lots of downtime, etc. There's a good reason companies don't switch clouds often.

If it takes more than a month to migrate a 20-node K8s cluster, then that's a red flag. Too much tech debt or a strong vendor lock-in? Either deserves attention.

Doing the migration might take a few late evenings; deploying all our apps and Helm charts to a new cluster takes just a few commands. Learning what needs to be migrated, deciding on how the configuration should look on the destination end of it, and designing a detailed plan and checklist for the whole process is the big task. Adding the onboarding of someone who's never seen the inside of your company and you're talking about a longer, more disruptive process. Tech debt isn't a factor here.

> Learning what needs to be migrated, deciding on how the configuration should look on the destination end of it ...

Sounds like you've identified an area of risk you should address

Re. managing GCP as Kubernetes resources: https://cloud.google.com/config-connector/docs/overview

That's very cool, thanks! Note that this allows selectively creating Kubernetes resources backed by GCP resources. Looks like it will not automatically sync everything that already exists, which seems like a missed opportunity.

We have about 200 long-running pods right now. On Cloud Run, that would cost us more than $12,000 in CPU alone, and that's excluding memory and request costs.

That also excludes stateful apps like Elasticsearch that would not be able to run on Cloud Run. Not sure what Google product is appropriate there.

Indeed, for small apps that average 2 instances or less during the month it's now cheaper to run App Engine Flex than GKE. Each instance costs about US$ 50/month in US central.

For those that are not aware, App Engine Flex runs services based on Docker containers with autoscaling similarly to Kubernetes. It has way less features than GKE, but if you are just running a standard app that only needs to connect to a database that is more than enough.

Bonus points: you can have multiple services, like back-end and front-end and make them available in the same subdomain - to avoid CORS problems. You can also host your front-end for almost nothing with App Engine Standard. It has an awesome CDN built-in if you know how to use it.

So let's say you have a small pile of apps that are all <app_server>, <redis>, <jobqueue/workers>, <db>, <frontend>, but not all the same DB, language, etc. They're all low traffic and you want to automate/simplify and containerize them. On developer machines docker compose works great, but you need to deploy in a cloud provider. What are choice do you have other than K8s?

AWS Lambda with SAM local?

How much would it cost for you to provision and colocate your own hardware, run a k8s cluster, and manage upgrades?

You might not be at the scale where this is feasible yet since that's probably multiple full-time engineers, but eventually the cost functions intersect.

Well, we don't even have a dedicated ops person.

Shouldn't the investment in Kubernetes bring its running cost down instead of increasing it?

> I don't understand the anger here ... Does not compute.

The saying: “the market’s perception is your reality”, is especially apt here. Google’s decision makers tends to forget that in the end they are dealing with human customers not machines. Contrary to the concept of economic rationality, humans are notorious for exhibiting behavior that, to the untrained eye, appear irrational.

A commenter helpfully explained their perception of the new pricing change:

“The anger comes from, a product was free and now it is not. A lot of people made architectural choices that depended on the price of 0. (You mentioned these cases in your post). However, i believe the bigger issue is, that Google Cloud broke essentially a promise.”

IOW, from their perspective, the pricing change was framed as a loss [1] which opened up a host of negative emotions (anger, mistrust, etc) that come with mitigating a loss that is imminent.

Google as an engineering company may look down on fields like psychology or behavioral econs, but if they genuinely want a fighting chance against AWS and Azure, they will need to court sales leaders with a strong humanities tinge, to avoid these kinds of decision-making that achieve the opposite intended effect — eroding people’s trust in GCP.

[1] https://en.wikipedia.org/wiki/Loss_aversion

It's not about whether it's worth it. I have no idea if $73/mo/cluster is more or less than what GKE is worth, but you can't ignore the time-varying aspect of this change. If GCP had started at $100/mo and just announced a price drop to $73/mo, your customers would probably be cheering, even though you'd be charging them the same! They'd be happy about being loyal to you, thankful that your offering gets better over time without asking, etc.

Let me put it another way. Lunch perks at Google are probably worth way more than $73/mo. Googlers would probably not feel too great about having to start paying that amount... Even though it's also worth it, people working hard to prepare food, etc.

It happens time and time again that GCP paints itself in a corner with an unsustainable offering and finds itself in a situation where it has to dent customer trust by hiking prices or retiring products on short notice.

Why?

GCP is in no position to pull this off, competitively. Now doesn't seem like the time to pull unit margins up. Google could do so on ads or with Maps API because it has the best product by far in those areas. But not in Cloud, a market where stability is a feature, and GCP is a late, small player against larger competitors who are giving customers a much more stable product.

It doesn't matter if GKE is the best thing since sliced bread, and keeps getting better — as the CTO of a cloud-based business, you have to decide whether you want to build on it over the long run and take the risk of it becoming super expensive over time; there's not a lot of reassurance that this is even a concern at Google at this point.

> Now doesn't seem like the time to pull unit margins up.

Tbf, you don't have the data to say that. Maybe GCP hit the growth targets they had and are not interested in growing more; maybe they've grown so much that service quality is going down; maybe they want to stall and build up other parts of the offer... there are a lot of reasons why they might want to cash in. Whether they could have foreseen it happening at this specific point in time, and how they managed the communication, is another matter. As others said, Google is saddled by default with a perception of commercial unreliability among the tech-savvy, so any new service they launch should factor that issue in terms of long-term optics.

> Maybe GCP hit the growth targets they had and are not interested in growing more; maybe they've grown so much that service quality is going down; maybe they want to stall and build up other parts of the offer...

No, they are on a death march for market share. https://www.crn.com/news/cloud/google-reportedly-set-ambitio...

> If the main value of GKE over DIY is $73, you should totally DIY.

I'm with you here, and the price feels about right (looking at GCE prices)--it doesn't feel like it's a ripoff or someone's trying to squeeze out more revenue.

> I don't understand the anger here

This is Google being Google. The fee probably makes sense, but as flawed humans, we're subject to the endowment effect and feel like Google's taking something from us. Amazon's all about this customer. This move is all about accurately pricing services, and it's Google putting correctness before emotions.

The problem is you are charging more without providing a clear case that the value is worth it. It looks like a bad deal from an existing customers view point.

73/month is not a lot, but it’s still turning back on the original value proposition. If it’s such a small amount, absorb the cost as cost of doing business, so you can retain customers better and grow new ones.

Thank you for the feedback.

> Google's inability to actually commit to long term support...

This is _exactly_ what Google is doing in this case. We are providing an SLA - a legal agreement of availability and support. These changes introduce a guaranteed availability of the management control plane.

Shouldn't that be opt-in? The management control plane is not something we consider critical to operations. I'd happily accept if it was unavailable for 1 and a half minutes a day versus these additional costs.

That's great feedback. I'll relay that to the product team. IANAL, but I think it would be legally challenging.

IANAL either, but I don't see why it would be? Just have a separate cluster type, e.g SLA Zonal, SLA Regional. The SLA already differentiates the current cluster types. Anthos Clusters are also not subject to any additional fees?

And having it opt-in will save face with those users of GKE where an additional $73/m is significant.

Hard to understand how it would be legally challenging. ISP's do it all the time when differentiating their business plans from residential. Both services run over the same infrastructure and you typically get the same/similar speeds, but a key difference is an SLA with the business plan.

Opt-in for the SLA and additional cluster cost would be fantastic. We run pretty small clusters but don't need any additional SLA's on top of what's already provided. Frankly we could care less about the control plane SLA.

btw. I would prefer to have something like a cost reduce if the cluster runs 24/7. Currently we also do not need that amount of SLA. but we actually have a single cluster and are a really small customer that reall choose GKE, because of no management fee (i.e. nodes are really expensive compared to non cloud providers). But we never used more than one REGIONAL cluster (we also never spun up a new one, we only change workers). And now it will cost us money. What a shame.

P.S.: german sites have the pricing wrong.

Sure, in this case I can see that. I was referring to those four points with respect to Google services in general. I'm sure I don't need to dig up a list of features and services that have been merged, shuttered, price hiked or moved into a different product suite over the years. Admittedly a lot of the issues are with the GSuite side of things, but it's sad to see this coming to GCP as well.

On a hopefully more constructive note, if this is the way it's going to be from now on, I would at least expect to see an exemption on such a management fee/SLA on preemptible nodes - having an SLA and management fee on the cluster whereby nodes can be killed in a 30 second window without prior warning seems to be a little more than pointless.

Even if your worker nodes are pre-emptible, the master nodes are not. The management fee covers running those master nodes and many other core GCP integrations (like Stackdriver logging and other advanced functionality). Billing is computed on a per-second basis for each cluster. The total amount will be rounded to the nearest penny at the end of the month.

He means that there was a sales pitch from all gcp sales guys to not charge for that. 99.95% is not enough IMO to charge 73$/mo.

As someone else noted, it breaks a lot of recommended architectures where you would have auto provisioning and a lot of clusters to separate concerns and keep costs down.

Finally, the pricing changes are starting to look like a pattern, every time Google deems the usage of a product is good enough, they will increase the price.

They are the Ryanair of the cloud.

Edit 1: moreover, it will increase the cost of composer, and on top of that, the recommended pattern where composer is paired with a kubernetes cluster for executing the workloads

> They are the Ryanair of the cloud.

Isn't Ryanair literally the Ryanair of the cloud(s)?

EKS only gives you 99.9% uptime, and I'm uncertain as to whether you could achieve more than 99.9% uptime on your own by DIYing your cluster in a public cloud provider without doing multi-region.

To put that in perspective, three 9’s allows you about 9 hours of downtime a year, which will certainly require multi-region and a dedicated ops team.

Two and a half 9’s is a whole different story. We achieved about 20 hours of downtime last year even without HA on k8s bare metal in Alibaba cloud. But I’m uncertain whether that’s a feat we can repeat this year.

> Finally, the pricing changes are starting to look like a pattern, every time Google deems the usage of a product is good enough, they will increase the price.

To be fair this is hardly new and by no means limited to Google. Any number of SaaS startups that have survived to at least moderate success have done similar things.

Look at UserVoice as an example: started out with a free tier plus some reasonable paid tiers with transparent pricing, then a year or two back killed the free tier and moved to a non-transparent "enterprise" pricing model with absolutely exhorbitant fees.

Plenty of other companies offer free to build their userbase and reach, then either water down the free tier, or remove it entirely. It's practically the SV modus operandi for the last decade.

>Any number of SaaS startups that have survived to at least moderate success have done similar things.

Google is not a startup, it is one of the largest companies on the planet.

I don't see how that's relevant: my point is that it's a tactic employed by a wide range of businesses including but not limited to startups and we shouldn't be surprised to see it here. It's not a pattern that has suddenly emerged.

> We are providing an SLA - a legal agreement of availability and support.

Do I still have to pay the bill first, fill out forms, get account managers involved, at some point receive a partial credit, and repeat this until the delta between what I was expected as the SLA credit and what I got as the SLA credit is less than the cost of the time to fight for another cycle?

What? A 73$ price difference to AWS was their main selling point?

It depends entirely on how a firm has their infrastructure set up - if you have small cluster(s) per client for isolation/compliance purposes, you end up with, for example with 250 clients, each one using, say, 1000 billable hours/month:

250 * 0.1 * 1000 = $25,000/month.

This is quite the price hike for something that was a) free until now. b) Not a service that warrants such a fee given that it uses existing (GCE) resources and can frankly be done manually by one of the DevOps engineers for a few hours/month and some scripting. It's just a charge for convenience it seems.

You keep noting how "easy" it is to provision and manage a Kubernetes cluster. From experience, properly securing and maintaining a Kubernetes cluster is a multi-person full-time job.

We already do provision clusters, using the aforementioned tools. There is some setup involved, but once done, provisioning and upgrade is relatively simple. Indeed, we used to exclusively provision and upgrade via Terraform/Ansible. When we started using GCP, any data that could be stored by a US company without causing compliance issues was offloaded to GCP over other providers due to the auto provisioning/management at no cost.

If you guys find it hard to maintain/upgrade clusters, that's your business. All I am saying is that as a company giving you business, with this change, you are now no longer the cheapest, most reliable or most convenient. As a result, we will be moving to provision instances with other providers from now on.

No, the $73 was a selling point of GCP. You could easily spin up different clusters at no additional costs (at least for the cluster itself).

Never trust free without an escape hatch.

Reply on news.ycombinator.com