Hacker News Reader

Comment by binwiederhier | Hacker News Reader

That's great. Though maybe the title should acknowledge that it's only for Google Cloud customers.

It's also quite odd that the article doesn't mention Let's Encrypt or the ISRG at all. I would have expected some sort of acknowledgement to their fantastic work over the years.

View on HN

>I would have expected some sort of acknowledgement to their fantastic work over the years.

i think their platinum-level sponsorship of the ISRG is probably more meaningful than a shout-out in a blog post.

That's actually a really good point. Though sometimes words are of acknowledgement are equally important. Likely not here though. I'm sure they'd rather take the money ;-)

Side note: I'm actually so proud that my company, Datto, has also been sponsoring Let's Encrypt for many years, and that I was the one to suggest it. It's not platinum, but it's still $50k per year.

Can you guys send an email to Atlassian?

That’s not peanuts indeed. Well done, Datto!

Well done Datto! Thanks for supporting the Internet.

yeah, words of encouragement can be great, but i don't think anybody at LetsEncrypt is in any doubt that Google is happy with the work they are doing.

also, i get the sense that getting wider adoption for FIDO outside of LetsEncrypt is a bit of a goal, so having a third-party announce support for FIDO without referring to it as "the LetsEncrypt protocol" is kind of a win for the FIDO people.

Who is the ISRG and what does a platinum-level sponsorship mean?

Parent 501(c)(3) of Let's Encrypt, among other projects, and $300-350k per year donation.

https://www.abetterinternet.org/about/

https://www.abetterinternet.org/sponsor/

> only for Google Cloud customers

for which there's no payment required as far as I can tell—just creating a Google Cloud account.

I think you do have to have a payment card on file, even if there’s no payments to be made.

... which is how they connect your account to who you are (or who you work for), so they can extract value from your social graph instead.

As the adage goes, if it's free you are the product, not the customer.

GCP, AWS, other IaaS providers, and more generally, any "post-paid" service company, needs your credit card, because the services these providers provide allow users to generate costs that can only be measured after-the-fact; and where those costs greatly exceed the free-plan limits, the provider wants to be able to pass those costs to the user.

As well, post-paid providers rely on credit-card billing information to deduplicate/KYC users. Without this sort of information, an, er, "ingenious" user could just sign up for a million free-tier accounts and lash them together into a quota-evading resource-sucking behemoth.

Sure. All that is true. And the credit card number ties your free GCE account to your profile with credit agencies that ad companies use to improve their targeting. It's not one thing or the other. It's both.

Do you think it's really that valuable for Google to spend engineering hours on an ad-targeting integration that will only improve targeting for maybe ~100k users at most?

And probably far less, actually. As many GCP users as there might be in the world, most of them are IT staff working for some-or-another enterprise; where that enterprise only has a single GCP billing-account administrator. Nobody else in the enterprise has their card on file. (And that billing-account administrator's card-on-file is just a corporate credit card, that tells you what the corporation buys, but tells you nothing about the individual. And their email is just a group/alias — billing@ or somesuch. Impossible to log into; impossible to browse the web as; no way to target ads at.)

You'd think the long tail of individual accounts could have more value, but those are the same users who GCP is least interested in recruiting to their platform, and the ones whose entered data is least trustworthy, because of all the spammers and crypto-miners attempting to use stolen credit cards to pay for service. You want to bind some poor random Joe's card-hubbed ad-profile to a spoke created by the person who stole their identity? That's negative average ad-targeting ROI!

Aside from providing a credit card that you give permission for google to hold and make transactions against.

Don't be ridiculous. Google isn't a scam company that would charge your card for no reason.

You’ve never put your own credit card on public cloud, I take it? Random, mysterious charges are practically a sport.

I did with several public cloud providers, including google cloud. Didn’t had any charges for something I didn’t order once.

But what if I were to lend my old android tablet to my visiting 4-year-old nephew, and I found they'd used my linked credit card to spend $500 on lootboxes in whalecraft?

Google is pretty famous for their nonexistent customer support, so I can't expect satisfaction there. And a chargeback would risk getting my entire account blocked.

> But what if I were to lend my old android tablet to my visiting 4-year-old nephew, and I found they'd used my linked credit card to spend $500 on lootboxes in whalecraft?

Google cloud billing doesn't use google pay, so...nothing?

I won't wade into if Google is a scam company, scam companies usually offer a support option they ignore which is more than can be said about google.

You gave them permission to bill for a reason. They decide the reason and amount and you pay. Seems risky without support.

No payment in cash. If it's Google, you're paying

Edot: Absolutely zero surprise this is getting voted to nowhere. CHEERS

They don't use GCP data whatsoever for ads. Thinking they'd peep into cloud customer data for the advertising boost (besides the small data point 'visited hostname console.cloud.google.com') is just as irrational as thinking Amazon would leverage AWS to look into Walmarts' suppliers' data to gain a competitive advantage.

Expecting good will of Google at this point seems sort of silly given repeating pattern of betrayal of trust.

Google Cloud is very different from Google's consumer product and advertising side. It has revenues of $13 billion which all comes directly from customers paying them money, not indirectly via advertising etc. Larger companies sign agreements with Google that cover what it does and doesn't promise, and enterprise companies are their highest priority target market. Google Cloud has its own CEO, Thomas Kurian, previously president of product development at Oracle, who was at Oracle for over 20 years.

The pattern you're referring to really doesn't apply to this business. If anything it's the opposite: Kurian's goal is to make Gcloud more like Oracle, and part of that involves making sure enterprise customers don't need to be skittish about the kinds of concerns you seem to be thinking of.

It's still losing money[1], which means that it's still cross subsidized by the ads business.

[1]: https://techcrunch.com/2022/02/02/with-a-22b-run-rate-does-i...

Yes, but that's because they're investing heavily to grow the business, as that article explains, which is hugely strategic for Google.

The fact that the current run rate is $22bn compared to the $13bn revenue in 2020 suggests that they're succeeding - that's some pretty significant growth.

It's a completely different business model from the consumer/ad side, and confusing the two is a mistake.

How do you square it with the fact that AWS is Amazon's most profitable division and is still arguably is innovating at a faster pace?

The counterfactual question one should be asking here is: Would gCloud exist if it were an independent company and not a bet?

Full disclosure: Xoogler here.

They're at different points in their lifecycle and have very different revenues and budgets. Amazon itself was famously unprofitable for many years, for exactly the same reason Gcloud is now: to grow the business as fast as possible.

Independent companies like that are funded by investors for years, all the time. The cloud market is expected to hit close to a trillion dollars by 2026. Gcloud doesn't have to take away a single existing AWS customer to win big.

> Xoogler

So what's your take? You think Google might just decide to shut down a fast-growing business with $13-20bn revenues and huge upside potential as if it were a free product like Reader? Or you think they somehow aren't able to make it profitable so will just give up?

Neither of those are really how these things work.

> So what's your take?

All I'm saying is that this business isn't a monopoly like search or a near duopoly like ads. The competition is stiff. There are two players who are well ahead of them and the ones behind aren't sitting still. I don't expect any miracles with run of the mill management consulting leadership at the helm.

Unless something has changed, Walmart does not want its suppliers to use AWS to mitigate that very risk.

https://www.cnbc.com/2017/06/21/wal-mart-is-reportedly-telli...

That’s my point, it’s irrational to think that. Amazon has so little to gain by doing that compared to the absolute mass of customers that would be migrated off of aws within a few years of such a scandal.

Counterpoint: at the same time, it's something you can do and not do at the same time - everything in AWS is virtualized (save for $$$$$ physical hosts almost nobody uses) and it's not like it's impossible to undetectably analyse/monitor running VMs in realtime.

So you can both do this (silently) and "not" do it (because it's impossible to detect), and it would thus be stupid to not actually do it because it's free data.

Zooming out I think this and the opposite view are equally possible. Just articulating what this perspective looks like.

That was because Amazon is a competitor to Walmart. It's not a risk that would apply to most companies. And it's probably not a very realistic risk, either. Walmart may have just been trying to hurt a competitor.

besides the small data point 'visited hostname console.cloud.google.com'

That's a pretty huge data point though. It puts you into a very specific market segment.

I'm sure they're just "using insights from their operational infrastructure to improve the services they provide to customers". Which is still something that all but the most skeptical of engineers and managers would fail to see a potential problem with.

For now.

They've changed free things to not free things many times in the past. Or started restricting features once it gets popular. Don't depend on it for anything you want to last a while.

I’m certainly overlooking something but the risk seems small since it uses an open standard. You should be able to fallback to another acme provider right?

Yeah, but... given there are other ACME providers, why would I choose one from the a company with googles reputation for sunsetting products?

As a Let’s Encrypt user and fan, I can answer this.

LE is great, but their SLO is significantly below our customers’ expectations. For us, Google wouldn’t replace LE, it would supplement LE for higher reliability.

Seeing more providers conforming to ACME at a price point of “free” is great for the ecosystem.

Is this with provisioning many new certs?

The recommended renewal cycle gives you a 30 day lead on failure becoming a problem, plenty of time for multiple retries or recovery processes to use an alternate.

The only issues I've ran into, have stemmed from DNS for wildcard certs, where a client's DNS provider is... pretty crap about updating records despite low ttls being set.

Some people on the cloud like to issue a certificate per machine, so no ACME = no new VMs.

It’s a web hosting business. New customers want effortless free TLS asap. We get customers who routinely create new sites who come to expect fast provisioning.

Does Google Cloud also have that reputation? I've seen renames and well-motivated deprecations, but not the Google Graveyard kind of events.

"This API you use has changed, you have 6 months to update your code" is not unheard of.

On AWS, the equivalent is "We no longer recommend this, and it's not visible in the AWS console unless you are already using it or have asked support to enable it, but it will keep working indefinitely".

More like: first they become the dominant acme provider, then the other ones go away so they are the only one left, and then they have you where they want you.

They have you where they want, you are willing to pay, ... but the close down the service anyway with a three months notice.

> first they become the dominant acme provider

The traditional way a big tech company becomes the dominant provider is to embrace an open interoperable protocol to minimize the friction of switching from another provider. Later, when they have captured enough of the market, they extend the protocol to gradually reduce the de facto interoperability with other providers to increase the friction of switching to any other provider.

https://en.wikipedia.org/wiki/Embrace%2C_extend%2C_and_extin...

Its probably already worked into the hosting bill somehow anyway... There are numerous places to hide the cost in any CSP bill.

Even traditional host providers are raising their costs for the same mysterious reasons, although hardware costs are historically trending lower over time.

> hide the cost ...

That's one way to put it.

Solving your customer's problems and making it more likely they will keep using your main offering is another way to put it.

For me creating an account is a sort of payment.

And the risk that Google AI decides you are not worthy, your account is deactivated and you loose access to this in the same second.

Centralization with no accountability besides blaming things on AI is problematic.

I mean sure but you have an account with Let's Encrypt too which can be arbitrarily banned. LE has even less accountability.

Fair point, but it wouldn't be banned because of some other unrelated activity. Much more isolated implications that way.

Quite odd indeed, maybe there are some legal reasons not to mention it. If they could, I feel the whole article could be condensed to "we just shipped a LetsEncrypt alternative gated to Google Cloud customers".

It probably doesn't mention Let's Encrypt because that's an extra opportunity for confusion since you do not (usually?) get Let's Encrypt certificates from this service.

This service has nothing to do with Let's Encrypt, you get certificates from GTS, so the connection would be that ISRG / Let's Encrypt was part of the work to develop ACME and so on. But that's increasingly ancient history. The goal of the work was that almost all CAs in the Web PKI would offer ACME (the people doing like a dozen issuances per month maybe not, but there's an open question about whether they should even exist) -- not just this one free service operated by a charity and here Google are following through.

I'd say that it's the same as if Zig announces each new compiler version or whatever and doesn't acknowledge that, yeah, Grace Hopper is in some sense responsible for the idea of compilers. [[ Hopper wrote the first "compiler" although today we would not consider her software to be a "compiler" but maybe a loader/linker. Anyway, the whole practice of having the machine do the boring job of writing machine code while a human just expresses what they wanted is down to Grace. Thinkers like Turing would have known this was possible in theory but Grace wasn't writing theory, she was doing engineering. ]]. Grace Hopper is important and worth celebrating, but it would be weird to insist on making a specific programming language that has nothing to do with Grace mention her every release.

I guess if you're an Oscar's speech writer going for that "I want to thank my parents, without whom I wouldn't be here today" vibe, then yeah. But otherwise it seems unnecessary.

It uses google cloud binding, but you can request and use these certificates from any system.

On this note, I wonder if there is any way to get a google cloud account, and use it JUST for this, without any $$. I don't think so, right? because best option you still need some sort of reverse proxy to your own server?!

As far as I can tell (I now have the API access, so I might test later), you only need to provide additional authentication credentials to Certbot. The private key stays on your server. The CA server _might_ reject requests that do not come from GCP, or attempt to obtain one for a domain name not setup with GCP.

There is no origin limitation. It works on premise or cross cloud.

Oh, so they've achieved parity with AWS. That's still nice, if not particularly impressive.

Well sort of. Using ACME means you're not locked into AWS API.

No, GCP has had arguably a superior TLS story for years.

For example they do managed TLS for their workloads like AWS but they operate their own CA rather than outsourcing to Digicert for certificate issuance which gives them a better SLA.

They have a global load balancer offering that enables TLS to terminate everywhere GCP is without having to manage a bunch of discrete load balancers, this also supports managed TLS.

They now support a very large number of certificates in the global load balancer product which allows SaaS products like hosting services to leverage the global load balancer rather than deploying a load balancer per 25 certificates (the limit per AWS LB).

And now let you enroll for certificates from the same CA they use even if you terminate TLS rather than having them do it for you. They do this via a standard API (ACME) which lets you have uniform and agile device compatibility regardless of how you deploy TLS. AWS doesn't let you do this at all.

(I should note I was the PM for most of these releases and am still the PM for Google Trust Services the CA used for this ACME release)

Nice!

Also because up till, I guess now, Google Cloud TLS is just LetsEncrypt.

Which does raise an issue: I'm not sure why you'd use this, given Google's history of killing projects. What's the compelling reason to switch - especially since they feel content to release this under the `alpha` command set for the CLI tool.

Don't forget the "going to the store for some milk" phase of google products where for a few years it doesn't get any feature additions or bugfixes.

I don't understand why anyone would go for this given LE is mature, stable, trusted, and well-supported.

Say one day it just stops issuing you new certs; now what? Call someone? Nope. Post in the forums? Not unless you want to get asked if you've cleared your Chrome cache.

The reason may be so that services on private subnets don't need internet access to use Let's Encrypt. Just a guess though.

You don't need direct Internet access to use Let's Encrypt, as long as you can arrange for the challenge response to appear in public DNS under the name you want to use.

Would you mind giving an example of what that might look like? Or linking to something? I've always struggled with needing to open ports temporarily on stuff behind my own reverse proxy to avoid passing the certs by hand, and it sounds like something that'd be useful to understand.

The magic phrase is “DNS-01” challenge. You place a DNS TXT record to validate control of the domain. There are lots of ACME clients that support a wide variety of DNS service providers. For example, I have a Home Assistant server which automatically issues certs using Gandi DNS and the HA Lets Encrypt support, all without being on the internet (except for the DNS entries)

It's the DNS-01 challenge[1]. This reduces the challenge to using some DNS provider with an API supported by a client[2] / [3], as well as the server needing to be able to reach the LE-API. We use this with the CNAME delegation into an irrelevant zone everywhere to get wildcard certificates for our LBs ( meaning: the _acme_challenge.example.com record is just a CNAME for _acme_challenge.dont.ever.use.this.example.com, and the servers just have credentials to modify records in the zone <dont.ever.use.this.example.com>)

1: https://letsencrypt.org/docs/challenge-types/#dns-01-challen...

2: https://eff-certbot.readthedocs.io/en/stable/using.html#dns-...

3: https://github.com/acmesh-official/acme.sh/wiki/dnsapi

The acme client needs to reach LE though. Or you need to do a dance where the client is outside of the private network and ships the certificate into the private network.

I think you're confusing internet access with reachable from the internet.

If I remember correctly, you don't need your server to be reachable from the internet, but you still need to be able to contact your DNS provider and the LE server, so you need internet access

LE has compatibility issue with very old Android/OpenSSL. https://community.letsencrypt.org/t/production-chain-changes... Possibly Google's cert doesn't have the issue?

Yeah, that transition didn't go as smoothly as they might have hoped. Most of us first-world programmers can just shrug and say "don't use unsupported versions," but I've had multiple non-technical clients call me up urgently and ask why a (relatively small, but not insignificant depending on the market, and definitely not in their control) subset of their users were seeing certificate errors.

So I don't recommend LE to my clients anymore. But it's a hassle to buy certificates the old way after having tasted ACME, so I'm always looking for an ACME-compatible alternative. ZeroSSL is backed by a more conservative Sectigo CA, but its ACME endpoints aren't very reliable. If this Google cert becomes widely available, I might just as well switch to it. :)

From memory, ZeroSSL also gets expensive after a couple of domains, and I had issues using certbot rather than acme.sh with it.

Nowadays you can get virtually unlimited 90-day certs from ZeroSSL if you use ACME through the EAB feature rather than using their API.

But their ACME support seems half-hearted at best. The endpoints often return errors for no reason, compatibility with clients is hit-and-miss, and they keep spamming you with renewal notices even if you renew the cert. For important domains these days I just get a cheap 1-year DV cert like the good ol' days.

Funny you should mention about the forums. There's been a fairly notable Chrome issue intermittently affecting users on MacOS X since late last year and Google seem oblivious to it.

https://support.google.com/chrome/thread/135844398/chrome-is...

So, yeah... I don't want to depend on a free Google Cloud account for SSL.

i am pretty certain (but happy to be corrected) that Google has never killed a GCP service that paying customers are using.

Google != Google Cloud Platform

> Which does raise an issue: I'm not sure why you'd use this, given Google's

.. the biggest advertising giant on the planet that sucks up as much data as they possibly can about their subjects.

> What's the compelling reason to switch

>up till, I guess now, Google Cloud TLS is just LetsEncrypt.

Google Cloud has been issuing certs for customer use off its own CA ( pki.goog ) for a while. It went GA April 27, 2020:

https://cloud.google.com/load-balancing/docs/release-notes#A...

You can see the documentation here:

https://cloud.google.com/load-balancing/docs/ssl-certificate...

GCP load balancers can automatically switch between pki.goog and letsencrypt.org if one goes down. Or you can restrict the load balancer to just get certs one of them by using a DNS CAA record.

Up till now, you could only use pki.goog with GCP load balancers. This new release allows pki.goog to be used with anything, because you actually control the private key.

Disclosure: I work in Google Cloud.

> GCP load balancers can automatically switch between pki.goog and letsencrypt.org if one goes down.

Awesome, glad to see other ACME clients using issuer fallback, like Caddy does!

Do you know if there are any plans to making the Google ACME CA usable without registration? Having to register for an account and using credentials is a huge barrier to entry for many less-technical users. Caddy is able to use LE and ZeroSSL because they both don't _require_ accounts (ZeroSSL recommends it, partially as an upsell, but it's not necessary).

The more no-registration CAs exist, the more resilient ACME clients can be.

Sorry, I don't know the plans. I don't work on that team.

Mid-year 2020 is not what I'd call running a service for "a while". That means its been GA just long enough to drop out of college.

You could drop out after one day. If you drop out fast enough you might qualify for a refund.

I heard of someone who enrolls in a community college every winter, then goes skiing on a student discount, then drops out quickly and gets a refund. Not very ethical...

Sorry, this was completely unrelated to your point.

If you are worried about `Google's history of killing projects`, you will anyway not be on Google Cloud.

And that they're using chrome browser to make certs a mandatory and increasingly expensive monthly subscription service for site owners even though for many sites that don't handle secure transactions it really shouldn't be a requirement.

In 2026 only rich companies will be able to maintain sites with the way we're going folks.

Certs are free. What do you mean monthly subscription? Also why would secure transactions be necessary? Preventing third parties from injecting ads / mining scripts / phishing logins benefits all sites.

LetsEncrypt is free.

SSL certificates are cheaper than most DNS domain renewals are. ssls.com has them for as low as $3.88 per year; and you can use Let’s Encrypt for free.

Reply on news.ycombinator.com