Automate Public Certificates Lifecycle Management via RFC 8555 (ACME)
cloud.google.com
cloud.google.com
It's also quite odd that the article doesn't mention Let's Encrypt or the ISRG at all. I would have expected some sort of acknowledgement to their fantastic work over the years.
i think their platinum-level sponsorship of the ISRG is probably more meaningful than a shout-out in a blog post.
Side note: I'm actually so proud that my company, Datto, has also been sponsoring Let's Encrypt for many years, and that I was the one to suggest it. It's not platinum, but it's still $50k per year.
also, i get the sense that getting wider adoption for FIDO outside of LetsEncrypt is a bit of a goal, so having a third-party announce support for FIDO without referring to it as "the LetsEncrypt protocol" is kind of a win for the FIDO people.
for which there's no payment required as far as I can tell—just creating a Google Cloud account.
Google is pretty famous for their nonexistent customer support, so I can't expect satisfaction there. And a chargeback would risk getting my entire account blocked.
Google cloud billing doesn't use google pay, so...nothing?
You gave them permission to bill for a reason. They decide the reason and amount and you pay. Seems risky without support.
Centralization with no accountability besides blaming things on AI is problematic.
They've changed free things to not free things many times in the past. Or started restricting features once it gets popular. Don't depend on it for anything you want to last a while.
The traditional way a big tech company becomes the dominant provider is to embrace an open interoperable protocol to minimize the friction of switching from another provider. Later, when they have captured enough of the market, they extend the protocol to gradually reduce the de facto interoperability with other providers to increase the friction of switching to any other provider.
https://en.wikipedia.org/wiki/Embrace%2C_extend%2C_and_extin...
LE is great, but their SLO is significantly below our customers’ expectations. For us, Google wouldn’t replace LE, it would supplement LE for higher reliability.
Seeing more providers conforming to ACME at a price point of “free” is great for the ecosystem.
The recommended renewal cycle gives you a 30 day lead on failure becoming a problem, plenty of time for multiple retries or recovery processes to use an alternate.
The only issues I've ran into, have stemmed from DNS for wildcard certs, where a client's DNS provider is... pretty crap about updating records despite low ttls being set.
On AWS, the equivalent is "We no longer recommend this, and it's not visible in the AWS console unless you are already using it or have asked support to enable it, but it will keep working indefinitely".
Even traditional host providers are raising their costs for the same mysterious reasons, although hardware costs are historically trending lower over time.
That's one way to put it.
Solving your customer's problems and making it more likely they will keep using your main offering is another way to put it.
As the adage goes, if it's free you are the product, not the customer.
As well, post-paid providers rely on credit-card billing information to deduplicate/KYC users. Without this sort of information, an, er, "ingenious" user could just sign up for a million free-tier accounts and lash them together into a quota-evading resource-sucking behemoth.
And probably far less, actually. As many GCP users as there might be in the world, most of them are IT staff working for some-or-another enterprise; where that enterprise only has a single GCP billing-account administrator. Nobody else in the enterprise has their card on file. (And that billing-account administrator's card-on-file is just a corporate credit card, that tells you what the corporation buys, but tells you nothing about the individual. And their email is just a group/alias — billing@ or somesuch. Impossible to log into; impossible to browse the web as; no way to target ads at.)
You'd think the long tail of individual accounts could have more value, but those are the same users who GCP is least interested in recruiting to their platform, and the ones whose entered data is least trustworthy, because of all the spammers and crypto-miners attempting to use stolen credit cards to pay for service. You want to bind some poor random Joe's card-hubbed ad-profile to a spoke created by the person who stole their identity? That's negative average ad-targeting ROI!
Edot: Absolutely zero surprise this is getting voted to nowhere. CHEERS
https://www.cnbc.com/2017/06/21/wal-mart-is-reportedly-telli...
So you can both do this (silently) and "not" do it (because it's impossible to detect), and it would thus be stupid to not actually do it because it's free data.
Zooming out I think this and the opposite view are equally possible. Just articulating what this perspective looks like.
The pattern you're referring to really doesn't apply to this business. If anything it's the opposite: Kurian's goal is to make Gcloud more like Oracle, and part of that involves making sure enterprise customers don't need to be skittish about the kinds of concerns you seem to be thinking of.
[1]: https://techcrunch.com/2022/02/02/with-a-22b-run-rate-does-i...
The fact that the current run rate is $22bn compared to the $13bn revenue in 2020 suggests that they're succeeding - that's some pretty significant growth.
It's a completely different business model from the consumer/ad side, and confusing the two is a mistake.
The counterfactual question one should be asking here is: Would gCloud exist if it were an independent company and not a bet?
Full disclosure: Xoogler here.
Independent companies like that are funded by investors for years, all the time. The cloud market is expected to hit close to a trillion dollars by 2026. Gcloud doesn't have to take away a single existing AWS customer to win big.
> Xoogler
So what's your take? You think Google might just decide to shut down a fast-growing business with $13-20bn revenues and huge upside potential as if it were a free product like Reader? Or you think they somehow aren't able to make it profitable so will just give up?
Neither of those are really how these things work.
All I'm saying is that this business isn't a monopoly like search or a near duopoly like ads. The competition is stiff. There are two players who are well ahead of them and the ones behind aren't sitting still. I don't expect any miracles with run of the mill management consulting leadership at the helm.
That's a pretty huge data point though. It puts you into a very specific market segment.
This service has nothing to do with Let's Encrypt, you get certificates from GTS, so the connection would be that ISRG / Let's Encrypt was part of the work to develop ACME and so on. But that's increasingly ancient history. The goal of the work was that almost all CAs in the Web PKI would offer ACME (the people doing like a dozen issuances per month maybe not, but there's an open question about whether they should even exist) -- not just this one free service operated by a charity and here Google are following through.
I'd say that it's the same as if Zig announces each new compiler version or whatever and doesn't acknowledge that, yeah, Grace Hopper is in some sense responsible for the idea of compilers. [[ Hopper wrote the first "compiler" although today we would not consider her software to be a "compiler" but maybe a loader/linker. Anyway, the whole practice of having the machine do the boring job of writing machine code while a human just expresses what they wanted is down to Grace. Thinkers like Turing would have known this was possible in theory but Grace wasn't writing theory, she was doing engineering. ]]. Grace Hopper is important and worth celebrating, but it would be weird to insist on making a specific programming language that has nothing to do with Grace mention her every release.
I guess if you're an Oscar's speech writer going for that "I want to thank my parents, without whom I wouldn't be here today" vibe, then yeah. But otherwise it seems unnecessary.
For example they do managed TLS for their workloads like AWS but they operate their own CA rather than outsourcing to Digicert for certificate issuance which gives them a better SLA.
They have a global load balancer offering that enables TLS to terminate everywhere GCP is without having to manage a bunch of discrete load balancers, this also supports managed TLS.
They now support a very large number of certificates in the global load balancer product which allows SaaS products like hosting services to leverage the global load balancer rather than deploying a load balancer per 25 certificates (the limit per AWS LB).
And now let you enroll for certificates from the same CA they use even if you terminate TLS rather than having them do it for you. They do this via a standard API (ACME) which lets you have uniform and agile device compatibility regardless of how you deploy TLS. AWS doesn't let you do this at all.
(I should note I was the PM for most of these releases and am still the PM for Google Trust Services the CA used for this ACME release)
Which does raise an issue: I'm not sure why you'd use this, given Google's history of killing projects. What's the compelling reason to switch - especially since they feel content to release this under the `alpha` command set for the CLI tool.
Google Cloud has been issuing certs for customer use off its own CA ( pki.goog ) for a while. It went GA April 27, 2020:
https://cloud.google.com/load-balancing/docs/release-notes#A...
You can see the documentation here:
https://cloud.google.com/load-balancing/docs/ssl-certificate...
GCP load balancers can automatically switch between pki.goog and letsencrypt.org if one goes down. Or you can restrict the load balancer to just get certs one of them by using a DNS CAA record.
Up till now, you could only use pki.goog with GCP load balancers. This new release allows pki.goog to be used with anything, because you actually control the private key.
Disclosure: I work in Google Cloud.
Awesome, glad to see other ACME clients using issuer fallback, like Caddy does!
Do you know if there are any plans to making the Google ACME CA usable without registration? Having to register for an account and using credentials is a huge barrier to entry for many less-technical users. Caddy is able to use LE and ZeroSSL because they both don't _require_ accounts (ZeroSSL recommends it, partially as an upsell, but it's not necessary).
The more no-registration CAs exist, the more resilient ACME clients can be.
I heard of someone who enrolls in a community college every winter, then goes skiing on a student discount, then drops out quickly and gets a refund. Not very ethical...
Sorry, this was completely unrelated to your point.
Google != Google Cloud Platform
I don't understand why anyone would go for this given LE is mature, stable, trusted, and well-supported.
Say one day it just stops issuing you new certs; now what? Call someone? Nope. Post in the forums? Not unless you want to get asked if you've cleared your Chrome cache.
So I don't recommend LE to my clients anymore. But it's a hassle to buy certificates the old way after having tasted ACME, so I'm always looking for an ACME-compatible alternative. ZeroSSL is backed by a more conservative Sectigo CA, but its ACME endpoints aren't very reliable. If this Google cert becomes widely available, I might just as well switch to it. :)
But their ACME support seems half-hearted at best. The endpoints often return errors for no reason, compatibility with clients is hit-and-miss, and they keep spamming you with renewal notices even if you renew the cert. For important domains these days I just get a cheap 1-year DV cert like the good ol' days.
https://support.google.com/chrome/thread/135844398/chrome-is...
So, yeah... I don't want to depend on a free Google Cloud account for SSL.
1: https://letsencrypt.org/docs/challenge-types/#dns-01-challen...
2: https://eff-certbot.readthedocs.io/en/stable/using.html#dns-...
If I remember correctly, you don't need your server to be reachable from the internet, but you still need to be able to contact your DNS provider and the LE server, so you need internet access
.. the biggest advertising giant on the planet that sucks up as much data as they possibly can about their subjects.
> What's the compelling reason to switch
In 2026 only rich companies will be able to maintain sites with the way we're going folks.
DigiCert and the like will typically require domain verification at the TLD+1, which is meaningless gibberish that isn't even remotely an RFC standard. There's no such "concept" in DNS, which is intended to be delegated.
So for example if I'm tasked with deploying a web app to "dev1.app.project.org.parentcompany.megacorp.co.uk" where the "project team" is based out of -- say -- Australia, then DigiCert will insist that I verify that I own "megacorp.co.uk", which... I don't. The parent company might not either. MegaCorp's UK head office does. They've never heard of me, and it'll take me a month to get through to someone who cares about my tiny, outsourced project down under.
This kind of thing has happened to me repeatedly across both corporate and government projects. A 2-week project can have a 1 month delay added to it because of this.
ACME gets it right, and nobody else does.
For some reason (probably industry collusion), X.509 Name Constraints were there from the beginning to enable an extensible generalization of this, but we really missed the boat on supporting them, until eventually doing so became very very hard. We're still making progress toward enablement, though!
It's pure rent seeking with approximately zero value provided in exchange.
Let's Encrypt demonstrated that the marginal cost of a certificate is close enough to $0 to round down to precisely zero dollars.
To see how deeply rooted this corruption is -- and it is corruption -- ask any major cloud provider like Azure or AWS whether they are willing to provide native ACME protocol integration to enable their customers to request Let's Encrypt certificates for arbitrary DNS-hosted services.
You'll hear nothing back. "No comment" or "We're considering it".
In other words: "We considered it, but then our boss's boss made it very clear to our boss that he was getting a kick-back from DigiCert and to never mention such topics ever again."
...at least, it's pretty hard if your CA is located in a Western country. Which is the big loophole here. The fact that we trust CAs headquartered in arbitrary potentially-unfriendly countries, to make claims about the entire space of domains, is pretty silly. Trusting e.g. the Russian CAs in the trust-store only when they make claims about .ru domains (a.k.a. the X.509 Name Constraint extension), would obviate a lot of the concerns people have about X.509's centrally-curated trust store model.
> [Criterion 2] Physical existence
> To check the organization's physical existence and business presence, the Certificate Authority must verify that the physical address provided by the Applicant is an address where the organization conducts business operations (not a mail drop, P.O. box or an address for an agent of the Organization). This address will be included into the body of the SSL certificate after verification.
That effectively translates to "there should be an address that criminal investigators can be sent to find and detain employees of the company."
> [Criterion 4] Operational existence
> To make sure that an organization is financially active and engaged in business activities, the Certificate Authority must verify that at least one of the following requirements is met:
> The Organization has been in existence for at least three years
> The Organization is registered in the Dun & Bradstreet database or Qualified Government Tax database
> The Organization has an active demand deposit account which can be proved by a bank statement.
That effectively translates to "you can't just make up an LLC and immediately get an EV cert for it, even if you do tell them your house is the headquarters. You'd have to put a good amount of time and effort into simulating a real business. So much that, if this were a spoofing attempt, you'd be noticed as in breach of trademark by the company you're trying to spoof, long before you got away with it."
> Professional Opinion Letter
> If you need to obtain an EV certificate urgently or prefer keeping the company details confidential, it is possible to send a professional opinion letter signed by a Lawyer, Public Notary or Certified Public Accountant. The person who signed the legal opinion or accountant letter should have a valid license within the country where the organization is registered or the country where the organization maintains an office or a physical facility. To expedite the validation process, we highly recommend requesting a Professional Opinion from a person who speaks English so that he or she can confirm the signature during phone verification with a Comodo (now Sectigo CA) validation agent.
And this, finally, translates to "you can mask your identity, but only behind someone who's notoriously signed a Code of Ethics that requires them to surrender your identity to criminal investigators when asked, without needing a subpoena."
https://arstechnica.com/information-technology/2017/12/nope-...
For anyone curious, see RFC 5280 § 4.2.1.10:
The name constraints extension, which MUST be used only in a CA
certificate, indicates a name space within which all subject names in
subsequent certificates in a certification path MUST be located.
Restrictions apply to the subject distinguished name and apply to
subject alternative names. Restrictions apply only when the
specified name form is present. If no name of the type is in the
certificate, the certificate is acceptable.
* https://www.alvestrand.no/objectid/2.5.29.30.html
Client support exists in OpenSSL 1.0.0, Windows 7, Mac OS 10.13.3, iOS 11.2.6. (Android?)
I think one technical 'loophole' is that while NCs apply explicitly to SANs, per the spec they do not apply to the Common Name. Though quickly skimming the RFC, I do not see anything that would prohibit them being applied to the CN. So you can probably do it under the guise of "undefined behaviour".
If any company should operate a CA, it should probably not be random enterprise companies.
For one example: https://wiki.mozilla.org/CA:Visa_Issues
The IETF's PKIX defines how X.509 certificates work for the Internet, because understandably X.509 is for the X.500 system and the Internet is not the X.500 system. PKIX defines the Subject Alternative Name (SAN) which allows the Internet's names (DNS names and IP addresses) to be subjects of X.509 certificates rather than needing the non-existent X.500 directory system for names.
I think you're mostly talking about the Web PKI. But the Web PKI was never "modelled" to work how you've described. At its outset, Netscape (who invented SSL and thus set this ball rolling) wanted pre-existing neutral services rather than they'd run everything and then obviously rival web browsers (including Microsoft's Internet Explorer) would have their own and it's pointless. Several important Certificate Authorities already existed at that time, issuing X.509 certificates in the X.500 system largely to banks, and were happy to take $$$ to issue certificates for this SSL experiment. Initially Netscape basically accepted any company that said they were in this business, and there were no rules (other than those the companies themselves decided on).
But in the modern era the Web PKI is in practice publicly overseen by m.d.s.policy, a policy discussion group of Mozilla. The lack of Name Constraints isn't because of some weird conspiracy, it's simply that Apple didn't support them for many years so if you used Name Constraints then now none of your certificates work on Safari or other Apple products (if Constraints are to work at all they must be marked Mandatory, and if you don't implement a Mandatory feature, you can't be sure if this certificate is valid, so you can't trust it).
For oversight to be effective the sort of delegation you envision is impossible, and accordingly where anything like it did exist the subCAs have moved back to being under physical control of the root CAs. In fact the big problem we had with Symtantec comes down to the lack of effective physical control, with CrossCert able to cause issuance from Symantec's systems yet having no effective oversight.
Also your timeline is badly off in thinking about the IP block allocations. CIDR happened in like 1993, Netscape's SSL doesn't happen until at least 1996. The companies that were issued class A IP blocks before CIDR are mostly smaller and few still exist in the same form, Apple, Comcast and AT&T maybe make sense, but Ford and Prudential Financial not so much. Microsoft are not on that list.
ACME-supporting CAs and other CAs are all held to the same compliance standards in the end, so nothing prevents the other from doing what you want.
I was trying to verify ownership OF A SUBDOMAIN (e.g. sub.example.com) and I was given a TXT record to add to my DNS. I added it to the subdomain sub.example.com but the verification kept failing. It wasn't until I added it to my domain (example.com) that the verification succeeded. Who thought that was a good idea?
It was especially frustrating since I had verified ownership of the domain minutes before. Why do I have to do it again for every subdomain?
Domain Validation certificates are insecure for quite a few types of security certifications due to how stupid easy it sometimes is to get your subdomain validated through social engineering.
Manager X that manages a small project in subcontractor level can get screwed over way easier than security expert that is managing top level domain of specific corporation.
So there are pros and cons of that solution.
It sounds like the roots are the ones that Google uses to sign it's own domains (like Google.com), which are these: https://pki.goog/repository/
So likely support for them is fairly broad.
Why does it have to be a third party? My understanding is that as a user you just need to trust the certificate and it's owner, and as long as the trust is there it shouldn't matter what that certificate is signing.
No longer especially that pre-LE, those same third-party sites already broke the rules, even with their EV offerings. Everything has been reduced to "is this domain at least controlled by them?" which is easily auditable (organisational verification has been significantly devalued). Now, Let's Encrypt only verifies domains and not trustworthiness (in fact, they won't revoke certificates known as phishing sites). Also EV certificates are nearly worthless unless you want to bypass many antivirus' HTTPS interception.
The strange thing is that some companies mix and match - aws.amazon.com is signed with their certificate, while www.amazon.com is signed with a DigiCert certificate. You'll find Microsoft seems to mix certs as well, also using DigiCert for some sites. No clue why this is.
People's saltiness about the consumer/advertising side of Google having killed their favorite free product is pretty irrelevant in this context.
> same people are making decisions in upper management.
That's not true. Google Cloud has its own CEO and a very different business model.
> they don’t make anything if it is not bringing more revenue.
Sure - they're a for-profit business. Google Cloud's revenue was $13bn in 2020, and as of last quarter it had a run rate of $22 bn. That money comes direct from paying customers. The business model of the consumer/ad side really is irrelevant here.
Business model is different indeed, but the parent company is still Alphabet, which in the end has huge influence.
I understand not all large companies have the luxury of picking their service providers but I do believe it's in everyone's best interest to diversify web service infrastructure. One, to hedge against catastrophic failures that may arise from their platform going down and two, to better balance the distribution of power on the internet as a whole.
One should not have to do anything extra unless the requester too also run a load balancer in front of their subdomain in question.
That’s why ACME challenge offers several different type of protocols (to get by the many caching servers.
It's especially great because letsencrypt is operated by US company ISRG and zerossl seems to be from Austria, so if you're not happy with your server being dependant on US, it might be a good option.
So if you’re spinning up tens or hundreds of review apps per day, you can’t get a fresh cert for each, and so you need to do something different than your production environment does. (A wildcard cert is the obvious choice.)
I hope this offering has a high enough quota that you can get enough certs to do review apps properly; the marginal cost to Google per customer is probably negligible, whereas LetsEncrypt doesn’t have other revenue generating offerings they can use to cover their operating costs.
https://docs.gitlab.com/ee/ci/review_apps/
As part of my pre-merge pipeline I deploy the application to k8s, create a public IP, DNS entry from the branch slug, TLS-to-the-pod with wildcard cert, and then run the full end-to-end test suite. If you get your docker caching right, it can be as little as a couple minutes to deploy the whole stack
This makes it a lot easier for engineers to iterate on their changes with stakeholders, not to mention it lets you test infra and API changes before actually merging them.
We got to tens of engineers using this, so I didn't optimize utilization heavily; this ends up being a bit expensive, but I think worth it.
(I also give developers a script to run this fixture DB locally too - it makes the onboarding process a lot easier if you don’t need to run the DB init and migrations, and generally means developers have a more robust set of test data to work with. )
>"Each of these have different scenarios where their use makes the most sense, for example TLS-ALPN-01 might make sense in cases where HTTPS is not used and the requestor does not have access to dynamically update DNS records."
I'm confused by TLS-ALPN-01. I understand the idea of using certs for domain verification but if there is no TLS in use how does the client verify this after the cert has been issued exactly?
That makes good sense. Thanks for putting it so succinctly. Cheers.
Q: Do you offer certificates from a pure ECC based certificate chain?
A: Not at this time.
Is this another such risk vector?
no because there are other CAs which offer support for the ACME protocol.
That's the good thing about open standards.
> Not at this time.
I thought punycode solved all integration issues and is meant to be backwards compatible ...
I think this is a security measure that prevents people from getting certificates for homoglpyh domains.
Browsers already offer some protection, but why not add an additional layer?
Wether it lasts or not, this surely has to be an issue for Google innovations going forward? If the perception is that any new thing will die, especially not-consumer-scale things, then how do they build traction?
This is presumably a core feature of Google Cloud, and if you’re already using their other cloud products, I can’t imagine being too worried about features randomly going away.
(Disclaimer: While I work for Let’s Encrypt, this is my own opinion and not necessarily that of my employers)
And that's before we take customer service into the equation.
Play Music died after 9 years, and that even featured paid souscriptions...the logic of "paid for and around" doesnt seem to be a solid argument, esp for a company making money primarily off of data - as of 2021 their cloud business is only 5% of their revenue, ~74% of their revenue is ad based the rest is devices (nest, phones), and Youtube.
I think its pretty reasonable to be concerned about the longevity of this product - maybe we will be wrong, but.
I get your beef with google service availability longterm and the google list of killed products, but this is an instance of a service which in true Internet form "routes around damage"
This isn't the killer "what if google stop" problem area. ACME based certification doesn't care: find another TA, and move on with your life.
Assuming there is a reason to switch, the switch back is not without cost. Ergo, either way, I'm not using this. Then again I don't use GCP for the same reason.
My bigger point is that logical or not, my perception appears to be common. And that may hurt Google's ability to get traction with unique services in the future.
No one cares about Google's CA, but perhaps this perception issue is a real problem?
mainly I suspect this is to pacify people inside GKE who don't like calling outside of the protected space to get certificates to exist.
What's fascinating to me is the apparent irrationality, from people who otherwise tend to pride themselves on being rational.
I can understand people saying they'll never deal with Google again because they killed their favorite free product or whatever. That's fair. But the attempt to justify that position as some sort of rational risk calculation, taking the actual facts of the matter into account, is misguided.
In some cases this is due to a lack of info and disinterest in correcting that, but many other cases just seem to be emotion masquerading as a thought process.
And I would suggest that if they'd just killed reader, then OK, stuff like that happens. But Google has killed a Lot of things. Probably for good reason. But they get a reputation for being "flighty" when it comes to new things.
Sure it's emotional, but that reputation comes into my thinking when I make product choices. I'll use HERE maps over Google maps etc. And I'm not terribly inclined to make their stuff part of my critical work flow.
So sure I'll Google search and YouTube all day long. But I'll use AWS or Azure over GCP. 90% because I can get someone on the phone. 9% because I wonder when Google will decide to kill GCP. And 1% because I don't trust the Google AI not ti just terminate my account with no recourse.
If that’s true, Google’s products aren’t subject to cancelation in the way that any given web service is.
In any case, given their ethical bankruptcy, I wouldn’t use anything from them given the choice.
(I feel like Google just doesn't understand branding. I remember when they launched the Pixel C after the Chromebook Pixel, and it ran Android instead of ChromeOS. What was once a Chrome brand became an Android brand. I guess because Chrome was doing pretty well at the time, and Android just reminded people of slow phones that ran out of battery instantly. Sigh!)
not "(Google Cloud) Print"
https://twitter.com/YoungbloodJoe/status/1504107839262011403
My point is though that while you say "most don't have that extreme a position", I'm not sure what % it is. And does that perception, that it exists at all, hinder them from future rollout?
I won't be trying it out.
yeesh
The equation does change if there are value-adds that start getting offered and used which make it more difficult to transition elsewhere (with docs, gdrive etc. being examples of value-adds for the free custom-domain email offering).
If not, possibly reserve a spot here: https://killedbygoogle.com/
Obviously kidding! Glad to see this brought online for GCP customers.
It's notable that Azure can do better here because they have to (the expensive Visual Studio product comes with "free" Azure credits, you aren't paying for them so there is nobody to "bill" if you run out, they just shut stuff off). They appear to do "better" by just eating the extra cost after shutting off. That's still a better user experience though.
This is a real problem. You set up a cloud service on Friday. You believe it should cost about $10 per day. On Monday you discover it already cost $500 so you switch it off immediately. Oops. Still, only $500 not a big problem right?
And then on Tuesday the billing software explains that ah, there's $1800 extra for that service, we calculate it eventually but we don't promise it'll happen immediately. Now you're $2300 down. And this can continue for several days at big cloud providers because "eventually" is apparently good enough.
Do not rely on any "free" google product you aren't willing to pay for.