Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they don't go through - but not suitable unless you're going to check that payments are made, which the OP seems not to have done. Perhaps GCP notifications should be better (I've experience the similar scenario with AWS and everything got fixed within a day) - but surely its up to the customer to make sure they pay.
> however we went ahead and made a manual payment covering all the outstanding amount + extra
(the entire last paragraph of OPs text has those and more details)
I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete description, no?
Given that OP paid, of what use is your objection about non-paying customers here?
You also omit this important part:
> They ... assured the project will not be suspended.
That's fair enough - somehow I completely misread/understood that part of the post.
But my main point was that you should never be using credit cards for services that you rely on - particularly when billing/payment is automated and hidden from you. Credit card payments and checks fail regularly (10%+ for recurring payment charges) for legitimate (you don't have funds) and less legitimate reasons (a banks AI system thinking the charge looks fraudulent due to something outside of your account etc). Vendors are likely to stop the service as soon as they can get away with so they don't loose money.
100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've delayed any action on your account until such-and-such date but let us know if you need more time".
I've been a Linode customer for several years, paying about $100/month. Just a guy running some small things, hardly a big customer.
If Linode can do it, then surely Google can. Yes, you will probably have the occasional non-payment, but is kicking your people off your platform for a simple administrative mistake/mixup that much better? Accepting the occasional non-payment is better than kicking off people at the drop of a hat. I've since paid several more $100/month to Linode and everyone won in this situation: Linode because they kept a customer who will keep paying, and me because I could get my stuff sorted without having to worry about having my services cancelled.
Google's policy is simply short-sighted on every level, and harms Google's business interests too. The reason they can do things like this is because they have money to spare. Every business that's not swimming in cash like Scrooge McDuck treats their customers like this; the "big tech" company are the exceptions.
No, because Linode is using humans to interact with you. Google will not humans in those situation, their mantra is to automate everything so they can scale up with minimal human effort, even when it doesn't make sense to scale up that way.
Fortunately they have customer support, and 5 minutes after emailing them they resumed my services again.
On GCP you're basically SOL
Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.
I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trusting it will because you think all the signs say they will. I do think this is solid advice for the production systems as well. All the signs can point to yes, but making extra assurances to avoid the bad path is valid for something critical like your production systems (or your life).
you mean like an email literally saying they acknowledge the situation and won't suspend your account?
I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).
Regularly XYZ does not happen, or not exactly in the way you agreed. They cancel your broadband too late, or your internet starts a few weeks too late, etc. You can try to get compensation, or even sue, but some of the damage has already been done at that point. While this okay for temporarily overpaying broadband, this is probably not the case for a company's main production system, where the risk is very high. The company might not even survive to sue.
Most places outsource their support to global teams don't have access. If you have an AM , in many places, they receive the same notice you receive, also they will personally send it and address it to you with a reminder. They also have a lot more sway as the sales side of every company has the power so they can get around all the process issues.
As the other commenter posted above, if you can have an account manager where you have critical services, it will provide you safety at the cost of a higher spend or minimum spending amounts.
Edit: Had about $1mm in GCP spend and couldn't open a ticket to get windows running properly. It took a week to find the AM and a day after he had it fixed.
I’m pointing to this as evidence https://cloud.google.com/customers
I feel like a lot of the problems that people run into with GCP are avoided with two main components.
Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about the future (details: https://cloud.google.com/apis/docs/resources/enterprise-apis) and secondly you need to buy actual customer support as an add on if you want to avoid the “I had to hit the front page of HN to actually speak to someone” scenario https://cloud.google.com/support
the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you.
There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promise edition" isn't much of an assurance.
I'd suspect most of those customers have rapid migration plans written up. For a reason.
I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".
Not sure what to say in response here…
I mean, it's an error.
Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.
So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.
In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.
That is your fault, and your customers have every right to blame you. You are responsible for the due diligence on which providers you use.
Blaming a 3rd party might absolve you of some of the blame, but your still not providing the service you agreed to provide.
And before you try to suggest anything else, the condition here was "having customers you can't let down."
You have customers you can't let down. That's the context.
That's over $100,000 a year, at that point having a Key Account Manger (or Success Manager or whatever) to talk to should be the expectation.
At other companies where we've had an active and involved account rep literally just shot off one email and got back a "no worries, I put a block on the account so it can't be suspended without my approval".
I know in the past I've had personal accounts go a couple months in the red when a card expired and I wasn't staying on top of the emails.
These sorts of things are really only an issue with one specific provider.
I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc?
Whatever anyone else says in this thread, that's what you need to determine for yourself. Whatever company, what is the support channel like, how responsive is it?
If you don't know? If you can't get immediate emergency response and support, and I mean immediate, then why are you even hosting there?
AWS, Azure and GC are all on the same level. Any of them gives you „the best“ technology.
This is recent topic and you can see that so many people prefer GKE. That's not my opinion.
Sure, any cloud works good enough.
True, but unfortunately they are literally the worst at customer service.
This is total standard in AWS and the ticket for this on GCP has been open for many years now.
That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web.
But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.
https://cloud.google.com/storage/docs/lifecycle#numberofnewe...
> GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)
You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.
Note also how the project is hosted under the GoogleCloudPlatform GH org. To me, when I see GCP projects that are set up like this (in the GH org but disclaimed as "not official" in the GCP docs), this suggests that Google engineers built it knowing it's a pain point; and those engineers will support it to the best of their ability in the capacity of being maintainers of this open-source project; but Google as a company don't want to officially support it (yet), and so your GCP support contract won't get you any business-level support for it.
It's sort of like how, in Postgres, there is code which is maintained by the Postgres maintainers, but which lives under contrib/ as an extension. It's essentially a lability-waiver for that component.
---
† I do have a guess as to why Google do things this way. At least where dev tools are concerned, Google seems to eschew the usual distributed-systems architecture for long-running jobs (of having a thin API client binary that submits jobs to a cloud-side control-plane daemon, which then drives the job forward, and which can then be polled/subscribed for job status by said client.) Rather, Google seemingly have a philosophy of designing local fat clients that reach into the cloud to drive backend processes as the "control node" for those processes. The Cloud Dataflow (⇒ Apache Beam) architecture is designed this way, for example. I believe it's the reason that the Google Cloud SDK ships with so many binaries — there are a lot of fat clients in there that actually drive logic, rather than just sending messages to daemons that drive the logic.
And, presuming developers are issued good workstations, I can see the advantages of this architecture. A local control node synchronously knows its own status, rather than having to poll for it; a local control node can use local resources (like how Cloud Dataflow can consume and produce local files on the ends of the pipeline with the same streaming efficiency as a regular CLI text-processing shell command); and an operation started by mistake, with local control, can be cancelled by just ctrl+c-ing the control process.
Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed, and therefore that said dev is available in case anything goes wrong. (I've personally dropped [async daemon-driven] Continuous Deployment, in favor of this sort of "synchronous dev-workstation-driven deployment.")
I wish someone at Google would write up a paper on this philosophy; it's pretty clearly implicit in a lot of their work, but I've never seen it mentioned explicitly anywhere. (Maybe it's just one dev-tools lead who has influenced a lot of these projects, doing what they think is "obvious"?)
That is object specific though, so probably won't work unless I use exactly one image.
> derefr 3 hours ago | parent | context | flag | on: Tell HN: Google Cloud suspended our production pro...
> But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.
https://cloud.google.com/storage/docs/lifecycle#numberofnewe...
> > GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion) > > You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.
No, it does not "exist". It's either builtin and then it exists, or it doesn't. What you linked is a way for me to build it myself. And I found this AND this solution is even linked in the years old Google ticket. And guess what, they didn't build it. On AWS no problem. As I said: years behind.
Sure, I can setup my own cron job (or here cloud run function / github action). But that's not what I expect from a leading cloud vendor. This is not a niche feature!
Please don't defend it, Google doesn't deserve it. Credits to whoever build the 3rd party solution, but Google really failed here.
> Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed
Just no.
Let's face it: Google is just too incompetent to do it. Not the developers there, but the company as a whole in the way it is organized. Even IF it were as you said and it would be a philosophy, they could say that close the ticket, but it's still open.
I can list you dozens of similar things with GCP and related Google services.
We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying systems are extremely stable and almost never cause us any issues (compute engine, GKE, GCS, Memorystore, Firebase, CloudSQL, etc). Just thought casual readers of this thread could use a counterpoint.
I can speak to this because a few months ago when I started a project, I went with Firebase. At the time the rationale was, I need something more complex than Netlify/Vercel, but not AWS that will suck up all my time figuring out IAM roles and other dumb stuff.
It helped speed things up in the prototyping stage, but I will likely move my infra over to AWS soon.
If I had an "established relationship with a sales / account team" then yeah, perhaps I would have been informed. Unfortunately my experience is that unless you're spending above some relatively high threshold (>$10k/mo? >$100k/mo? >$1m/mo?) then Google don't prioritise your business.
Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?
From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineering manager at Google Cloud up until earlier this year, given the context of the discussion and your stance on it.
Your parent said OP did what they were supposed to and paid, but got cut off anyhow -- to which you replied "You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't."
I'm not sure how to interpret that other than you saying OP is probably not being truthful in their account of the situation.
However, I remember a case when I was the culprit - spam was being sent from one of my servers (one of the users had a common username and password, something I should have never allowed) and Hetzner notified me immediately, but they didn't block the server in any way (something I'd expect). They gave me some window for explanation/solving the problem. I was very happy with how they handled the issue - but of course others may have had different experiences.
I suppose billing works extremely well for most GCP customers as well.
As for me, I hosted a side project on Hetzner a few years back for some European presence. IIRC I used PayPal for payment and they didn't support PayPal recurring payment, so I had to settle an invoice each month. Used them for about two years without problem, then missed payment for two months during a chaotic period when I didn't notice their monthly invoice emails. I got exactly one email reminder about missed payment titled "Reminder" (yes, that's all, I got all their correspondence in front of me right now) before the final "Cancellation of Contract" email and termination of my account. At that point, politely pleading with them to reinstate the account was useless, I only got a threatening letter back demanding a wire transfer of the ~€60 I owed within three business days or risk being sent to collections.
Was I in the wrong? Sure. Were they friendly or patient? No.
I have my Hetzner account set up so they pull the money automatically (through SEPA Direct Debit). Not sure how it works with credit cards, but it's not like pull payments are not possible with Hetzner.
Ultimately its as bad as car dealerships to require a special connection to get basic tasks completed.
At certain scale (in my case, 5 figures/mo spend with a vendor and up), paying upfront is not something finance usually likes and NET-60/NET-90 is the norm.
Otherwise don’t take their money and advertise that everything will be easier and more reliable
That’s why NET30/90/etc exist — without an incentive to pay sooner, invoices will just sit around on the receiver’s books until it’s convenient for them to pay (e.g. to make their financials for a particular quarter look good.)
Invoices under contract law are like PayPal’s arbitration process: “always favours the buyer.” A seller would need an explicit pre-existing contract clause stating a due date for an invoice, in order for a due date declared on an invoice to have any force; by default, it doesn’t. And because invoice billing is something companies tend to offer mostly to companies they’re trying to stay on the good side of, they don’t tend to sit them down for a binding contract negotiation when doing so.
I don't think OP's post is about payment options but about problem handling.