Disclosure: I work at Google on Google Cloud.
You guys need to understand the scale of distrust this issue raises.
If you want to fix it, give up on putting lawyers and spin doctors up front.
Instead, give this disastrous issue the response it deserves and get the CEO and Chairman of google to make s loud, clear and focused apology and say loudly that we do not delete your entire computer systems over account payment issues until defcon 3 is reached.
Why the F would I build my SAAS business on google cloud without that message being yelled repeatedly and loudly and clear from the top level of the company, as much so Google understands it as the market understands it.
And by the way, I pay ALL my cloud accounts on credit card. I note carefully that at google this means I have a higher chance of having my entire computer systems deleted than if I use some other payment mechanism. Your policy here appears to me to be using systems deletion add leverage to get me off credit card payment. All of which says to me that google still just doesn’t get it.
Google is bad, but Amazon is really just as bad. They both have absurd policies with no recourse, that work in the 80% case and fail very badly in the 20% left.
[0] https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amaz...
Amazon has a big issue in that all their services are linked to a single Amazon account, and if that account is blocked you can be prevented from accessing entirely unrelated services. In that case, maybe it was ok to suspend a brand new AWS account until accounts could be verified, but the linked accounts that were also blocked should have been looked at manually because of their volume.
I think the main problem is that fraud is real problem and if you don't want your account to ever be considered fraudulent, then you should take ample steps to make sure that it is not.
There are countless stories of people being careless with their keys causing tons of crypto-mining servers to launch.
There is no win for cloud providers here. Either you get PR smashed for charging careless customers, pay tons of money to wave off their mistake, or get ridiculed for trying to block customers whom you think have been hacked.
If you can’t provide that level of support you don’t deserve my business.
---------
Greetings from Amazon Web Services,
We were unable to validate important details about your Amazon Web Services (AWS) Account. Your AWS account has been suspended. While your account is suspended, you will not be able to log in to the AWS portal or access AWS resources.
If you do not respond to us within 3 days of this email (by Saturday, April 28, 2018), your AWS Account, and any data will be terminated.
At your earliest convenience, please send us a copy of a current bill (utility bill, phone bill or similar), showing your name and address. Please provide for both the credit card holder and account holder if different.
IMPORTANT: When you send us your information, please include your email address, so we are able to process your fax as soon as possible. We can be reached via secure fax line at +1-206-922-5831 and 866-437-9072.
Once you have sent the requested documents via fax, please contact us at aws-verification@amazon.com and include the following details:
- The billing address and phone number of the credit card on the AWS account - The billing phone number of the credit card on the AWS account - Business name and phone number (if applicable) - The URL for your website (if applicable) - A contact phone number where you can be reached should we need additional information - Potential business/personal expectations for using Amazon Web Services
If you have any additional information you feel will expedite this process, please feel free to include that as well.
We apologize for any inconvenience this may have caused you and appreciate your patience with our security measures.
Thank you for your immediate attention to this matter.
Sincerely,
Amazon Web Services Team
And in general, I find it difficult to buy your "I don't own a fax machine argument". I don't own a photocopier, but that's a small blocker, no?
Actually, you did provide some context, which is that it was a "demo" account on a personal credit card. This, at the very least, implies much less spending than in the blog post that kicked this all off.
More importantly, what's missing is the human/support response and attitude from AWS, which is the OC's primary concern.
Anecdotally, from the comments I've seen on HN, AWS customers (at least beyond a certain size) seem generally pleased with the availability of human judgment during support issues, especially so when paying for that support.
While this might not strictly apply to some other particular situation, my experience would lead me not to ever trust Google when they say "do this seemingly logical thing because it will make you safer".
Furthermore, this response could be construed as reinforcing the OC's precise concern regarding Google's attitude:
> that you are probably wrong.
As I mentioned above, in this case, WE were absolutely wrong, and the blog post details all the ways we're fixing it.
You did admit error and apologize in the blog post (perhaps not explicitly or early enough in the text for some people, but that's a separate issue), and you specified how you were going to fix it. It may take time for the reputational effects of those fixes to permeate.
That's not the issue.
The issue is that, here, you have, effectively, said, "but the customer could have..".
Even the implication that the customer might share in the blame, no matter how objectively true it may be, takes credibility away from the sincerity of an apology and can easily be seen as evidence of the persistence of same, old attitude. Given that the statement happened after the apology, further time is unlikely to make a difference.
1) This was, in NO WAY, the customer's fault. At all. Not even a little bit. They did stuff that was totally normal for a customer to do.
To remedy us not accidentally flagging totally normal stuff as fraud again, we talked about 7 steps we were taking (that was the first 7 steps in the blog).
2) IF, and only IF, you'd like to take ADDITIONAL steps to avoid us accidentally flagging you, we also offered three steps (at the end of the article). However, these were totally optional.
Hope that helps.
Getting warmer.. but:
Even here, you're saying that the customer has (or could have) some role in preventing your error (accidental flagging). This is merely an issue with the wording (here, not necessarily in the OP blog).
The "protect your account" wording of the blog post itself seems good, as do the specific reasons for each additional step.
My only suggestions would be to add the concept of urgency in reachability to the phone number, as time sensitivity seems to be important to many (but not all) customers, and to substitute something for "don't miss important alerts" (which, again, could be read to imply blaming the customer) that has a more positive wording, that expresses that all the relevant parties would then be empowered to have visibility into those alerts (rather than avoiding the transgression of missing them).
See also https://news.ycombinator.com/item?id=17567900 (which I have no idea as to the veracity of but figured I'd call your attention to)
> Hope that helps.
It does (at least IMO), and I hope you take my feedback in the contstructive spirit in which it is intended. (I don't personally have a strong opinion either way, as I lack enough direct experience).
In our case, we're adding a second human in the loop (we already had one) to double check this. Customers can also add their own human to reach out to by connecting to a sales person (it's free and extremely low pressure!)
> Established GCP customers...
> Online customers with established payment history...
What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
> ...review for flagged fraud accounts...
> ...event that fraud or account compromise activity is detected...
You gotta stop using the f-bomb like that. This piece is supposed to be a follow on to a story of a customer who apparently was conducting completely non-fraudulent activity within their account. Your copy should, like your culture, assume "innocent until proven guilty". Saying something is "flagged for fraud" in a PR piece breaks that rule.
> We will provide ways for customers to pre-verify their accounts with us if they desire...
Sounds very passive-aggressively optional. Nor does it tell me what I actually gain here, or even a hint of what might be involved for verification or how potentially simple or easy it may or may not be - do I need my counsel to post via registered service on company letterhead confirmation of my organisation's official trading details, or do I click an link in an email?
> There are also steps that customers can take to help us protect their accounts including...
The word "protect" is suitable if you were a bank, and you wanted to encourage customer behaviour to prevent them being victims to fraudsters. But you're not, so really this language is suggesting customers need to "protect" themselves from your algorithms and automated systems, which is kind of inappropriate to be saying.
This is accurate.
>> Online customers with established payment history...
> What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
They can't tell you. Bad actors will even binary search the parameters of your fraud detection (source: I worked in fraud detection). Minimizing the information feedback loop to the baddies is super important to avoid becoming a loss function they can optimize against. This is true of every fraud department.
Abuse detection is cat-and-mouse. Say as a provider I track your true IP through proxies by embedding some flash that calls home. You figure it out and block flash or do some clever sandboxing. I work with proxy providers to build blacklists. You somehow get the blacklist and just avoid all proxies on it. Etc etc.
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
One saying I’m sorry isn’t an amends. Neither is offering a contrition-free remedy for past bad acts. Amends requires both contrition and a remedy.
For example, inserting the following (or anything similar) after the first sentence would have demonstrated some contrition: “We have truly been humbled by our prior lack of insight, and we are very sorry for what happened to any customers affected in any way by our shortcomings in this area. We realize we need to regain your trust and with that in mind, have thoughtfully examined all the feedback we received.”
Learning you all actually thought it through and consciously chose to save face does not inspire confidence. Nor does the underlying subtext that customers big enough to pay by check and get an account rep get a free pass, while little bootstrapped guys like me can still have the hammer dropped on them, albeit with slightly more scrutiny than last week.
We screwed up. We are really sorry. This was a confluence of unexpected events that slipped through our systems (which included both humans and algorithms). We know it will take a long time to rebuild the trust we've burned, and we are committed to doing it.
In addition, as I said earlier, the options to ensure that you have additional protection (WHICH YOU SHOULDN'T HAVE TO DO, I'M JUST OFFERING IT) are at no cost:
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards (this is much lower than you would generally think)
While you're right people will want an actual plan on how things are going to change, they also want a genuine expression of contrition. As hguhghuff noted, deleting someone's business is a serious, serious issue.
The _only_ admissions of guilt or wrongdoing was were these lines:
"Here’s what we are doing to be as good as our word, and provide a more careful, accurate, thoughtful and empathetic account management experience for our GCP customers."
"We sincerely apologize to all our customers who’ve been concerned or had to go through a service reinstatement. Please keep the feedback coming, we’ll work to continue to earn your trust every day."
Which, if you yourself were scared that your livelihood could be wiped away, would you think that was a sufficient apology?
It doesn't help that Google tends to lack a human touch when it comes to its image. But here, now, you all have an opportunity to change that. I know it sounds "politics" or "public relations" or whatever, but all we need to know is that you fully understand the fear GCP customers felt and what that means for us.
I could still see everything and all my systems continued running.
Who's we? Did you have a part in writing this? What does "not make it sound too lawyerly" even mean? This press release screams "we made a mistake but we're going to use paragraph after paragraph to ensure we don't actually say that". I have a hard time believing that the people who make up the Google Cloud team don't recognize this.