Comment by verst | Hacker News Reader

Comment by verst | Hacker News Reader

Parent Full thread

Once upon a time I was one of 3 people globally that was the entirety of the GCP support team :)

You should have seen the extent to which the rest of the team and I tried to resolve customer issues.

Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it: The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine). Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes.

Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs.

And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article.

Good times :)

I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.

One thing that AWS Premium Support does very well is make situations like this seem unremarkable. This is the default way you do work. Going above and beyond is solving the issue, the next issue the customer hasn't asked you about, and giving practical guidance on things like cost optimization or availability concerns after weeks of deep diving.

I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.

Is paying extra 3-10% of your already expensive invoice worth the premium support that you _might_ need? Not sure I would consider that customer obsession... more like insurance.

What I am saying is that this obsession was normal to us also. I didn't know any Googlers (whether in support or engineering) who didn't care about customers.

We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in.

I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :)

These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now.

And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer)

Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.

I really appreciate that. Both as a customer, but also as a fellow practitioner.

At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer.

As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me.

As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that.

I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for.

I don't hold this against the support folks at Google.

I hold this against the leadership at Google.

This is their strategy to contain costs or to juice sales.

If it were working, I think their numbers would be very different in the market. So the proof is in the pudding.

Something isn't working at GCP.

I don't think it's the engineers or the teams. It's the leadership and the strategy.

What you are describing unfortunately is due to the external vendor teams managing support -- a business decision as you rightly mentioned.

A lot of support became outsourced and the original support team became more of a tier 3 backline support with the first 2 tiers being handled by vendors. This started for GCP just after I left in mid 2013.

The vendors responsible for support have contracts to meet certain customer satisfaction metrics and time to first response metrics (based on customer tier and ticket priority). The majority of inbound support cases would get auto assigned to the vendor team. Only the top customers would reach Google employees directly.

The vendor and their staff simply have no incentive to troubleshoot an unusual issue - they also cannot have access to the tools that may expedite such troubleshooting. It's all about pattern recognition to quickly resolve a ticket. As many vendors were overeager to escalate support cases to the Googlers it became necessary to impose more requirements on the vendor team to be able to escalate an issue - invalid escalations were being tracked and the vendor would be penalized for the total of these.

I built the internal tool via which GSuite and GCP outages or significant issues would be managed from a support perspective. This had the advantage that vendors didn't need to waste their time (poorly) parroting (with a delay) the status of an outage as related incidents could be (manually) grouped together and then all be updated at once by the current incident manager on the support team - freeing up lots of support folks to work on other support cases.

I definitely agree this vendor-based support model is a frustrating experience for everyone involved. I witnessed that myself after leaving Google. By the way, I'm very surprised GCP tries to upsell you on consulting services based on your support interactions.

Running a support organization for a very technical product with so many unknowns and variables at scale is hard. I wonder who does that well and how they accomplish it (from an operational perspective).

I have no idea how AWS manages support internally. I work at Azure - until last week as a Developer Advocate, now as a software engineer in Azure CTO incubations - I have no idea how our support organization is run or what their quality is. However, from what I hear Azure support is eager to help anyone and everyone. I remember being very confused I got a phone call to inquire whether I needed help within a day or two of signing up for an Azure free trial.

Reply on news.ycombinator.com