Tiered support is an anti-pattern (2013)
peter.gillardmoss.me.uk
peter.gillardmoss.me.uk
Another support anti-pattern is an untrained offshore support team whose entire mission seems to be to close the case and move on. There is little in life more enraging than a support rep who refuses to engage logical brain and escalate based on clear evidence of a technology issue.
At my most successful startup, we had an extremely smart guy in the support seat; he was eager to learn from the dev team how to triage, and so we'd get bug reports complete with network tab screenshots, api paths which had caused an error, etc. He could easily have been a skilled programmer had he chosen to pursue it. When we were acquired, he was merged into a larger team that had much less autonomy, and his morale tanked over time.
The zendesk ("here are articles that might be appropriate") approach sucks too, as do chatbots.
Perhaps fear of hallucinations is holding people back, but it seems like an LLM based product could take over the "keep it cheap" support vertical. 1) Have people file a report in their own words; 2) synthesize an answer for them, but only if the LLM identifies with high confidence their report as being a known problem that has an answer; 3) cluster reports from users based on salient features and escalate new clusters to the tech team; 4) and if the company can afford it, route everything else to a human team who can categorize, click a few buttons, and send the LLM back with an answer, or escalate.
In my company, we have free support forums, email support for the customers who jump through some hoops (to make it slightly harder than searching the forums), and phone support for select clients. But the "power users" in the forum are just our internal customer support staff operating a handful of sock puppet accounts, and it's the same people who answer phones and email.
The forums are a great searchable historical record. The email support is versatile and scalable. And the phone support is a non-scalable value-add that makes our top tier clients feel very special.
We were 10x slower than the real support techs at resolving cases so this didn't improve support by all the typical pinhead boss metrics. Customers tended to think it was pretty awesome they were talking to a dev regardless of whether we solved their issue. The real benefit of the experience though was FOR US, and was transformative. I credit a lot of my better instincts regarding product development to having that experience so early in my career. For example nothing makes you more sensitive to performance issues than sitting on the phone just waiting and talking to some random person for two minutes every time anyone tries to load a particular screen - you get back to your "real" job and the first thing you do is go into the bug database and start asking where the hell is that bug and when are we going to fix it.
I own my own business now and I have never left, and will never leave, the email support alias, I will read every one of those emails I can until I die (and respond to the best of my ability as well).
Later I worked on the team for a web-based product which had feedback fields all over the product. It took everything anybody wrote in those fields and piped them, unfiltered, into a screensaver that was installed on every computer used by the development team. When I was new to the team my manager told me "You just have to accept that you're going to see a lot of profanity on people's screens, that's how it works here." LOL! He wasn't lying. But you'd get exposed to dozens of random bits of feedback about the product over the course of each workweek. Tons of bugs and user pain points were discovered this way. You'd be in the middle of a 1:1 and some expletive-laden complaint about the feature you designed that went live three weeks ago would pop up, well OK, guess we're going to have to deal with that before the meeting is finished! It was wild.
I remain of the view that relentlessly and aggressively exposing your entire product team to tons of customer feedback through any avenue you can is the right choice. Got a popular "water cooler" or #random/#general type of Slack channel? Start piping a bit of customer feedback into it. You can email Slack channels! Do it today.
Wrote a whole blog post about this: https://www.mooreds.com/wordpress/archives/3451
Many people have a drive to solve their problems themselves, and if you make content available to google, they'll find it. It's a fantastic way to self-empower your users and customers.
We even have an internal slack channel where I'll post questions and answers from the other support channels (email, slack) with a view to making them generic and turning them into forum posts. I don't get to that task often enough, but it is sure fun to do when I have time.
I also wonder if the incentives are a little bad with this setup. Product devs' crappy code turn into tricky puzzles for our support team. Legend has it, RAD Game Tools used to have just 1-2 devs on each tool, and you'd do the support for your tool. Feels like this would pretty strongly encourage you to make your stuff good. And you'd have everything in your head too. I wish I knew of more similar setups in the web world.
I strongly recommend having the PM be the highest point of escalation in the support chain. It really incentives them to make a quality product.
* our public facing docs
* our public repos
* our youtube channel
* our forum
and spit out answers. It's not right all the time, but it is most of the time and, crucially, it shares not just an answer but a link to the docs that led it to that answer. That makes it easier to fact check the LLM.
We've found it very useful for scaling support and making that corpus of knowledge available to folks both outside and inside the company.
I wouldn't trust the LLM to keep your secrets. There are countless examples of people convincing LLMs to say things they shouldn't. On the other hand, I wouldn't trust an LLM to spit out your source code without hallucinating a bunch of fake functions either, so maybe it balances out.
It's genuinely strange. People will even do it for banks and phone network companies.
As a company, all you need to do is have XP tiers for posting and someone will be willing to dedicate their life to becoming a Finco Bank Guru and running the forum for free. Usually they have a rather "Boomer" energy, so I get the feeling it's retired people who think the parish council/HOA doesn't meet often enough.
On the other, a lot of these are people willing to help someone else for free and make their knowledge practically public in the process. Seems aligned with a hacker/wikipedia/oss ethos in many ways.
Sometimes the right answer is just the right answer.
Helping other people can be deeply satisfying, even without payment or karma points. Maybe you are already doing it yourself? Otherwise I can only recommend you to try it some time. ;)
I started a support forum for a video game once. I wasn't a boomer or retired. I just liked the game, had the ability to help, and the time. I guess some people just like being helpful. Certainly the company who made the game profited off of my free labor but I wasn't looking for money (to their credit they did offer to send me to E3 one year at a time when tickets weren't being offered to the public).
Your comment isn't the worst, but the word is usually used as a placeholder for some bigoted stereotype against someone retired. Disclosure: I'm not a boomer and the word is not used much in New Zealand.
(Also, like, there are clear biological differences by age, which are sometimes relevant, while the concept of race is a mistake. It seems understandable that people would be less concerned by ageism than by racism. Also, people who live long enough will experience the different age brackets themselves, which puts a natural limiting pressure on how badly people are likely to treat others based on their age. I imagine people would be less likely to hate people of “race” M if they knew that in a few years they themselves would be of “race” M, while the people that currently are would not be.)
Why reach for a vague zeitgeist term that everyone knows, but no one has a precise definition for, instead of taking the time to use a specific word?
We could also talk about how it’s usually not just used as an ageist slur but to describe a state of mind - if you ask the kids there’s plenty of old people who aren’t boomers - but saying it’s as offensive as being racist is completely inappropriate.
To sort of quote the comedian John Mulaney about the n-word, if one of the words is one you can’t say and have to use “the n-word” to describe it, that’s the bad word.
That "while lynching" part is doing all the work here.
I don't recognize what the article describes at all. There are tons of questions answered by L1 which are not even issues with our software. And us devs certainly feel responsible for what we produce, and we still talk to our customers when needed.
So I think it's more a culture and possibly management problem than simply layering.
To this day that is still my issue when call ISP or telco support. Why am I trying to have some 19 year old, with access to nothing but my billing information, trying to solve the issue of my router not connecting?
The type of first level support most technical companies offers could be solved by better monitoring, better self-service and well written documentation.
But as someone developing a product, sometimes when you’re closer to your users you find that many users get told to RTFM on a particular feature and maybe the actual problem is that the feature is unintuitive.
That said, I agree with the sentiment that as devs one shouldn't be far removed from customers. For example I do take time every now and then just looking at support working if they've got a customer showing an issue or similar. Or just hang around to hear what they complain about.
This is of course much more difficult if support is in another building and not just down the hall.
It really helps me understand what they’re doing better. It short circuits a lot of bureaucracy. And it’s great when you can push out a change for someone an hour after they wanted it for simple things. So they know that you can make changes fast to address their problems.
And it’s just nice to get to know the users and be friendly. We get to chatting. And it just makes everyone more chill knowing that the developers care.
Understandably being able to call a developer should be expensive, but when you tier it, then customers feel ripped of, because they have to buy something they don't need, to get that one thing they really do need and are willing to pay for. In our case we where willing to pay some fixed price per year for access to phone support, and then an hourly rate if we called, but that wasn't the pricing structure.
In fairness the support structure and pricing from Elastic has improved since then, but the issue is the same, if you only need one thing from the higher tiers, then the pricing doesn't make much sense to you as a customer.
But it is priced accordingly not because it includes all those other things, but because it is dang expensive to provide 24/7 on-call technical support.
> In our case we where willing to pay some fixed price per year for access to phone support, and then an hourly rate if we called, but that wasn't the pricing structure.
Sure, I get it. Elastic wasn't selling what you want and that stinks. But there's a cost to a company of offering everything ala carte. It makes support more complex (because support staff have to be trained on snowflake clients), makes it harder to sell online (because customers get confused by options and bail) and harder to sell via a sales team (again, complexity).
That said, I bet there are Elasticsearch consulting companies who would be thrilled to quote you for on-demand on-call support (googling for "elasticsearch oncall consultant" turns up a few). I'd be interested to hear how their pricing compares.
I don't doubt your expertise in technology. But there are so few companies that can actually do what you describe that it's not a viable market for vendors.
The first problem is that the kind of staff you need when things go really bad are (a) really hard to find and (b) are looking for permanent employment. That's why support vendors want term contracts--it's the only way to make the economics work.
The second problem: While you say you just want to call your support vendor when something really bad happens, that's actually the most expensive and least satisfying approach for most users, even very expert ones. Far better to engage before there's a failure, e.g., at design time or during regular health checks. Data base upgrade planning is case in point where early engagement pays off in spades. That includes things as simple as having an expert on-call when the upgrade happens.
YMMV. That's at least been my experience from 18 years of supporting open source database software.
Then you don’t have the problem of paying for things you think you don’t need.
Your problem is that you want the highest value (direct on-call team with actual skills is expensive AF) for the budget price - as someone who helped design support tiers this is super common but obviously not economically viable.
edit: in any case on that price level the price structure should be more flexible I agree
So if the tier with which provides on-call is $100.000 and the tier below that is $75.000, then the cost appears as $25.000 to me as a customer. That not true, when you dig into it, some of the cost is covered by "unused" benefits of the tiers below, so that one benefit on it's own might be closer to $50.000. If you truly don't need the stuff below, then you'd want more flexibility in the price structure and be able to spend the $50.000, rather than the $100.000. Instead the provider now ends up with no sale.
> direct on-call team with actual skills is expensive AF
More than most realise. My issue is that there are occasions where you don't mind paying for that service, but you're not allowed to, because the tiers are designed so that it becomes more expensive than it should be. I've seen a number of support contracts, where they only tier that actual provides much value is the one where you can call an expert, but that's hidden behind "Call Us" or "Super Premium Platinum" priced such that only governments and VC funded companies can/will pay for it.
It’s like saying every person in a hospital should directly go to the operating theater. The 10% people needing urgent care would like it but it’s a very inefficient way to use the time of the staff. That’s why you have a reception (Level 1) that’s triaging cases and checking if you even need a doctor.
This is the real human psychology trap. By definition, L1 doesn't understand anything it can't handle.
Unfortunately, at many companies L1 is also yelled at for escalating too many cases.
Consequently, the internalized message for L1 to not get yelled at is "Juggle the ticket you don't understand convincingly, until the customer abandons interaction from frustration."
A few anti-patterns I've seen in most support shops:
1) Bad KPIs / no post-hoc sampling and review. Without regular, random, higher-level sampling and classification of how tickets were handled, there's no signal that support is technically botching a large % of incoming tickets.
2) Disincentives to escalation, even when it's a true positive escalation that should be. IMHO, support should be incentivized to escalate true positives.
3) Failure to upskill support agents and build mix-of-expertise within the support function.
4) To 3, failure to pay support for the value they're creating. In a sane world, a few of your highest skilled support people will be paid like developers, because they are.
If your case is an emergency, you can cut through all the queues and go straight to the operating table.
Whereas most L1 support systems are either non-existent; a FAQ loop that gets you nowhere; or a Dialogflow/GPT chatbot with 0 reasoning skills, no technical knowledge, and limited domain knowledge (really just a more convoluted version of the FAQ loop).
If you're lucky, after some digging you might find a hidden link that lets you talk with a real human being.
These systems are deliberately set up to massively increase the effort required to get to talk to a technician, as those are a limited resource and companies don't want to spend money on stuff that would actually improve their internal processes.
And if hospitals worked like this then the majority of people showing up for any problem whatsoever would die before receiving care. This is not how triaging works.
Hint: It's the advertisers.
Regarding the former, my company pays for google workspace and an agent happily answered our questions about the recent less secure apps password phasing out process.
You cannot hire someone directly as L2, as there is a learning curve for company- and product-specific issues, and a freshly hired "L2" would have to learn. However, they are shielded from the most frequent issues by a hoard of L1s and thus cannot learn about the typical problem areas or troubleshooting approaches.
So they have to either graduate from L1s or accept a downgrade from being a developer.
We've managed to do that by recruiting from our customers. During support cases and such we'll notice someone is more technical, and then at some point approach them with an offer.
This way they already know our software well and can get fairly straight into a L2 position. Of course in the start they'll get some more L1 stuff and such, but it significantly reduces the learning curve.
Customers aren't too miffed as the alternative would almost always be that they'd go to a competitor.
In medical s/w development, L2 can be nurses or doctors, they might be attracted to this position due to relatively low load (compared to clinical medicine), flexible schedule etc.
I don't think there is / possible a single answer to how to find these kinds of specialists.
I dont think they would have to come from L1 from the same company / project though. Experience does allow one to learn faster, communicate more efficiently, etc.
I think it's also good practice to have devs rotates on some small shifts of L1/2 so they don't get out of touch with the reality of maintaining their stack.
Once you get beyond having a single dev team, a layer of support is necessary: you can't expect consumers to know which of your dev teams they ought to be talking to.
My employer has an amazing support team, who are effective champions for the user when interacting with development teams. They're fantastic.
That said, most of the issues I see from support are straightforwardly bugs, even if they're not normally easy ones to fix. But when they have information they think is worth sharing, it's always been worth my time to listen.
The obvious solution to this is a distributed team. There can still be some holes (e.g. Jan 1 is, afaik, pretty universally celebrated), but having worked at companies that had a team in the US, Europe and Asia was...really nice with respect to this.
The only valuable metrics, in my mind, are "Did we actually solve the issue for the customer" and "Have we effectively prevented this from being an issue in the future".
https://en.wikipedia.org/wiki/.uk#Active
Which is pretty cool, considering that when I registered a domain under ".us" in 1993, I used our street address as the leaf node, and I've seen a few random "StonesFamily.org" or similar; it's nice for UK persons to stake a claim here. How long has it been active? How do they resolve disputes? Do they permit subdomains for family groups?
I think pretty much forever at this point, the only addition to .uk in recent memory is the ability to register .uk TLD itself (before it was all second-level only).
I think it might have been around the 2000's when they introduced me.uk.
> How do they resolve disputes?
AFAIK me.uk falls back to the general rule of "first-come first-serve". The only real requirement for me.uk is that it has to be registered in the name of a natural person and not an agent, trustee, proxy or representative. Any transfer or renewal has to continue to adhere to this rule.
There is no UK nationality or residency requirement for me.uk either AFAIK.
Disputes fall back to Nominet's standard Dispute Resolution Service.
> Do they permit subdomains for family groups?
Not AFAIK. Well, of course you can register the base me.uk and DIY DNS subdomains.
The crucial issue though is to also have competent CS working with the dev team. It's not easy to have two separate teams share info and collaborate efficiently, it requires skills, and these skills need to be on both sides of the table.
That makes it harder to retrofit these setups in an existing org when these aspect where completely ignored at hiring time for instance.
Today, after some practical experience, it looks like some conclusions this article makes are unwarranted. For instance, the idea that the development team should be directly involved in addressing customer issues, if implemented, often results in development plans being ruined, QA doing a lot of busy work, release schedule going down the drain.
Ultimately, the problem is the lack of expertise necessary to triage and to schedule development necessary to address the problems the customer is facing. Low-tier customer support is usually a low-paid position that doesn't require expertise in the product being supported, or anything at all really. Is there a solution that doesn't involve rising the requirements for the support staff (and rising the pay)? -- I don't know. Doesn't look like it. It seems like asking for having your cake and eating it too.
Thinking about it in degrees of separation is fine from an abstract view but collides with reality