Comment by ddebernardy | Hacker News Reader

Comment by ddebernardy | Hacker News Reader

Parent Full thread

Similar story here.

In terms of scalability, my takeaway has been this: in every company I've worked since, I've fought tooth and nail to make sure that the engineers are the ones handling the level 2 support and up.

Support is invariably happy because they only need to triage and process the really dumb questions. They offload the time consuming stuff and end up cutting their budget needs.

The engineers, on the other hand, invariably get extremely annoyed by the deluge of UX and/or recurring questions that swamp them. Oftentimes it's to the point where they spend so much time on them that they've no time to build new features anymore. And then, magically, they aggressively prioritize and fix the more time consuming/support intensive issues to get them off of their hands.

Everyone is better off. Clients are happier; support is happier; marketing and sales are happier; finance is happier; and so are the engineers. The latter are less happy initially, but they rapidly get a better feel of (and end up appreciating) how their work impacts end-users.

This is great in some cases, but can also be really dangerous.

It's structuring the entire company around the selection bias of people who complain about your product, and not focusing on the hypothetical majority of users who never have issues with it, who may outnumber the group of complainers by 10x or 100x.

At that point the right thing to do is to pay way less attention to things that represent a support load, and iterate on already working features used by the 10x or 100x.

Sometimes it's the best thing to do, sometimes it's a horrible idea. It's not some no-brainer that software development in general should be structured like this.

Agreed in principle, but not in practice.

I've yet to join a company that had a marketing department on top of things enough to pinpoint what the hypothetical majority of users who never had issues actually want.

(I'm sure there are some out there; but I've yet to find one that will hire me. And for clarity, I usually join companies as part of the marketing team. After a few weeks of getting a feel of what end-users actually want, fixing the existing product niggles turns out for one reason or another - maybe it's just the type of companies I work with - to be the or one of the lowest hanging fruits from an ROI standpoint.)

You wouldn't give this to marketing, but make the decisions data-driven where you try as hard as you can to get a sample of what your average user is doing, and whether they're being hindered by bugs or some features.

This is easiest with hosted SAAS. It's why companies like Google care so little about user feedback. They can e.g. do an A/B test on their UI and find from their data that moving some button makes it easier to find (fewer false clicks), no matter what the minority of users who contact them and swear up & down that the new UI sucks say.

With self-hosted or desktop software this is harder. You might have some phone-home feature that sends aggregate statistics, or simply pro-actively contact a random segment of your userbase. "Hey, you use our product. Does it mostly work for you, or are you being hindered by issues? If so which?".

That's still a huge selection bias (people who care enough to respond to a survey), but still beats the even bigger bias of people filing bug reports.

And I'm saying this as exactly the sort of person who'd be annoyed enough to file a very detailed bug report against some piece of software I use.

Moving the button around the screen is the kind of thing that will drive you to a local optimum and dump you there, with no sense of purpose or direction to a better place.

Direct user feedback shakes you out of that dead-end, and towards a newer, much better dead-end on the path to success. Maybe the button is solving the wrong problem, but users latch onto it as the first thing that comes to mind. You can't find that out unless you interrogate your customers, and most engineers wouldn't do it unless their back is against the wall. Happened to me last week - spent an hour on the phone+hangouts with a customer, and it turned out they needed something other than what they asked (and we didn't have). Happens all the time.

[BIG-CO] employees are just talking to each other, and then A/B testing their way to success along the nearest promotion-worthy metric. The courage, or the necessity, to overcome the aversion of the users is a competitive advantage for a small company.

EDIT: want to add that I am enjoying our conversation here, thanks for joining.

Not sure that google is a good advert for this method. Their products tend to be increibly frustrating to use...

The expectation here is not for engineers to see requests that can be handled by the frontline support team. And when you have a sizable number of people asking for features you don't want to build (for a good or bad reason), you can pretty easily turn that into something that the frontline team (or their automation) can come back with:

"Thank you for your suggestion. This is a common request" and then "but we're not planning to build it for [some reason]" or "we're considering it for the future". Depending on if you want articles about how dumb your reasons are, or articles about how you promised you were working on it three years ago, and it never happened.

Your comment makes me think of one of the saddest tragedies of software development that bit me in the past. Ages ago I used a 'shareware' (thats what it was called back then) FTP client called BulletProof FTP. It was magnificent. It performed its task marvelously well and only had one bug I was aware of that ever caused me any trouble.

Then they kept working on it. With every version, they bolted on some extraneous garbage that made the rest of the application worse and worse. MP3s were popular, so they added searching FTP sites for MP3s... something that I find hard to imagine any human being ever did or wanted to do at any point. I eventually just had to stop using it, they made it far too aggravating. And they never even fixed the bug (after a disconnect, it would use the random port previously used for a data connection to try to establish a command connection and if the reconnect had been quick enough, the server would still be spewing binary data into what the client expected to be a plaintext connection, resulting in random garbage).

I am suspicious of your division: "complainers" and "majority who never has issues".

I agree some people are born complainers, and you should not implement things mentioned by just one user or one especially noisy user. I also agree that support requests shouldn't be the only way one understands users. But I suspect there are very few improvements valuable to people who don't contact support that would never be mentioned by someone who does.

To me the people who contact support with suggestions aren't just the most irritating fraction of your user base. They're also the most dedicated users. The ones who love your product the most. The ones who are doing something you didn't expect because they are an early adopter in a different market. The new users that you'd like to have more of but won't until your product is easier to learn.

Presumably, that's why you still have at least level 1 support: make sure they are able to filter out the noise and get 'real' issues sent to level 2. (aggregation can be part of the filter for complaints/feature requests that aren't bugs)

There are other ways to accomplish it, but the basic idea is to never let product development get too far removed from the actual customers. Large companies can get away with not listening to the customer (for a while at least), smaller ones not so much.

I think improvements for products can be devided into positives (e.g. brings value, new features) and solving frustrations (bugs, slow speed, bad UX, etc). By listening to all user complaints you can almost complete eradicate frustrations - which makes your product fun to use and can massively reduce churn.

I've always found it easy to see the whole system, and how it looks to others. (Ironically,) I can't understand how most engineers don't, and get annoyed instead.

I once spent a couple of weeks working out how to give helpful error messages in the yacc parser of an open source project, for issues that arose repetitively, annoying the maintainers... but it was ignored. They are still getting annoyed by those issues to this day. And I'm still annoyed about it.

Same here! I figure there must be places that apprecuate this kind of thinking. So now I'm focused on only working in those kind of placea.

That assumes the engineers can prioritize the tasks and can't just get the list of frequent issues from support.

Also it assumes that the engineers don't find a better way to deal with the support questions, such as canned "answers".

> That assumes the engineers can prioritize the tasks

It doesn't. If the product manager is prioritizing things, he or she invariably gets backlash from the engineers when they put forward that they're swamped and can't deliver unless they deal with X, Y, Z first.

> and can't just get the list of frequent issues from support

Which are then, all too often in my experience, ignored or downplayed by whoever prioritizes tasks over at engineering.

When the engineers have skin in the support game, they pay extra attention to what may or may not lighten their "not very fun" workload.

No, this strategy really does depend on the engineers having a good way to prioritize things.

If you have engineers doing level-2 support and you have product managers saying "Do this other thing", then the engineer now repeatedly has to resolve a conflict among the possibilities of:

A) Check email and slack to see if something new has come in.

B) Check your error reporting/monitoring to see if Sentry has reported any new exceptions/errors from existing services.

C) Issue #1 from a customer

D) Issue #2 from a different sort of customer who has more revenue impact

E) Issue #3 which was actually bubbled up from someone in sales.

F) Non-urgent but important Project X from the product manager which you are actually 30% of the way finished with and consequently, it is the thing that is sitting in your brain.

G) Check hackernews as a salve to the stress of a bunch of ambiguously-prioritized tasks.

Accomplishing engineering tasks requires allocating your brain's working memory and directing your focus on ONE thing at a time. The harder that is, the harder it is to do anything but option G.

If you put engineers in this situation, but you don't give engineers the training and structure to arbitrate decisions about prioritization, then you're setting them up for needless conflict and failure. If you just designate all of your engineers as being responsible for level-2 support without any additional structured then each one needs to tell the product manager, "I'm not going to work on this project which will bring the company $X Monthly Recurring Revenue for another week, but I'm going to deal with these support requests. I am in charge of prioritizing what gets worked on. Your job as a product manager is not to decide priorities."

Which is a silly thing to tell a product manager. There should be an existing consensus around how this sort of thing is handled and prioritized. If your product managers aren't allocating time for fixing frequently-encountered bugs or UX problems, then the solution isn't to set them up for a conflict with engineers. The solution is to have someone exercise leadership of the product managers so that they is alignment on the general importance of getting bugs fixed quarterly goals and so that when someone (PM or engineer, whoever is triaging level-2 support queries) exercises judgement about the priority of a bug relative to other work, others broadly agree on that decision.

You need to have some structure accomplishes this.

That requires a good support team that follows process.

It falls apart when support doesn't train newcomers in what engineering needs, or when support just "throws things over the wall" to engineering.

I once worked with a very bad "director" of support who couldn't figure out that a customer was sharing something to the wrong email address. She did absolutely no diagnostics. When I called her on it, and explained that she couldn't throw everything over the wall to me, she quit the next day. Her replacement was much, much better.

> That requires a good support team that follows process.

> It falls apart when support doesn't train newcomers in what engineering needs

It requires as low skill level from support as you can get. Some rudimental diagnostics skills and understanding of the product itself are needed; from there, if it's not in the FAQ, let engineers handle it in full.

One of three things then happens - in my experience, anyways:

* The engineers prioritize fixing the stuff they don't want to repeatedly deal with.

* The engineers populate a FAQ for the stuff when it's low impact enough that they're OK with support copy/pasting the FAQ item until they address it - if ever. Which can lead to further questions that get fixed or expand the FAQ, but that's fine too.

* (In rarer cases) some of the engineers quit because they think support is beneath their pay grade.

> It requires as low skill level from support as you can get. Some rudimental diagnostics skills and understanding of the product itself are needed; from there, if it's not in the FAQ, let engineers handle it in full.

Management has to enforce that tickets aren't thrown over the wall. I can't count the number of times that your approach is misinterpreted as, "I don't need to know much about the product or try to understand the customer's problem and communicate it correctly."

Anyway, without support that takes pride in, well, support, it leaves customers very unsatisfied.

Reply on news.ycombinator.com