Mistakes startups make when talking to users
survata.com
survata.com
Surveys can be a useful tool, but nothing replaces a real conversation. Even asking more open ended questions in an email or survey form can give you so much more information than radio button answers (and unless you're a huge company, it's not too hard to read a few hundred open-ended responses).
Along those lines, I'll share a snippet from patio11's Microconf presentation last year (posted on HN yesterday), regarding interpreting the responses you get to a more open-ended conversation:
"If you’re solving a problem people actually have, they will say at this point, 'Shut up and take my money.' If someone says, 'That’s kind of interesting, tell me when that exists,' you have not successfully identified a problem that people actually have."
You don't get that kind of insight from radio buttons.
The theory seems to be "Oh, we'll get more information that way." Or, from the more honest ones, "This was less scary than trying to talk with people one on one." But surveys are mainly useful for confirming or denying particular generalizations from real conversations.
Surveys throw out all sorts of data. Every raised eyebrow. Every excited look. Every long pause where they try to figure out what they hell you are talking about. Every gesture, every tone of voice, every tangent. You really need that data when you're first starting out. Early on, the problem isn't the things you know you don't know. It's the things that you're pretty sure about that are false.
Not true in my experience. In fact I see exactly the opposite. The folk doing surveys seem far more likely to fool themselves that there onto something. They're usually some way to cut up the quant data so it looks like "success" by some category or another.
Surveys are a great tool in the right sort of context - but early customer development isn't one of 'em.
I do a little workshop on survey design. The first slide is:
The First Rule of Surveys: Don't do them.
The Second Rule of Surveys (for experts only!): Don't do them yet.
(yes - it is a deliberate paraphase of Michael Jackson's rules of optimisation ;-)
As for the contents: it's a PR promotional post for the startup disguised as a list of tips. The article also misses the point: "talking" involves dialogue, which is a two-way street. Conversely, a survey is a formulaic, one-way exchange of information that pales in comparison to a real conversation.
It can be summed up as: Asking the user for some sort of interaction and then failing to respond to the user's action.
This can be blog posts where there's an open question at the end with the goal to engage readers and failing to answer to interesting comments.
But I see it more frequently when I go for some sort of SaaS-trial, my account expires after a week or two, they ask me for feedback and since I'm not convinced of the product yet, I might give them feedback and state my concerns. That is actually the last chance for them to make me change my mind.
And they miss it. They simply fail to reply or to address my concerns that I stated in my feedback.
I don't have any numbers but my educated guess is that they lose quite a few customers this way. Some lost me this way and I bet there are others.
If you actually want to look at how to get better at talking to users - rather than writing surveys - I'd recommend getting a copy of Andrew Traver's pocket guide "Interviewing for Research" http://www.fivesimplesteps.com/products/interviewing-for-res...
There's also an upcoming book on interviewing by Steve Portigal http://rosenfeldmedia.com/books/interviewing-users/ - googling around his name will find you some useful interviewing slide decks.
Indi Young's book Mental Models has some good advice on interviews in one of the chapters, and is also a useful read http://rosenfeldmedia.com/books/mental-models/
(Bias disclaimer: I know Andrew, and I do workshops on interviewing ;-)
There have been plenty of times when I have willfully taken surveys to help out companies I like then immediately gave up after being shown a survey with 20-60 questions. No.
What pisses me off is that they're asking for my time and effort without giving me anything in return. Like they feel entitled to taking 15 minutes of my time.
The worst is when they only display 1 question per page and there's a little progress bar at the top. And after 10 questions you only see the progress bar 7% full. At least it's not as bad as the time Starbucks/Chase Duetto Cards hired someone with a studdering speech impediment to call me for a phone survey... That lasted 30 minutes.
Also - it doesn't have to be the same 3 questions to each visitor. Draw them from a pool.
At the beginning they asked about overall experience (it sucked due to long checkout queues so I got the bare minimum, waited and left). They then asked me which "departments" I shopped in using their internal terminology which doesn't match what real people use (eg I think "cold goods" was one of them). Then for each department they asked me exactly the same set of questions, then they asked them again for the store as a whole. Then I got asked about 10 questions for the self checkout (which took longer than using the thing).
Did I mention I had to enter a 19 digit code to start, which included 7 zeroes in a row in the middle of the sequence?
Pay for a streaming video services? Depends, 99% of current services I wouldn't pay for. How often do I get a new mobile phone? Depends, sometimes two in a year, but the current one is already 3 years old. Fuck if I know the "average". How often do I check my mail per hour. Depends on where I am and what the fuck I'm doing, of course.
Etcetera, etcetera.
Survata is one tool, but nothing beats direct, 1-1 customer interaction. Can't remember where I read it, but one CEO of a startup made it his job to handle a decent amount of customer support email. Of course, that only scales so far, but out of the gate -- that's priceless interaction.