Hacker News as a case study to test the wisdom of the crowd theory
venngage.com
venngage.com
> Such a specialized community is a great place to get feedback on the technical details of your product, but not to validate your ideas. The responses from people in these communities tend towards scepticism to negativity, even for products they use.
Or with the methodology. Take for example the Dropbox thread [1] which the OP's sentiment algorithm classifies as having a "negative top comment".
This is what the purportedly negative comment says:
> The only problem is that you have to install something. See, it's not the same as USB drive. Most corporate laptops are locked and you can't install anything on them. That's gonna be the problem. Also, another point where your USB comparison fails is that USB works in places where you don't have internet access.
> My suggestion is to drop the "Throw away your USB drive" tag line and use something else... it will just muddy your vision.
> Kudos for launching it!!! Launching/shipping is extremely hard and you pulled it off! Super!
In what universe, besides the one of a sentiment-analysis-algorithm, would that be considered a "negative" comment? Moreover, even if that last, congratulatory line didn't exist, the rest of that comment is very well-worth listening to. It might have been wrong (has Dropbox penetrated corporate IT as well as USB keys? USB keys have definitely taken a beating in reputation and in convenience...but has Dropbox been an easy transition in corporate IT? At Stanford, we have Box)...but it definitely wasn't non-constructive criticism.
If a submission gets a lot of upvotes, to me that's a positive-enough sign of validation. Is it really helpful to the submitter to see a dozen/hundred comments that are merely, "Awesome! I like it!"? I often like reading the comments before I check out the submission, because I don't want to have to parse Press-Releasese to understand what the product does, or who it may compete against, or what its flaws might be...Even if the comments were completely devoid of constructive and insightful criticism -- I'm sure after a big launch, it's helpful, yet annoying when people immediately nitpick grammar and typos -- if you're a founder of a great product, the hemming and hawing of HN is probably the least of your obstacles on the way to success.
The algorithm likely has issues identifying if criticism is constructive. If you look at the negative/positive score of each word in isolation (e.g. fail, problem, drop, can't, hard) the comment is overwhelmingly negative.
You actually have to parse and understand the comment to prove that it's very positive. They could have ditched the algorithm, but then they would lose a quantifiable positivity score and instead would have had to rely on human intuition - which has it's problems, e.g. spend the whole day looking at negative comments and neutral comments might start looking positive.
The study may have had fundamental issues at the outset.
If you assume the people looking to validate their ideas are familiar with the HN community, then this effect makes the analysis more accurate.
> I have a few qualms with this app:
> 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
> 2. It doesn't actually replace a USB drive. Most people I know e-mail files to themselves or host them somewhere online to be able to perform presentations, but they still carry a USB drive in case there are connectivity problems. This does not solve the connectivity issue.
> 3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this?
or skepticism in:
> It's pretty nice, and I was thinking to myself - hey cool, I could make an online backup of my code. Then it occured to me - who the hell is this guy, and why should I trust my code to be on his server!?
> That's a huge issue you should consider. Why would people feel comfortable leaving their valuable stuff on "Drews" server?
Or this[1] conversation. Or this[2] one.
That comments page is, I agree, not a troll-haven or something of the sort. But we do have a tendency to try and pick apart every potential flaw we see in something.
It's not a bad thing! Thorough vetting of an idea is the only good way to find potentially fatal flaws. But if you're looking for what the response of the enthusiast Internet community is going to be, this is definitely not the place to go.
A trial by fire we are.
If as the owner/presenter you aren't looking for that by posting your program/model/business to HN, I think you've not done your research. HN is great for getting constructive criticism. Not all constructive criticism needs to be accepted or acted upon, but if you can't really justify why it doesn't apply to you, then you should think about addressing the problem presented. Otherwise, you're just looking for people to congratulate you, and there's lots of other places to get that. (for the general you, of course).
I do hope you didn't miss the very next sentence that followed the quote you selected.
To clarify, they said they manually classified the top comment instead of leaving it to the judgement of the algorithm: ", we manually categorized the top comment for each thread."
As for why their human eyes judged it as "negative", I'd speculate it was phrases such as "The only problem is", "That's gonna be the problem.", "your USB comparison fails"
So yes, the top comment's criticism is constructive, but at the same time, it can also be subjectively classified as "negative." The positive back pat at the end of the post was not about the product itself but for the accomplishment of launching something. (You could say it's the attaboy "trophy for participating" consolation prize to soften the previous paragraph's criticism.)
1) A flawed Theory of Mind applied to the general public: I think it's fairly safe to say that the technical crowd (which includes commentators of slashdot/HN) will lean towards the Asperger/autism spectrum. It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how products could be accepted by the masses (who are not geeks/nerds/experts). The famous examples being the slashdot dismissal of the iPod in 2001 and the iPad in 2009. (And Sara McGuire's article that this thread is about has more examples from HN.)
2) Skepticism/negativity is easier to itemize and write, and as a strange bonus, it is perceived as smarter and more insightful analysis. Looking at all the things that are "wrong" with a product might be described as a type of "closed-ended" thinking. On the other hand, imagining the different ways a product could succeed involves more "open-ended" analysis. The problem of writing open-ended thoughts is that it looks like naive futurism, or unsophisticated cheerleading. (E.g. haha, Back to the Future's Bob Gale thought we'd have hoverboards in 2015! Dean Kamen thought Segways would sell millions of units and would revolutionize the world! etc).
Because of the combinations caused by faulty Theory of Mind and biases towards the shortcomings of products, the commentary on new business ideas will almost always end up being negative.
That's a pretty bad conclusion. Is it any easier for non-"geek/nerds/experts" to judge the iphone on its mass market appeal? I don't think so. In fact that's pretty much what you figured in the BTTF "predictions" failing.
The thing is a lot of people here on HN, and in the tech community, like to judge products based on how easy or hard they are to build, and that has zero relation to how good the product is or how successful it will be. For a non-techie, it's in the realm of the impossible either way, so that stuff doesn't matter at all.
This is also true but that wasn't my point.
I'm saying that the armchair quarterbacking from nerds is often predicting the business success (the mass acceptance) of a product. There is a huge disconnect between how expert tech communities think and how everyone else thinks. Steve Jobs was one of the few that seemed to bridge this disconnect. But Mike Lazaridis of RIM Blackberry could not. Jobs was more consumer-minded rather than technical. Lazaridis was more engineer-minded rather than consumer (e.g. consumers want a physical keyboard more than a touch screen.)
Note that you can make a lot of money out of a product that flops in the mass market like an iphone. Out of any group of ten Americans, roughly ten drink water, use indoor plumbing, and have access to utility generated electricity, all rather successful products. Its also true that nine out of ten Americans don't have or use an iphone, however, its possible to make an enormous stack of money out of stuff 90% of the population is uninterested in.
For other examples of the same phenomena of making fat stacks of cash off practically no one, see pop music, television, professional sports, AAA video games...
Its a failure to understand the market. Around 9 out of 10 comments about an iphone should be somewhere on the spectrum of "eh" to "that sucks" because a random sampling should indicate iphones are only about twice as cool as Congress. People on /. complained about the ipod because they're normies, not because they're nerds. Normies don't like that stuff, look at the actual sales figures, therefore normie hangout like /. is not going to like the product.
Denigration and name calling of people who don't like a niche product is just Apple fanboyism.
[1] suggests that one in four americans own an iPhone.
1) http://fortune.com/2014/01/16/npd-better-than-1-in-4-adult-a...
http://spectrum.ieee.org/biomedical/diagnostics/engineers-an...
It makes me wonder if daycares tuned specifically to the needs of children with two engineers for parents. Maybe a slightly earlier intervention would prevent some of the expected kindergarten issues.
Also, purely "positive" comments such as "Cool idea!" don't add anything to the conversation.
That said, though, having your ideas validated or condemned on HN shouldn't mean much in the grand scheme.
Or more recently, Dropbox, which was Ycombinator financed. Someone linked to an old thread about them and it was largely dismissed as "just FTP" and destined to fail. The famous HN quote is "For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software." Yes, trivial for just about anyone, right?
I'd also add a 3rd bullet point about these forums being echo-chambers. I know I personally rarely write anything against the libertarian loving majority nor do I ever criticize any FOSS projects because in the past when I have, its been downvote central. This has a chilling effect on speech and forums like these just become extremists cheering each other on over time.
>will lean towards the Asperger/autism spectrum.
I like to think that techies come in all sorts of neurotypicalities, but the ones obsessed with fighting on the internet over politics/tech news are the ones well into weirdo territory. There are lots of techies with moderate views and with great horse-sense. They're just not spending all day on HN and Reddit "correcting" everyone like a real life Sheldon. The non-weirdos seem to be the ones who get shit done and are out there being successful or spending too much time creating to be bothered by online forum debates. The weirdos just have higher visibility.
Success is defined by a combination of customer appeal, investor persuasion, and ability to execute effective sales and marketing.
An initial Show HN typically includes details about the product and maybe the underlying technology, but doesn't usually tell you much about the other elements.
So I think it's unfair to judge HN on its prescience, when HN is as much about technical critiques as business potential.
It could be interesting would be to look at VC judgements of startups. VCs have a pitch deck, they have some hints about the financials and of customer growth, and they may know something about the reputations of the management team.
With the extra detail, you'd expect better prescience. I have no idea if that's what happens, but I think it would be very interesting to test the hypothesis that VCs make expert decisions.
What you're describing is a startup investor, not someone with autism. Even pg thought Facebook was lame. I thought the iPad was silly.
Predicting the future is hard.
In fairness, the Ipad is silly. The general population is slowly figuring that out. [0]
[0] http://fortune.com/2015/07/16/apple-ipad-watch-analysts/
Same here. It also serves as a phone in addition to watching TV on it and what not on it. Good stuff.
I'd bet that a vast majority of users commenting on HN are not trying to predict what other people think, but whether or not it is worthwhile to invest their own efforts into the subject.
I can sit around saying "smartphones are crap" all day long because I don't have a need for them, but that doesn't mean I believe they'll fail because of it. It means I'm personally not going to involve myself in the smartphone business.
If you notice, I wasn't claiming that 51%+ of posters are predicting what others think. I was characterizing a common pattern of commentary from a vocal (and possibly minority) HN crowd.
As an example, the following is 7 quotes from 7 different people pulled from the top 9 parent posts of the HN Docker thread. After reading each one, ask yourself if the poster was talking about himself, or was he talking about others?
- "Most corporate laptops are locked and you can't install anything on them."
- "It does not seem very "viral" or income-generating."
- "It's is problem everyone is having, and everyone knew it."
- "Why would people feel comfortable leaving their valuable stuff on "Drews" server?"
- "If you are looking for a wider audience than those who already know the context of dropbox, make a video where you lay out the case for use of dropbox using simple examples from user point of view(think a college student)"
- "How are you going to scale up your storage to meet the demands of the users? "
- "Your main competition is not USB drives: it is HotMail, GMail, and Yahoo! Mail. Once people are taught the "email it to yourself" trick, they love to use it"
Right, this almost seems like common sense. I see the Hacker News community as mostly entrepreneurially minded devs who are very invested in the Silicon Valley startup ideal. Obviously there's nothing wrong with that, and there are a lot of really smart people here. But geniuses in a particular field are often inept in others. Hacker News readers don't represent the average consumer, for instance, so I wouldn't put too much stock in their opinions about a primarily average-consumer-facing product like Airbnb. I would, however, put a lot of stock in their opinion about my new dev-facing app like Heroku.
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
Yes, completely trivial. Every user will definitely have no trouble doing that. (Also, accessing your files with an FTP client? Really?)
I didn't know Dropbox had versioning control?
----------
Just tried it again now, and found the option on the website as 'manage versions' under the context menu of a file (no idea how I missed it the first time). Still had to download the old version and replace the file manually rather than an explicit revert button.
I think articles like op's are good in conditioning the HN crowd to be more aware of their own biases.
Update your model of the world and continue on.
Someone I know got really stoned one night, FTP'd into a shared Dropbox folder, accidentally uploaded a file, tried to delete that file, and ended up deleting the entire contents of the Dropbox on everyone's computer at the company. We were never able to recover the files, but at least there was nothing too important.
With respect to the article, I've done data analysis on Hacker News myself (e.g. http://minimaxir.com/2014/10/hn-comments-about-comments/ ) and I'm a little skeptical. I was not a fan of using NTLK to classify positive/negative because the models used for training sentiment are based on reviews, not internet site comments, and when I had used NTLK, the results were skewed significantly. I used an alternate model mentioned in the linked post which IMO worked better.
The examples used in the article seem more like correlation-implying-causation and cherry-picking. There are many, many counterexamples of startups people loved on HN which died a painful death.
Not that I am arguing with their larger point - if you believe in what you are doing, do not let comments on HN stop you from pursuing your vision.
[1] I'll start: Fab https://news.ycombinator.com/item?id=2444387
There are others.
Q.E.D.
Remember though, that something being illegal doesn't mean impossible, or that it will forever be illegal.
It is, however, also impossible, because nobody has invented a car that can drive itself yet.
I believe my original theory has been amply proven.
But I'm sure no true self-driving car is not limited to driving in fair conditions.
If you were hoping for a good discussion, being clear is extremely important.
Our current political, social, and economic hierarchy is inherently by definition ideal and permanently unchanging and all disagreement is thoughtcrime to be voted down. Everything we've been indoctrinated to believe is right, because might makes right.
In a virtual world, nothing matters more than geography, specifically where you live, and that's not a bug but a feature. With a side dish of urban bicycle riding apartments are the only politically acceptable to discuss solution for humanity. Seriously HN may as well be an urban bicycle blog some days.
There's something inherent about the nature of writing code that means programmers should not only tolerate and expect extremely low corporate social status (think of working conditions and hours and position in the hierarchy and respect (or lack thereof) for ideas), but should embrace the low status and attack outsiders who disagree.
A secular prosperity gospel along the lines of who cares how many become unemployed, let them eat cake, I don't care about historical analogies and guillotines. Sure I was born on third base and think I hit a home run, but they can pull themselves up by their bootstraps, if they really wanted to.
Although nothing is more important than internalizing and understanding technological scaling problems, all macro level problems in economics, business, finance, culture, or international relations merely requires replicated and "turned up to 11" micro level solutions. You know, just like when bubble sort is too slow , the best solution is more faster processors, right?
(added another sacred topic: Much as young adults always believe their generation invented sex, drugs, and rock and roll, they are the inventors of the concept of combining arts and science to earn a buck, nobody has ever earned a buck by putting an artistic face on a boring engineering project)
(and a second added sacred topic: New means better, and new means less bugs than old. New is good inherently because its new. Also technology in IT isn't an endless rotational circular wheel of the same old ideas over and over again with the same old problems over and over again, its strict linear progress like an infinite highway to the promised land of suburban paradise or something)
(and a third sacred topic: voting should indicate how well the voter agrees or disagrees with the opinions expressed in a comment, not how well its written or how interesting it is, voting should solely be a popularity contest.)
That said, I still feel I'm earning a Godwin point for evoking this topic.
Well, about these concerns about security, they weren't totally wrong.[1]
[1]: http://www.thelocal.es/20150821/airbnb-to-beef-up-security-a...
The only one on this list with an 'ultimate outcome' is Homejoy. Codecademy has no business model that I know of, and Meteor isn't even close to profitable.
Quora is also used as a 'success' case and they also don't have a business model.
I find this post does a pretty poor job providing any sort of supporting evidence for it's claims.
The degree of 'magic' in Meteor, for instance, is something still discussed a lot, despite it's obvious benefits and success.
(Boom, boom)
I find this place kind of intimidating and as a result I find myself putting a lot more time/effort into my comments than I do elsewhere. I think that's a good thing.
Crowds, in certain structures that yield a useful dynamic, produce wisdom. People transacting in a market allocate resources effectively. Gamblers and bookmakers betting on football matches make good odds. Wiki-editors nitpicking away in wiki-world produce an incredible encyclopedia. Open source, etc. There are also lot of examples of crowds being stupid, and structures that yield a stupid dynamic.
It categorizes the first comment on DropBox[1] as negative. What? That comment points out a legitimate potential hurdle to user adoption, gives some marketing advice, and then praises them for launching.
While the article's instinct may be correct, I don't see that it has shown data that supports it.
-
For example, if you look at the Quora thread ( https://news.ycombinator.com/item?id=1197146 ), a comment could say something good about the site and also provide some feedback (e.g. https://news.ycombinator.com/item?id=1197230).
Is this comment marked positive or negative? If feedback is somehow clubbed with negative comments, I would doubt the accuracy of the analysis.
It also amuses me to no end that someone is collecting and analyzing data from HN based (in part) on my sometimes idiotic comments and silly debates with Internet strangers.
Many new ideas and applications are marketed on HN quite validly under Show HN.
However, the first pass at marketing an application may not reflect its form further down the line.
For example, we often see poor landing pages, or simply landing pages that don't convey the problem succinctly enough to quickly satisfy our short attention spans.
A few months or years down the line and these development teams have homed their product, ironed out the bugs and built landing pages that are easy for anyone to understand quickly.
Interesting none the less.
Hence, what is launched on HN != its final form and cannot determine future success.
And what 'more general crowd' are you so confidently comparing to, exactly?
However, their specific example is sort of wrong. AirBnB has indeed blown up, but this has been due to two major factors:
- Traditional landlords, rental properties and yes Bed and Breakfasts moving onto Air BnB, embracing a new, popular platform.
- Extremely high real estate markets in some areas making it attractive and almost necessary to rent out room(s).
It's not like AirBnB has created a change where most or even a significant portion of Americans as a whole rent out their rooms. The reality is that the objections the commenters brought up are real, and most people aren't interested in the hassle the comes with renting out a room, and most people's rooms aren't necessarily attractive to rent.
In the end the original idea people had of how AirBnB would have to be successful; convincing most people to rent out rooms, an convincing travelers to rent rooms in random people's houses, is what was wrong. They only needed that to happen in certain markets, and for landlords in general to realize the advantages they had by renting out their properties on AirBnB.
I love HN, but one still needs to remember that the audience, although influential, is quite different then a truly diverse mainstream crowd.
Accounting for confounding variables would be nice too (for example, if field X is generally more liked by HN than field Y, equally positive feedback on a startup doing X would tell us more than that on a startup doing Y).
Still an interesting idea, I just think that people should not forget that "data science" is a subset of statistics, and any kind of data analysis is not really useful without at least some familiarity with it.
I'd had the ones that are paid to say good things about a product they probably never used. If we're lucky they compensate each other ;-)
* Most of the successful startups will be acquired and merged with other more successful startups or established companies.
* Most ideas are repeated and competed aggressively.
* Most of the successful startups primary winning demographic / target audience turns out to be younger crowd, especially with college students, and the young crowd tend to be better at advertising startup ideas.
* Many successful startups you can name right away simply reinvent the existing platform by applying media attraction and sharing among crowds.
* Only way win is make noise. A lot of noise. Yeah. Back to pg's top recommendation, selling your product to users.
* Unless you have billions to burn on your own, don't try to be SpaceX or Telsa. You will never be them (or the same guy who founded both entities).
My own unscientific conclusions.
"They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown."
- Great founders describe the MVP accurately, even self-deprecatingly: "Sleep under my kitchen table." They mention the flaws, because they want the most useful feedback.
- Less-confident founders describe the gold-plated, fully-featured potential future version of the product and don't mention the flaws.
At least on a superficial read, the great founder's description seems less promising, but they usually end up with a better product.
I certainly see this dynamic at work in pitches I've been given.
Read the Zenefits post (https://news.ycombinator.com/item?id=5242681). I don't see a single negative comment as implied by the second graph on the venngage.com page (looks like 7 positive, 6 negative). Unless the negative comments have been removed between venngage pulling the data and today.
They have just survived and succeeded. Why? Who knows.
But obsessing about HN comments is truly not worth it.
The AirBnB thread is a great example of this. The risks associated with having strangers in your home is something that AirBnB has to market around and have protections from. (ie. the stories about crazy parties that trash apartments)
Aside from survivor bias, people were just showing their concern with security. And Airbnb success depends mainly on proving a feeling of security for its users (that why they have ID and facebook verification, for example).
For the most part, the purpose of posting something to HN is not to take a vote of that sort. Thus trying to view it through that lens is kind of silly.
Also: misspelled neutral in one of the graphs.
Nice post, though. Keep it up.
But that's vulnerable in the other direction. If 90% of startups fail, it's very easy to predict failure.
An interesting non-statistical inquiry would be to look at the problems people are complaining about and seeing whether they're still present in the successful projects (e.g. dropbox privacy).