Hacker News Reader

I guess we’ll just have to accept that we may never know how many people were previously aware that they could create polls on HN.

It’s not just that online polls do not provide valid data, it’s that they provide misleading data.

Potato potato, as they say. It wouldn't be surprising if there was no link at all between reality and the poll.

Consider as a worse case; there is a political question with a fairly clear reality ("the legislators should legislate the value of exp(1) to be 2.5 to make life easier"). The silent majority ignores the silly poll, and the lunatics who believe this is a good idea all put in a vote because they want to promote their mad cause.

The poll could therefore be biased to be exactly wrong. May as well call it invalid data and save the time thinking about it.

Opinion polls, properly constructed and conducted, measure opinion. They do not measure any truth other than the truth of a specific opinon.

Polling for the value of e is rather like polling for the piloting of a jetliner. You could certainly do it. It would end rather badly in virtually all instances.

The point is the poll wouldn't be picking up the opinion of the crowd. It would be picking up the least representative sample of people; those who have an incentive for the poll to mis-represent popular opinion.

I only picked exp() as an example because it can't start a political argument.

I think we're largely in agreement, if discussing different elements of the same question.

HN is very representative of the type of technologists that frequent HN.

It's the kind of thing that's very popular among those who like that sort of thing.

Then we have to accept we are being misled about how many people did not previously know about HN polls.

But by who? Who is the puppet master?

Deception and bias can emerge without intent or control.

Which again is a major point behind statistical data gathering methods and practices. Invalid methods -> invalid results. Regardless of intent.

(Though of course, those intent on deceiving can construct biased samples to drive their agenda.)

@dang? /s

I'm just annoyed there's no CowboyNeal option

This was also my immediate reaction, especially given the non-serious 'dang is a lizard' option.

We're getting old. Most people here wouldn't recognise it.

"dang is a lizard" is the new Cowboy Neal.

On HN you use dang instead. Didn't you know?

Until a few hours ago I didn't know you could have polls at all

I was joking,I didn't know either.

For those that don't know and were born after 2K Cowboy Neal is (was?) a moderator(owner?) over at slashdot.com. His name usually appears(ed?) with some snarky option on slashdot polls.

every data is misleading when you dont know how to interpret it

Corollary: all data are relational.

Are you suggest that my vote of ‘yes, dang is a lizard’, ‘yes’ and ‘no’ is misleading?

This depends entirely on the lizards.

Misleading implies intent, no?

This is always very apparent on /r/SampleSize (a subreddit dedicated to posting and taking surveys): While Reddit as a whole has far fewer women than men, women often outsize men, Germans are usually the 3rd biggest group after USA-ians and British, and transgender sometimes make up 15%. Luckily, most people there (judging by the comments) are aware of that.

Most serious online pollsters are using random sampling through panel providers and stratify based on known demographic statistics. Samples are re-weighted by raking (which reduces bias but increases variance) to better match population statistics.

This of course doesn’t mean all sources of bias are eliminated, but you also can’t eliminate all sources of bias in phone polling. Just like those taking polls on the internet, certain slices of the population that you can’t control for or don’t know about may be more or less willing to participate.

> “Another reason is that people who volunteer for polls may be different from other people in ways that could make the poll unrepresentative.”

See also “most of what you read on the Internet is written by insane people” on r/Slatestarcodex - https://www.reddit.com/r/slatestarcodex/comments/9rvroo/most...

It’s definitely worth remembering that just as polls can be misleading about sentiment, so can the voting of comments.

These criticisms are not limited to just online polls. One advantage that online polls have is a larger sample size. Gallop polls, for example, tend to have an extremely small sample size of less than 2,000 people while an online poll can reach hundreds of thousands. Both online and offline have advantages and disadvantages. At the end of the day, if you're being shown a poll that you didn't pay for or participate in then it's likely intended to influence rather than educate.

Systematic bias in participants would be a far bigger worry to me than sample error from few participants.

I'd take a properly randomised poll with 200 participants over a non-random one with two million any day of the week. At least if participation is truly randomised you can calculate the maximum possible extent of the error. With bias, who knows?

Yep. People tend to underestimate the power of truly random sampling. For questions with a roughly equal split, a poll of just 100 people properly selected randomly from a population of 100 million has 99% confidence of being within 12% of the value you would get if you sampled the entire 100 million. This falls to 4% with a sample of 1000.

And the other way, you can’t make up for bad sampling with response count; the classic case study is the Litery Digest poll of the 1936 US presidential election, which polled over 2 million people but got the result wrong by over 30 percentage points.

Right. "Dewey Beats Truman" is another notorious example:

https://news.ycombinator.com/item?id=29757119

As household telephone service became near-universal, telephone surveys did become highly valuable and generally accurate in measuring public opinon. That's reversed over the past decade or so as both practices (refusal to pick up unknown calls) and service (landline service is at or below 20% in many parts of the US) have radically reformed.

Political pollsters have commented on this at length, and it's a major concern. See Nate Silver at https://fivethirtyeight.com among others.

A larger random sample reduces error by the square root of the sample.

If the sampling is biased, however, all bets are off.

A 100,000 observation poll has 100 times the costs of a 1,000 poll (data must be collected from 100,000 times more samples), but offers only 10x greater accuracy.

You'll find this mathematically in the definition of standard deviation, which divides by the square root of the sample size: sqrt(1000) ~= 31.6, sqrt(100000) ~= 316.

Larger random samples are useful where you're exploring many variables, or very small portions of the population. They afford greater precision. The accuracy however is dictated by the randomness.

I had a strong lesson in this a few years back when the question of active user participation in Google+ came up. I'd had one too many hand-wavey assertions that the site was far more active than was generally claimed in the press. I'd realised that G+ had a set of sitemaps files, and included in those were sitemaps of individual Google+ user profiles. Present on the web results for that URL was an indication of whether the profile had never posted publicly at all, or, if it had, what the most recent publicly-posted content was.

Each sitemap file had roughly 50,000 entries. There were something on the order of 40--50,000 profile sitemap files, about 25 GB in all.

I took a gamble and made the assumption (later tested and largely validated) that profiles listed within a given sitemap were themselves a random assortment. This checked out on any number of eyeball analysis (creation dates, user names, global regions, and activity status all seemed both random and uniform over a few tested files). So I selected one sitemap file (itself at random) and over the course of a few days using a pretty modest laptop and broadband connection pulled down and web-scraped some 50,000 entries. A simple pattern match told me whether or not the profile was active, and when.

Within the first 100 profiles viewed, the trend was very clear. Only about 9% of profiles seemed to have ever posted any content. That percentage varied between roughly 7--12% initially, but rapidly converged as my dataset grew.

I let the run continue regardless. I resampled my sample subsetting it variously ("Monte Carlo estimation") to see if the values varied (I believe either 60 or 100 record subsamples), and again, the same 7--12% or so was returned for each. Looking at recent activity (within the month during which the analysis was performed), was 0.3%. The analysis also revealed just how much the forced integration of YouTube and G+ had inflated G+ activity numbers (a bit over 1/3 of all most-recent activity).

This generated some blow-up on G+ among members there, and I got called a few things, as happens. Google themselves never formally responded (I did hear from a few Googlers who contested findings or methods.) A few months later, an Internet marketing group, Stone Temple Consulting, re-ran the analysis based on my methodology but on a 10x larger sample (500k profiles), selected from across a much larger set of the sitemaps. They fully confirmed my own headline numbers, though could (thanks to their larger sample) offer more precise insights on smaller groups within the overall population. I had absolutely no participation in the follow-up study, and was unaware of it until it was made public.

Stealth edit/update: Then as now, what annoyed me most about the whole episode was that Google were so obviously dissembling, making up numbers and/or outright lying about Google+ activity, and the press were largely lapping it up, when a very modest investment of time and effort would put paid to the lie. And after I'd done the analysis, armchair warriors continued whinging even as I'd fully documented methodology and tools used, enabling anyone to replicate and confirm or deny findings. Eric Enge of Stone Temple was the only one to do that. I don't generally hold marketers in high regard, but he earned respect from me in doing that.

https://ello.co/dredmorbius/post/naya9wqdemiovuvwvoyquq

https://blogs.perficient.com/2015/04/14/real-numbers-for-the...

(Stone Temples has since been aquired by Perficient.)

> A larger random sample reduces error by the square of the sample.

You mean by the square root of the sample, right?

Point. Yes. Corrected above.

Thanks.

I wonder what could be done if HN itself were to analyze data from an HN poll, combining it with other data they have on the participants?

Quite a lot about a person on HN can be inferred from that person's comment history. Inferences could also be made from their IP address history if that is logged. Upvote, downvote, flagging, and vouching history if logged would also provide some information.

I wonder if there would be enough to correct for some of the biases?

Pew Research is a biased source because CGI Script polling poses an existential threat to their business model.

No.

Pew Research (and pretty much any other credible research institution or academic) are domain experts, understand sampling methodology, and the very-well-known biases which occur from biased and/or self-selecting samples.

Public polling can be exceedingly accurate based on surprisingly small samples (roughly 300 in most national political polls, for example, and even that is generous), so long as the sample is in fact truly random. Oh, and that the responses are not similarly filtered.

One of the most famous cases of a poll which failed due to sampling bias was in the 1948 US Presidential election, and resulted in the Chicago Tribune erroneous headline, "Dewey Defeats Truman". This was drilled hard in my own stats education several decades ago and remains a sharp lesson in the risks of biased sampling. For a good descrition of the error, see:

https://web.archive.org/web/20180823071102/https://textbook....

Were you under the impression that the HN voting feature was intended to predict U.S. election outcomes? I can see you like Pew but their magical authority on sampling and biases doesn't apply here.

My point is that your statement is an unsubstantiated shallow dismissal which suggests a very poor understanding of statistical and research methods.

Pew was simply the first well-articulated reference I turned up. There are numerous others. You've offered nothing other than your uninformed opinion and casual insults and accusations. Please don't do that.