Hacker News Reader

Comment by pflats | Hacker News Reader

When you don't start your y-axis at 0, you skew the interpretation of your data. At best, this is a significant mistake, and at worst, this is intentionally misleading.

Take this graph:

http://i.imgur.com/bBzCK.png

It looks like women are rated as more than twice as smart as men. Huge difference.

Except until you run the numbers. Women are rated about 4.3% "smarter" than men. Not twice as smart, like the graph implies. Not 20% smarter. Not even 5%.

Please, pay attention to your graphs. They're great tools, but they can mislead as much as they can help elucidate.

View on HN

Gah, this data is so interesting, but is not presented very well.

First change I would make is put all the red bars next to each other, and the blue bars next to each other; there is more value in comparing across enthicities/genders than there is in comparing across variables. Second change I would make it get rid of the eye-popping primary colors and choose two more neutral ones (two different shades of beige maybe?)

Finally, Y-axis scale changes too frequently. Also error bars would be nice as has been mentioned.

edit: Now that I think more about it, at a glance it would be nice to show a scatter plot with a dot for each of Male/Female/Ethnicity/etc in the way that you have a scatter plot on your home page. http://judg.me

> Also error bars would be nice as has been mentioned.

Not just nice but absolutely crucial. If you're looking at differences of 2%, you need tens of thousands of samples to say anything definitive. He's only got 1000 photos total, so I'm pretty sure most of this article is just an analysis of random noise.

You're right.

Also, maybe I'm just dense but it took me a REALLY long time to wrap my head around the scoring matrix of the website, and even after a few practices, each photo I still find myself having to think painfully hard about what each quadrant means. I have to imagine that this would add even more noise to the data, if people's heads are hurting and don't know where to click.

One suggestion would be to provide an intuitive guide for each of the quadrants, not just the axes, e.g. "Outgoing idiot," "Shy nerd" etc. but of course you risk biasing the responses based on the language you use. The other option would be to simplify the input, only rating one variable at a time.

Yep, there are two sources of variance: the variance amongst a sample of 1000 photos selected from a hypothetical pool of all people, and the variance in how the sample of 100 subjects rated any given photo compared to the "intrinsic" qualities of the photo.

I don't see any analysis at all. I don't think there is enough evidence to claim any of these results. In a 1000 photo sample we would expect ~3% variability (not defined), or 0.3 on these scales.

Pick another 1000 samples, redo the blog post and see if there is anything in common (not that that would make it statistically significant).

Don't judg.me by my graphs :)

And worse than that, the y-axis range shifts. For example, the range on the Fat vs. Normal chart is much smaller than the surrounding charts, giving the illusion that the spread is much larger than it is. (site is down or I'd link).

True, and I was having a tough time deciding which option to take (whether to start the y-axis at 0, or 5). As far as averages are concerned, the differences are in the tenths and hundredths - so starting the y-axis at 0 would have it more difficult to see any differences across the graphs.

| so starting the y-axis at 0 would have it more difficult to see any differences across the graphs

Then maybe there isn't a real difference. See statistical significance.

A (somewhat blunt) rule of thumb, not just for you, but for anyone in this endeavor:

In the future, if you are making a graph like this, and the differences are so small that they don't show up when your y-axis starts at 0, they probably aren't important enough to talk about.

If you've taken a stats class, you probably know there are times when small differences are meaningful. Any time you're analyzing your own data, though, these differences are rarely worth even considering. Taking a survey and analyzing the resulting data is very hard. There are countless nuances that can affect your results, both mathematical and psychological. Look at political polling - even the people who do it for a living regularly get things wrong.

starting the y-axis at 0 would have it more difficult to see any differences across the graphs.

That's the point: the differences are small.

I would go further: the differences probably are not significant. (Small differences may still be significant. In this case, I doubt it.)

Reply on news.ycombinator.com