Comment by nstart | Hacker News Reader

Comment by nstart | Hacker News Reader

Parent Full thread

I absolutely love the gist of the article: interpreting quantitative research is something most of us are fundamentally illiterate on.

That's definitely how I feel about it at least. Others might feel differently.

That said, how does one actual build skills in this area. It feels like the author is able to do this here because it's a research paper in an area they have knowledge in.

How does one build up a general ability to critically evaluate numbers in a case like this? Does one have to know how to go look up formulas?

I recall an article from some time ago where authors released a paper about chocolate and health benefits or something similar but they intentionally gamed the numbers to find something that correlated. The press ate it up before the authors revealed the truth.

How does one go about becoming more literate in evaluating those kinds of papers and conclusions?

I'm willing to put in the time to read carefully and calculate. I just wish I knew what skills to acquire.

To evaluate stuff like this, you just need basic physics knowledge (enough to do dimensional analysis with) and estimation skills. One fun read along these lines is "The Art of Insight in Science and Engineering".

For stuff like the bad social science you mentioned, often you can catch it if you know basic statistics along with the common tricks people use to p-hack. See, for example, "The garden of forking paths" by Gelman.

Even if the statistical analysis is not given explicitly, or the issue is more subtle, for social science that touches on everyday life, you can usually tell if it'll hold up just with common sense. Try yourself using this quiz: https://80000hours.org/psychology-replication-quiz/

Fermi questions[0] (oft maligned when asked in interviews) are a good practice for this sort of thinking.

Anything that is either out of range on scale, or that ducks the question, is worthy of further evaluation and at least some mild skepticism. Theranos promising hundreds of blood tests out of a thimble of blood would be the former. The article referring to Volts of electricity is like a car salesman replying to 'how many seats does it have?' with 'The seats are upholstered in Fine Corinthian Leather' the statement isn't incorrect, it is just useless and, in context, intentionally misleading. Salesmen do it to sell cars, scientists to sell research, journalists to sell papers, and conspiracy theorists for, whatever reasons, I guess.

[0] A few examples: How many piano tuners are in New York City? How many ping pong balls could fit in a school bus?

> Fermi questions[0] (oft maligned when asked in interviews)

I really don’t get when people bemoan these. Do they not get that it’s supposed to be an estimation, and it’s not actually a quiz of whether you’ve memorized a seemingly useless bit of trivia? Do they not see how it can be a useful skill to have, especially in an engineering field where you frequently deal with many orders of magnitude? (If acquiring this lock takes 100 ns, am I doing this often enough for this to cause a user-perceivable delay longer than doing it single-threaded?)

The problem is more when you're a software engineer, and you're asked something like "how many manholes are in Los Angeles" - it is so completely tangential (at best) to any of the estimation skills you'd use on the job.

As the author notes in the article, all this takes is basic knowledge of physics units, and the only math required is addition and multiplication. So it’s a matter of basic skills to some extent, but that’s not really the key.

The way you get better at this is by doing it, a lot. Every time you read such an article, or some one relays such information to you, run the math through your head as much as you can. Make notes of the things you don’t know that prevent you from pushing the thinking further, and methodically learn about those. Don’t think of yourself in terms of particular labels - eg “I’m not much of a physics person” or “I’m a software guy” - your job is to understand the world, solve problems, answer questions.

It’s a constant effort and it’s not easy. It helps to surround yourself with others with the same drive. It helps to teach, as it forces you to constantly reassess the fundamentals, and the common mistakes beginners make (see Feynman’s writing on that topic).

This is roughly High School level physics (basic, not AP or any honors or anything).

In this particular case, there's the knowledge of the first basic law of thermodynamics (You can't win, the limiting case on energy conversion) and Kinematics, typically taught as the first half/third of physics. (Broadly: Mechanics/Kinematics, E/M, Optics and Waves)

In most physics courses, you’ll hit conservation of energy even before it’s presented in thermodynamics packaging. This kind of calculation should be less than a week’s worth of casual effort for someone who is familiar with algebra.

A physics undergrad degree will teach you all of these skills. I imagine, you can memorize a fairly large set of skills needed to do this in a few months, but practicing these skills to high proficiency takes a few years - which is the length of an undergrad degree.

Street-Fighting Mathematics [1]

[1] https://ocw.mit.edu/courses/mathematics/18-098-street-fighti...

Reply on news.ycombinator.com