“Default Position Should Be Skepticism” and other advice for data journalists | Hacker News Reader

“Default Position Should Be Skepticism” and other advice for data journalists

propublica.org

182 pointsdanso28 comments

The problem is incentives. As long as they are incentivised to maximize click through rates, you'll get hyped up headlines. There really isn't that much exciting stuff happening every day in science, and even when there is, someone writing a clickbait headline will be better off than someone writing a measured one. An excellent Twitter handle in @justsaysinmice. https://twitter.com/justsaysinmice All it does is says 'in mice' for all studies where hyped up headline and sometimes the main body leave out the fact that the study is only in mice.

Same is true for any sort of science. We're trying to find some sort of a story, and thus we bias ourselves towards bugs that lead to a meaningful conclusion. Most experiments fail, but a bug can make them look like a success.

I find it especially important if you're applying supervised machine learning techniques. Far and away, the easiest sort of bug to make is mixing up training and testing and holdout data (or not doing it at all). It's such an easy bug to make because you're trying to improve the model accuracy... and a really easy way to do that is to train and test on the same data.

Oddly funny to me that there are data journalists out there not following the headline advice. The average person really is doomed.

You would have thought the meteor with Martian bacteria, the University of Utah's cold fusion and the OPERA team's superluminal neutrinos would have driven this lesson home by now.

Not just skepticism, but extreme skepticism. Literally assume what you are looking at is more likely anecdotal than solid evidence or patterns and even more likely politically biased.

Skepticism requires an unlimited amount of work for verifying a chain of evidence, if one piece is wrong then the whole body of data could be unless. Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

Even an average person isn't even going to spend the morning verifying a weather report demanding more data for an accurate prediction. Instead they'll elect individuals which they trust. And it's unfortunate that they're either wrong, corrupted or exploited.

Even today I can't even watch my old favorite youtubers without being pitched some VPN, Web hosting or audiobook service. I feel the people I trust for entertainment have been compromised to say and act a certain way to make some corporate CEO happy who is manipulate numbers so some data journalist can come to some incorrect conclusion about the world and award him some approval rating.

> Skepticism requires an unlimited amount of work for verifying a chain of evidence, if one piece is wrong then the whole body of data could be unless. Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

In theory, perhaps, but not in practice. In practice, the bar is usually at two standards:

1. at least two uncorrelated/independent sources of truth say the same thing (replication)

2. peer review turns up no clear flaws in methodology

Then your conclusions are good enough for most people. You will not be able to completely rule out the possibility of an incomplete understanding or a theory that is mostly right and a little wrong, but the vast majority of untrustworthy science in the news fails at basic diligence, not at "it looks right to the best of our knowledge but turned out wrong."

> "the vast majority of untrustworthy science in the news fails at basic diligence"

I totally agree. I think it's due to our times, because many people just replace religious certainties (which seem to make a come-back this days btw) by scientific certitudes. Yet it's not how science works...

> "it looks right to the best of our knowledge but turned out wrong."

It's the underlying of scientific process and it needs to be repeated until integrated :) With that in mind, you can hold opinions (because it's useful in life ^^) while managing to change your own mind if a better explanation appears (even if no one is immune to confirmation bias but we can try, can we ?)

Skepticism is the application of the null hypothesis. Building from nothing other than observables we can establish a system of order where in testable statements can be made to assert logical arguments to establish a set a first order principles via a criteria on true or false evaluations. This is the foundation of science and mathematics. To state:

>Skepticism requires an unlimited amount of work for verifying a chain of evidence

Is an equivocation that any statement that is attempted via conjecture and criticism is impossible. However, you are using a machine built entirely on those principles to communicate the idea that logical foundations are impossible. Congrats. You have won today's irony top score.

A lot of data journalism isn’t necessarily done on deadline, and when it involves public data (e.g. government records), oftentimes the resulting story is about how the data is wrong or insufficient.

Skepticism by definition requires no amount of work. What you are describing is the action of taking a position, or wanting to take a position be it for or against.

> Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

Doing that is actively trying not to be a skeptic. A skeptic would wait for the party making the claim to present the set of data validating the position.

Now, I do not now if the default position in the media should be skepticism, but your misrepresentation of it makes me wonder if you are confusing what being a skeptic means and maybe that is why you find it so troublesome?

The default position should be skepticism for everyone. Including hacker news. I know I get constantly downvoted for skeptical view points here in the comments, I wish it were not so but quite a few people here engage in bullshit. Most commonly seen types of bullshit here is, ancap philosophy, cryptocurrency advocacy, diet fads, and chemophobia.

Your impression, that HN is skeptic-hostile, is the opposite of my experience.

I browsed some of your comment history for evidence supporting your position.

I saw that your downvoted comments fit into buckets I often see downvoted (what I observe, not what I do myself):

- excessive profanity

- making a factual mistake

- saying something sarcastic

In the case of sarcasm, HN seems to dislike it either because the comment is taken sincerely or because it "doesn't add" to the conversation.

I didn't see any evidence that you were downvoted for skepticism.

I use profanity, I don't see it as excessive. "Bullshit" is probably my most often used obscenity but in the world of skepticism its used as an interjection when someone is dead wrong in the same way a religious person would blurt out "blasphemy" whenever someone takes the lords name in vein or such. Also, I do get downvoted for skepticism, not for factual mistakes. Show me where you believe I have made a factual mistake and I will show you the academic papers that back me up. Additionally, According to "The Relationship Between Profanity and Honesty" by Gilad Feldman et. al. https://journals.sagepub.com/doi/10.1177/1948550616681055 those who use profanity are typically more honest than those who don't. So why would hacker news be against profanity if it holds the skeptical view.

I doubt many people here are "against" profanity. From only my own anecdata, my assumption is that it makes your comment sound emotional or childish to people.

For example, I agree with what you wrote recently[1] about diet fads, but you did yourself a disservice by starting the comment with, "What? No! Fucking hell."

Your position is reasonable, but it begins like a lot of angry, irrational rants that we see on HN. I assume some people downvoted without getting past the first sentence.

I think something similar happened when you railed against reddit[2]. I agree with you, but you come across as cranky.

Finally, I see some examples of personal comments that are often downvoted. In one case, you say someone is a libertarian without knowing them (and seemingly in a dismissive way)[3]. You also have at least one post flagged for being personal[4].

My overarching suggestion is this: HN has a very different implicit culture than reddit. While I agree with you on almost all of your ideological positions, I also agree with HN's downvotes most of the time.

The culture of "unemotional, value-add comments only" gives HN a much higher signal-to-noise ratio than reddit. There are also surprisingly few flamewars here.

If you find your emotions appearing in your comment, take a few more seconds to re-read and edit them out. I think that in the marketplace of ideas, yours are fairly reasonable and popular, but the tone seems to rub people here the wrong way.

1. https://news.ycombinator.com/item?id=20141441

2. https://news.ycombinator.com/item?id=19572278

3. https://news.ycombinator.com/item?id=18781643

4. https://news.ycombinator.com/item?id=18781789

Conversely, do you understand how petty it sounds me? "Ew, he used a cuss. Better down vote him." Does swearing detract from the formulation of my argument? No. Does it add a little spice to conversation? Maybe. However, the idea that I should not use a tone of voice that conveys shock when someone states something outrageous is beyond ridiculous. Yes the emotion and swearing is extraneous, but I'm not going to respond in plebeian fashion just on the off chance I might offend someone. We're a bunch of grown ups here, no need to pretend like this is some sort of library where we can't raise our voices a little.

I will admit that perhaps I got too personal on occasion. Also I was not aware of the rules for comments as I had not previously read them.

Journalists can't even competently review and summarize a published study.

The default position for data journalism should be: don't try, you fucking hacks.

Should I be skeptical about the default position?

Wickham doesn’t say to be “skeptical about the default position”, which implies being skeptical of the status quo. He seems to be saying be skeptical of what the data purports to be, until you understand its origin and collection process and etc.:

> But until you’ve understood the process by which the data has been collected and gathered ... I think you should be very skeptical. Your default position should be skepticism.

This seems reasonable to me. Data is inherently a simplification of the reality it purports to observe and store. Not adequately researching and challenging the data is taking too much of a leap and faith.

Yes, in that skepticism should not be taken to an extreme.

The problem with skepticism is it makes it harder to change your own opinions when they are incorrect. The goal is not to reinforce your own opinions, that’s going to happen either way. The goal is consider information that you would like to be true can also be false.

Yes. If you figure out better position then it should become the new default. An interesting question would be if the default position depended on environmental factors.

The "default position" should be skepticism for everyone, all the time - not just data journalists.

Easy to say, hard to practice. I often don't verify that the bag of rice I bought with a digital monetary transaction is rice until I use it for the first time, and I don't even check it for biological agents at that time!

Life Improvement Protocol:

Move as much as you can of your decision space into "default trust" environments.

Save the "default skepticism" environments for areas where your counterparties are particularly sketchy and/or the parts of your job that make you a professional risk-sniffer-outer (journalist, investor, cop, etc.).

Default skepticism is a hell of a bad way to go through life buying rice and greeting your coworkers in the morning.

Your sensory inputs provide a hell of lot of information about some rice.

I'd also argue that experience, in many locales, recommends being skeptical that there's anything wrong with food sold in a store.

Default skepticism doesn't mean verify everything every time. It means that when confronted with a new claim, a claim you've not decided yet its validity, you should abstain from taking a position until you have enough evidence to accept or refute the claim.

In the case of your example, I gather you've bought rice before, you've probably bought at that store before, you've used money before and you've eaten food before. By the looks of it, you've probably cooked before with that same kind of rice. Hence you are not evaluating the rice and all its production line from scratch every time you buy the rice. You already evaluated that claim and you've found that the store and their products are acceptable to you and that is why you probably bought the rice.

In that case you abandoned long ago the default position because you already had the required evidence to take an action. Which is the only purpose of the default position.

The default position is not the forever position.

The problem is that now you've evaluated that news agency or political leaning and you've stopped evaluating it, and 15 years later the message has changed but you haven't noticed.

Why would you only evaluate the news agency? You evaluate "from the claim" not "from the claimants".

I know that is what people do, and that is why this article exists, to tell them not to do that. Ironic I know. But hopefully someone will get it.

Reply on news.ycombinator.com