Hacker News Reader

Comment by cpsempek | Hacker News Reader

I'm probably being too harsh but...

Good on them for writing this, it's important to admit when you're wrong. However, I feel like this outlet has a larger responsibility to be actual data analysts as well as journalists (over say a more traditional journalist for a more traditional news outlet). As such, why was the analysis done in the postmortem article not done prior to publishing the original articles? A good analyst is one you can trust, and trust for an analyst is built by drawing conclusions from highly defensible data, and highly defensible data is data which has undergone sever scrutiny of the analyst before conclusions are drawn, not after. Also, they should probably update the now erroneous articles with a disclaimer indicating that much of the research is now invalid.

View on HN

It's true that we have lost trust in their analysis, but there is an important thing to remember here: fivethirtyeight is not a peer reviewed journal. You should treat their articles in roughly the same category as a CNN article on something Trump tweeted, not in the category as an article published in Nature. While they do tend to do better data analysis than the Associated Press, 538 articles are not refereed and should not be treated as such. The assumption of a "highly defensible" data analysis is a little strong for them. This is very apparent on their sports section, which should give you a clue about the actual rigor of their analyses. Don't cite a 538 article in your academic research unless the 538 article is directly citing peer-reviewed research (and even then you should probably just cite the original research).

If you think of them as a news organization that uses data as it's gimmick to sell page views, you will be less surprised at events like this (disappointed, yes- surprised, no). They have the same incentive to sensationalize things as a regular news organization. Their mission is not to increase the knowledge of the human race, their mission is to bring in page views to sell ads and make money.

It may sound like I'm coming down harsh on fivethirtyeight. I genuinely enjoy reading some of their articles, but I make sure to remember what kind of organization they actually are and don't fall for the trap of thinking they are a think tank staffed with postdocs.

This anchors motives too much in economic determinism. It's a useful mental model to remember this is a factor, and it can come to dominate and cause problems, but it is not always the one true factor to rule them all through which we should view all motives.

You basically end up arguing that it is all about money, and real journalism cannot happen under profit-seeking organizations. It also trivializes that journalism's big challenge right now is how to do real journalism when the big tech players have vacuumed up their ad revenue.

I also wouldn't treat 538 as the same as CNN writing about a thing Trump tweeted. They actively talk about and discuss their journalistic goals. They try to be openly self-critical about what and how they cover topics. They are trying to compete by not doing the same thing as other organizations.

I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept.

I'm a statistician, and we always work with investigators that are experts in their field- unless we are researching statistical methods, then we act as our own experts. The statisticians handle the data analysis and make sure that the investigators don't make silly data mistakes. The investigators handle the reasoning and mechanisms behind the research. When they work together they can collect good data that they are familiar with and know how to interpret correctly. When they work separately they are prone to mistakes.

Fivethirtyeight (and also the ASU researchers) fell into this trap. They were not involved in the data collection, so they don't really know what each variable means, and just took someone's word for it who, (back to the economic motives) has an incentive to sell their dataset and not to tell the researchers to go look elsewhere.

I'll admit I was too harsh in comparing fivethirtyeight to an article about Trump's tweet, 538 is typically decent investigative journalism. However, I maintain that it isn't on the level of peer-reviewed academic research. Articles on 538 don't go through peer review. They aren't submitted and languish for months with revisions and answering follow up questions. I'm not saying that this mistake would have been caught by a referee, but in the peer-review process a referee could have caught it by asking questions about the analysis process and if it's valid to use a variable named "X" was being used as a proxy for a variable named "broadband access."

>I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept.

It's much much worse than this. I was once interviewed about my research by an NPR reporter. He had already decided what his story was about and tried a variety of tricks get me to say some pithy quote he had devised so he could use it on air. The problem was that my research actually debunked, rather that supported, the story he wanted to run, and the pithy quote was scientifically unsupportable.

I often wonder whether some of the quotes I hear on NPR are edited versions of the interviewees saying "no it wouldn't be accurate to say x" cut down to "x".

I think this is fair and is a sound refutation to my stance on how rigorous they should be. Your reply reads like a debearding-of-santa, which is maybe what I needed in this case.

I expect way more of FiveThirtyEight articles than I expect of CNN articles, and I value FiveThirtyEight articles much more than I value CNN articles.

I think the idea is that we all make mistakes. The best we can do is admit them and learn from them, just like any other postmortem.

Also, they did update the erroneous articles:

https://fivethirtyeight.com/features/lots-of-people-in-citie...

https://fivethirtyeight.com/features/the-worst-internet-in-a...

Also, they did update the erroneous articles:

Ah so they did, my mistake.

I think the idea is that we all make mistakes.

I am in total agreement, we all do make mistakes, and it's refreshing when people own up to them. I guess my main question still stands though. As a top-tier data journalism outlet, why was a data QA not performed prior to analysis, or, before publishing results? And if there was data QA performed, where did it fail? It just seems that they got bit by a time crunch to publish content amid the FCC repeal and sloppy analysis was the result.

Everyone can do better. It's impossible to research every story from first principles of physics.

They probably just didn't think of this possible flaw in data. I mean, it's not like the university researchers caught wind of any problem, either.

Reply on news.ycombinator.com