Hacker News Reader

If you were a mapmaker hired to make maps of the city for people, yes you should absolutely be shamed for trusting some random guy on the street without verifying it.

What the study exposed is that science journalists, whose literal job is to understand this stuff and explain it to the general public, suck at their jobs. You absolutely can blame them for just publishing whatever anybody says without verifying it.

I think you missed my 2nd paragraph. What I was saying was: no, this article didn't show that. The only thing it's exposed so far is that "science journalists don't watch out for this vulnerability". Well, so what if they don't? There are a million things people don't watch out for. Is this one actually causing problems in reality, or is this a hypothetical thing that doesn't occur in practice? To show this is a real problem requires providing evidence of real-world p-hacking in the wild that has had significant actual adverse consequences.

That was the whole point. Science journalists dont do the jobs we think they do.

Our presumption is that they do not trust anything as true facts until they verify and vet the information. I.e do what a journalists do. Report whats true

But thats not actually their job anymore. Their job is to get eyeballs. Doesnt really matter whats true.

In no sense is this limited to science journalism. Bizarrely, the most reputable journalist are probably at TMZ, because they get sued for libel if they are wrong.

When there is no cost for being wrong, there is no incentive for truth.

No, the prank pointed out multiple failures in different stages of the process.

> But even if we had been careful to avoid p-hacking, our study was doomed by the tiny number of subjects, which amplifies the effects of uncontrolled factors. ... Which is why you need to use a large number of people, and balance age and gender across treatment groups. (We didn’t bother.)

This poor design should have been reason enough not to publish the study - p-hacking or not. But...

> Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine.

It got published anyway in a "journal" with a fancy sounding name that doesn't actually do any rigorous peer review.

The team then issued a press release that the press regurgitated without any scrutiny. If they had bothered to look and if they were good at their jobs, they might have noticed that the journal was not reputable. Anybody familiar with any academic field can recite off the top of their head what the reputable journals they'd want to be published in are. If it's not one of those, it deserves a closer look. That's without even doing some basic googling about the team behind the research which should have been able to uncover that they were nothing but a website.

The problem here is a breakdown in society. Why do sham journals exist? Why do their press release reach the ears of journalists? Why is there no web of trust of credibility? Why have all our institutions (flawed as they may have been) collapsed?

> What the study exposed is that science journalists, whose literal job is to understand this stuff and explain it to the general public, suck at their jobs.

I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue.

Not that I disagree with the claim that they should explain things to a general public to accomplish the above, but that's not how our system is currently set up. If doing a poor job makes them more money - that's the real issue.

I’m related to a well-known science journalist. They absolutely consider it their job to verify.

There are irresponsible people who try to cram in more visible output by skimping on less visible quality. Everyone else grumbles about those reporters behind their back, like a developer whose code is undocumented and insecure.

Setting incentives to deter that behavior is management’s responsibility, as in any industry.

Since it’s an open secret in the business: The New York Times Science section has failed to do this in notable situations over the last decade.

>I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue.

That's because cynicism and acceptance of brokeness has become the norm.

But while journalism did start out as the "wild west" of yellow press and cheap wriging, and while it's that way today...

... there was also a period of many decades where people expected better of journalists, and they did get considerable better results...

This is not the job of science journalists. The job of science journalists is to make money for someone. The honor system is just the name that people who dont understand corruption give to no system at all. If you want journalism to be better, you probably have to make advertising and PR illegal.

>This is not the job of science journalists. The job of science journalists is to make money for someone

Well, if it isn't their actual job, it should be. And they should get no sympathy when they get it wrong.

There are analogies that make this seem bad (fake construction worker) and analogies that make this seem good (ethical security researcher).

We need to go past the analogy and make a deeper assessment of the factors that matter. For example, what does the bad actor stand to gain? How much harm can they cause? How often is this happening?

For example, if the fake construction worker scheme was used to rob people and it was happening regularly, it might be worth doing something about it.

And "doing something about it" doesn't necessarily mean throwing away trust. There are a bunch of other interventions when you look at the problem holistically, e.g. can we reduce the benefit, reduce the prevalence, reduce the net harm?

Imagine someone posing as a fake construction worker, shutting down the road, and the city noticing and telling everyone the road is shut down.

Then the poser explains this is something people are doing all the time for personal gain, and the city is enabling them, and this is a problem.

That seems like a public service to me!

Do you also burglarize houses to show people that they are not secure?

> Do you really want a world where everybody mistrusts you and nobody believes your word without cryptographic proof or something?

That's why we have a balance with indemnity/damages, which also can be punitive, which simply state: "Of course you can fool people. However, better don't otherwise you will regret it, buddy."

I side with you. Trust does not mean you cannot be fooled in the first place. That's why it is called trust.

A better analogy would be a fake construction worker lying about a street being closed and a bunch of people (the press) spreading the lie to the whole town without checking if it is any true.

The fake construction worker has to go, but that's only half the problem.

No, this prank was helpful.

Any system (in this case the system of how information originating in science makes it into the general public brains) needs to be tested from time to time to see if it works.

This prank was a test, the system failed (to stop this from reaching headlines).

And now everyone is harmed more by eating bad food, and no one fixed journalism.

Your analogy would apply to someone posting a fake article on one of the news sites. The correct analogy would be the council shutting down a street based on any plan with no checking of credentials repeatedly for decades, and then someone pointing it out by submitting a provably wrong plan.

p-hacking and journalists publishing junk science or downright lies is a very well known problem. People have been pointing it out for years. Publicly shaming them via a story that will sell well is just using the systamic flaw to patch itself. It won't change the net number of sham stories these journalists publish and if anything slightly increases the quality by marginally diluting the harmful stuff being pushed by snake oil salesmen with something comparatively harmless.

Not going to try to pick apart your analogy or think of a better one. But the study was real. They did “real” science (poorly). They didn’t lot about results or invent data. What they did likely happens everyday by people who don’t even realize, or worse by people who do fabricate results.

What they showed was how easily terrible science gets published to the masses as if it were reliable.

Incompetence (negligence) has to stop being an excuse. Sure, punish it less than malicious lies, but it needs to be at least as bad as, say, speeding in a car.

Unfortunately, we all live in a world where we must verify whom to trust, based on corroboration and behavior over time, and be willing to be skeptical until something is confirmed. One study is never enough; maybe even a few are not enough, given history of studies & opinions being reversed. This is regardless of what people * should * be doing.

I want a world with massive disproportional punishment for people who do such thing. Make dishonesty a life shortening activity, were you spend a mintime of a year in jail.

> "I want a world with massive disproportional punishment for people who do such thing. Make dishonesty a life shortening activity, were you spend a mintime of a year in jail."

Having been raised to truly believe that "honesty is the best policy" I would in theory be inclined to agree with you, but we sadly live in a world where dishonest people would absolutely guaranteed find a way to abuse such a system to their benefit to harm every honest person they felt a random urge to, while avoiding any consequences to themselves (probably by pawning those consequences off somehow on some nearby unsuspecting decent person). I have no faith or hope that humanity as a whole is headed anywhere but straight to Hell (if I was foolish enough to believe such a place exists). The vilest of humanity have taken control of pretty much every aspect of society, and the rest of us are just their playthings until the miserable end they've wrought comes to pass.

The situations are not analogous in so many ways.

One of the main points in science is that credentials shouldn't matter (lots of great research is done by people without a Dr title and lots of shit by those with a Dr title), the research was published so they could actually go and look at it easily. And given the timescales and stakes involved they should have:

Not following a construction worker direction can be pretty bad while doing so the most you lose is a bit of time.

This was also the actual job of science journalists and it takes roughly three minutes to look up a paper and check the basics like sample and effect size. They definitely have the time.

Also, there is a metric fuckton of junk science. Heard of the replication crisis? Your analogy would need 1/3 of men posing as construction workers to not be construction workers!

If it's not the scientist's fault, then why is it the journalist's fault, who is far less expert, and why is it not the reader's fault?