Comment by ChicagoBoy11 | Hacker News Reader

Comment by ChicagoBoy11 | Hacker News Reader

I'm surprised that so many of the comments seem to attack the claims in the post along the dimension of "well, anyone can write a blog post." I think as a community HN would staunchly advocate that the recipe for building great companies or software is an almost radical devotion to reducing any and all barriers to entry, except in this domain somehow people feel that the rigidity and exclusivity which journals provide somehow lead to the betterment of science. Makes me think about pg's "what you can't say" post.

There is short shrift given to the distortion that gets introduced in order to "get published". The "file drawer" effect is very real, and it gives us a very skewed view of what the actual scientific work being done really is; p-hacking and other statistical trickery are table stakes to anyone trying to survive in the academic world, as being published is a key metric to success yet it is surprisingly orthogonal to being right. As Brian Nosek's "reproducibility project" is uncovering, there is a non-trivial amount of "science" out there that is pure garbage, and much of it was produced in order to "get published."

In a world where "random hackers" can write twitter bots to parse scientific papers and uncover obvious mathematical flaws (which sometimes invalidate central claims of the paper), I don't understand how we aren't immediately gravitating to standards which promote more and more openness.

Lastly, the biggest issue we have to address is the fact that notoriety is now the chief indicator of success, and not genuine scientific discovery. In college I had world-famous faculty who have made a career, fortune, and placed themselves well on the spotlight on the backs of research which was later completely reversed -- and not in the way "science is supposed to work" sense, but rather in the "you purposefully fudged the math" sense. And yet there was virtually no reputational damage done to their career. Worse yet, I think it is clear that this lack of ethics actually is a direct cause of the success they enjoy today. Given that these are pressures faced by all in the academic community, true openness is the one avenue we have to counteract this sort of thing, and hopefully enterprising individuals come along and find a way to make "truth-seeking" reputation something researchers care about.

One of the reasons why I sometimes dislike my industry (design) is it's filled with a lot of Medium posts that get highly upvoted by say, emerging or other adept designers, yet it's rare that someone will substantiate their claim with data to prove that a rule or belief that they state is actually true. Sometimes I feel like designers believe x only because other designers believe x, when in reality non in-group behavior might suggest something completely different, that x is irrelevant, unnecessary, or confusing.

What I enjoy about learning about other subjects that make claim in research papers is that it is at least legitimized through a rigorous process. One can substantiate that y is better than x because tests show doing y requires less computing time than doing x. Or that doing y results in higher accuracy of the model. While getting a research paper published is an imperfect process where even high quality ideas get rejected for at times pretty trivial reasons, I can appreciate the process itself mostly results in a sharpened idea.

> Sometimes I feel like designers believe x only because other designers believe x, when in reality non in-group behavior might suggest something completely different, that x is irrelevant, unnecessary, or confusing.

Parallax scrolling sites come to mind. They look pretty, but quite often are terribly dysfunctional to navigate and use for a customer who just wants to learn about or buy(!) your product.

Is _your_ assertion backed by data?

Here's a study that disagrees with you, at least in terms of how many people have trouble with parallax (2 out of 43). http://uxpajournal.org/the-effects-of-parallax-scrolling-on-...

Not to say that study is a comprehensive judgment on the design pattern.

TL;DR for anyone wandering by:

> We hypothesized that PS would improve UX, which is defined in this study as the emotions that are aroused when a user interacts with a product or technology. The PS website was perceived to be more fun than the no-PS website. With respect to perceived usability, enjoyment, satisfaction, and visual appeal, there were no differences between the PS website and the no-PS website.

Great that someone formally studied it, though I'm a little put off by their subjects demographics as compared with potential customers of any given business. That is, I feel like students at a university are more adept at using interfaces than "the average user" who is going to be your potential customer.

1/5 factors they tested did signal added value, the 1 being "fun". More fun, but not more usable. I'm not yet convinced that fun correlates to sales in this context as much as usability, ESPECIALLY once the initial wow factor wears off.

I assume by "trouble" you mean motion sickness. Two of them had motion sickness, not just run-of-the-mill confusion from the usability of the site.

All that said, the main point I was making was that often designers spend a great deal of time trying to get PS working just right not because they think it will bring in more customers or make their site more usable, but because that's what other designers are showcasing and they want to emulate that trend.

That's because a lot of folks on HN come from Harvard or Stanford or MIT, where it's drilled into you that this pedigree is super important. There are companies that will not hire you for an engineering role if you don't have a CS degree from a short list of universities unless you're part of an acquisition (and even then).

This gets ignored the second they get $800k in VC money but for some reason the same idea never translates to "real science" or academia.

I support your message in spirit. But I have a concern.

The more open publishing becomes, almost by definition, the more dilution happens to the citation process. Want to know where this happened before? Trace back to Google's origins, where the "brilliance" of the "citation" technique made it so much better to begin with. But what happened afterwards? Was the interlinking just left as it was, pristine and still as valuable, just waiting for Google and other search engines to exploit it? Of course not. Any kind of system which can be gamed will be gamed. And what happened to the internet after Google became the sheriff? I would say it is now a fairly overcomplicated and twisted system (especially for the hobbyist) with the cat and mouse game going on between SEO companies and Google.

The distortions that are introduced today in order to get published, will be replaced by new distortions that will be introduced to get cited.

> In a world where "random hackers" can write twitter bots to parse scientific papers and uncover obvious mathematical flaws (which sometimes invalidate central claims of the paper), I don't understand how we aren't immediately gravitating to standards which promote more and more openness.

I understand your idea but the thing is, nobody has tone of free time to read an unknown hacker's blog (if he/she really does). People are busy for working so the best way is to read a peer-reviewed paper. This thing is like a brand, I guess.

The same constraints you cite here applied to "peer review" articles. The academics doing the peer reviewing are also "busy with work" and want to do their own thing, and especially in cases where reviewers are anonymous, the incentives facing the academics charged with the review are absolutely horrendous.

My point with the part about the bot that checks errors is that, if we are being honest, in far too many cases that "peer review" is less than ironclad, to put it mildly. In that scenario, the journals actually do us a disservice because they attempt to signal that the content you are reading went through a strict bit of scrutiny before it reached your eyeballs. That is a dangerous assumption to believe in if it turns out that it really wasn't.

I guess it depends on the size and culture of a given research community. Researchers are people too and petty things do happen. But, as a referee, I have followed through error propagation (once I saw erorbars that looked amazing and wondered how I could get such beautiful results and discovered they made an error in the error propagation). When reviewing methods papers, I have checked the integrals and equations, looking for errors (including minus signs) because eventually it will go into some code that most people won't read. Then, there are just simple checks on the science and on history of the field. I don't think that I'm unique. Refereeing well takes time, but it's part of the responsibility/duty of being a researcher. A friend's metric is that he should referee at least as many papers per year as he publishes.

By all means, when you read a paper, be skeptical and go through it yourself, but in my field, referees do actually provide some quality control.

This is as it should be. But the reader of the article has no idea whether you looked at it or someone with a far less significant commitment to the task than you. And what's keeping you and your friend doing this time-consuming work is your commitment to science and your sense of ethics and responsibility, which your community certainly does not universally and uniformly share. That commitment is also the first to go when career, money, lack of time, etc. start coming into the picture, or the pressure to abandon it is certainly high.

That's what we need to find a way to alter: If you are doing great science -- including the painstaking work of carefully reading a paper and following minus signs through calculation and checking integrals and whatnot -- that should be explicitly rewarded in the profession, and shouldn't just rely on intrinsic values a researcher holds in order to survive.

I don't see journals today being the leaders in driving that innovation.

You're inflating the flaws, while diminishing the success of the existing system. Pick a field and pick the top 3 journals. Now how many of the papers published in those journals were junk science? The fact that you have unscrupulous journals and editors, doesn't go away with blogs. Its all the same. Personally, I don't care what the medium is, but I'm worried that organizations whose purpose is to highlight good work might die out. I think the problem is similar to selecting 3 resumes from a pile of 2000 resumes. If you look at the existing HR system, yeah, it sucks and good candidates are often overlooked over bad candidates who get hired because they're ex-Google or w/e, but then again you can't just dump two thousand resumes on your dev manager, and have them go through each one.

Also, before you propose tearing down a system that has produced results for decades, but is flawed, you have to show that the replacement is actually useful and capable of at-least performing on par. I have no problem with casual discussion on HN (in fact I actively avoid the insane people who want "proof" for everything) so I don't really care to hold your comment up to a gold standard of absolute certitude, but you should probably give your ideas a bit more thought.

> In a world where "random hackers" can write twitter bots to parse scientific papers and uncover obvious mathematical flaws

Do you have an example of this?

I think a StatCheck-like clone was built that would then go on twitter and try to find the author, but I readily admit I couldn't find it again. But I think StatCheck itself is a good example -- just a bot that can go through a paper and point out math mistakes.

Reply on news.ycombinator.com