Hacker News Reader

Comment by afpx | Hacker News Reader

Furthermore, it would be extremely useful if scientific research was transparant from inception to outcome.

Scientific ‘papers’ are archaic forms of knowledge transfer suited for a time when physical paper was the only way. These days, science would serve us better if we could see everything involved in the process. I don’t want to see only the results, I want to see the whole notebook and all the hurdles along the way. Why can’t we follow the researchers and the accumulation of evidence in real time?

View on HN

Sounds nice, but as a scientist I can say that it would be a huge overhead to have to constantly prepare half finished work for public consumption throughout the entire process of a study. Scientific papers make sense because the point is to summarize your results, discuss what they may imply, e.g what theories they support or deny, and describe what you did in a reproducible way so that the work can be verified. The writing of a paper is an act of communication. It's important that it is an explicit thing that researchers are trained to do properly, not something that they just shit out at the end of a poorly done study that didn't work, or worse, in the middle of something where they can't draw any conclusions yet. E.g. journalists already pick up on wrong or hype-inducing interpretations of poorly written press releases of published papers. Imagine if they could also pick up on half-finished work. "Study being done could mean something amazing, maybe, if it works!" You think fake news is a problem now? Imagine the world you propose.

Now, if researchers were encouraged more to describe in detail their process and everything that went right and wrong throughout a study, along with data and algorithms and everything, perhaps as appendices or in supplemental material like blogs or videos etc., and and I find this is what is happening lately, that would be fine, and a nice ideal to strive for.

But realize that scientists are already required to not only perform the study but write about it, and convince every skeptic that they are right, go to conferences and get an article accepted by a journal which can take a year or more. And now add to this that they are required to prepare the data and software for public consumption, make videos and blog posts that describe everything, answer all questions that the public has. Think about all that overhead you are demanding that goes so far above and beyond doing the actual science. It's not small. And they are not paid extra for it, in fact their paycheck is probably half what they could make doing closed science for a for-profit company. Meanwhile their job as an academic is only to explore new ideas and convince their peers of their worth. Why should they go the extra mile, for free, for every member of the public who demands answers and transparency? Sorry, but it's too. much. work.

Actually, when reading papers, I find the ‘communication’ parts to have the least value. It’s probably just me, but I try to skip over those parts because they tend to bias and frame the results. I want to understand the intent, strategy, methodology, math, models, data, outcomes, etc. but not editorializing. Plus, many good scientists aren’t necessarily experts at prose or communication, anyway. Why can’t the communication parts be provided as addendum?

For people who work in private R&D, like me, we try to keep lab notebooks so that we have records of pretty much everything imagined or tried, warts and all. Sure, it can seem haphazard, unformatted, and not packaged for general consumption. But, it’s meant for collegues anyway, and it fits the purpose.

As for your last paragraph, I totally empathize with your frustrations. But, those seem to be more structural issues with the economics and politics of science and the academies. I certainly would love to see change there, as well. I wish research could be funded in different ways. But, as a consumer of research, I can only really describe my particular use cases.

What I mean by "communication" is what you describe as "the intent, strategy, methodology, math, models, data, outcomes." What do you mean by editorializing? If you mean discussing implications on existing theories, I think this can be done badly or done well (like all parts of a paper.) But when doing research it's important to read follow-ups and rebuttals. We can't forget that science is a process, an on-going discussion. It's not a single result or paper.

I think many readers (science journalists, in particular) only read the abstract, introduction, and then immediately jump to the conclusions.

The introduction section almost always frames the paper with opinions and biases. And, the conclusion section often focuses too heavily on how great the results are (and the author’s prowess) without identifying interesting issues or irregularities encountered in the methodology, data, or outcomes.

I get that publishing is a huge part of a scientist’s career success and for self-promotion, but I see many papers that over emphasize those aspects. Again, I don’t fault them. One can’t easily get funding without a reputation. But, in my idealized (and probably unrealistic) world it would be different.

I do want to stress that I’m not complaining. What I have access to is amazing. But, sometimes I wish I could see it all.

> Sounds nice, but as a scientist I can say that it would be a huge overhead to have to constantly prepare half finished work for public consumption throughout the entire process of a study.

How about writing into a (public) blog, instead of a lab notebook?

First, my notes are not nearly well-organized enough to have any kind of coherence for anyone but me. Do you think it's interesting for me to answer public questions, to defend ideas that I haven't even completely thought through yet? But secondly, why would I want to publish my half-baked, unproven ideas? I can see two possible outcomes: (1) I completely embarrass myself because they are stupid ideas that I haven't verified, or (2) they are good ideas and someone takes them and runs faster than I can and beats me to a result. I see zero advantage for me to do what you say.

I think this idea of let's do every step of science 100% in the open are completely forgetting that science is a social process. Putting things out there has consequences. The onus is on the scientist to verify things before saying them. There is a word for "scientists" who do not. Personally I don't believe that that is something that will change in the next 100 years.

The current “science as a career” model of science with public grants etc stems entirely from the development of the atomic bomb and World War II resulting in Vanevar Bush proposal of the NSF. I sincerely doubt that science will be practiced anything like it’s current form in 100 years. What it will look like is an interesting question.

Great so people hostile to your research (eg Republicans) can turn your research political because they can take your notes out of context (try using the word "trick" in your research journal or emails) and use it against you or your research area.

“Why should they go the extra mile, for free, for every member of the public who demands answers and transparency? Sorry, but it’s too. much. work.“

I understand your exhaustion at the idea, truly, but in an era of publicly funded research, the public does have a right to demand transparency for the same through Congress (in America).

That said, a more reasonable level of transparency, post-facto, ought to be sufficient for for most - emails, data, etc. get requested of members of the Congress, for example. Or for a more representative example, take a look at the explanatory efforts of the Federal Reserve and when they go beyond the plain economic research to also provide detailed public transparency.

Oh I agree with you that these things should be made available if requested in a serious way. I also think scientific papers should be open access, for similar reasons. I'm not advocating keeping anything secret. And in general if someone takes the time to email me about a question regarding my work, of course I'm going to take the time to give a thorough answer to the best of my ability. But here I was responding primarily to the idea that scientists should report every bit of progress (or non-progress) in real time as it happens, and also to the statement that the scientific paper is an out-dated concept, which I happen to disagree with.

Yeah, that idea that you are responding to is the daily reality for a lot of software professionals. Especially if you replace the noun “public” with the noun “management”.

A growing number of people seem to agree with this ([0],[1]). There are a couple of places ([2],[3]) where one can preregister a hypothesis and experiment before collecting data. There are also some journals that are starting to encourage preregistration ([4],[5]). It would be nice if this phenomenon spread beyond the few fields it's started in.

I don't know about needing to see all the intermediate data, that seems extreme if the methodology is properly described and the raw data is available.

0: https://www.bitss.org/2014/06/13/preregistration-controversy...

1: https://www.apa.org/science/about/psa/2015/08/pre-registrati...

2: https://aspredicted.org/

3: https://osf.io/

4: https://www.psychologicalscience.org/publications/psychologi...

5: https://www.journals.elsevier.com/cortex/news/registered-rep...

> preregister a hypothesis and experiment before collecting data

This reminds me of an old website, I think it was called Half Baked, where people could (I think anonymously) post just any ridiculous idea for an invention. It was fun to read. Unfortunately I can't find a link.

But man, when I think about it in terms of science and in the more modern context of preprints, it sounds like Flag Planting, Extreme Edition. Even the ML community, who encourage preprint publication more than any other field I know of, have had some pretty heated exchanges about the validity of using that medium for flag planting, and that is when the "flag planting" at least consists of a short paper properly describing an idea, with some rudimentary empirical result and analysis.

So I dunno, it's an intriguing idea, but not without controversy I think.

Because they rightly fear that someone will scoop them. The Nash equilibrium is to wait till you’re done to publish.

Reply on news.ycombinator.com