Retire This Idea: Scientific Knowledge Structured as “Literature” | Hacker News Reader

Retire This Idea: Scientific Knowledge Structured as “Literature”

edge.org

158 pointsjordigh85 comments

I what is missing from the article is an understanding of the incentives of academics. Quite often putting a piece of research "to bed" is the goal. Why? Because any good academic has a long list of things they want to get to.

Very few people want to get mired in the (inevitable) problems that arise in a paper for eternity. This is why there are review papers that summarize the state of the knowledge at a given point. These articles are useful summaries of what came before and what (at least appear to be) dead ends were found.

Maybe I'm missing some point with the author's article. But getting units of production out and finished (i.e. papers) is a useful process. I'm not sure what would be gained by keeping documents editable forever. Scientific literature is not code.

Most of the great scientists worked on a single idea their entire life and expanded upon it, improved it, corrected it and were never really done with it (einstein comes to mind). Scientific knowledge is never final

I personally have a huge beef with the way life scientists publish their results in tiny tiny bits, that makes it extremely hard to cross-check them with other studies, to find out if they were later disproved or even to figure out the average of some quantity. Try making a concrete model of something and you will find yourself searching for experimentally measured values in tomes among the blabber of introductions and discussions, like monks did in the middle ages... Databases sometimes exist, but they are not mandatory so you never know if they are to be trusted. We need a structured way to catalog scientific results , and journals are not it (esp. with politics which lead scientists to publish in marginally related journals ).

That's not necessarily true, and I'd be incredibly worried if it were true. I think it's more an artifact of the design of this version of 'science' as an institution.

Freeman Dyson was a brilliant drop out mathematician who happened to meet Richard Feynman and demonstrated a proof for Feynman's work (which led him to winning a Nobel). He says,

"I think it's almost true without exception if you want to win a Nobel Prize, you should have a long attention span, get hold of some deep and important problem and stay with it for ten years. That wasn't my style."

Here's a list of Dyson's awards: Heineman Prize (1965), Lorentz Medal (1966), Hughes Medal (1968), Harvey Prize (1977), Wolf Prize (1981), Andrew Gemant Award (1988), Matteucci Medal (1989), Oersted Medal (1991), Fermi Award (1993), Templeton Prize (2000), Pomeranchuk Prize (2003), Poincaré Prize (2012).

I think it's in fact very true and there is a very good reason what that is the case.

Most expert know more and more about less and less because in order to understand something you need to dig ever deeper to understand the specifics.

What is needs is expert generalists that are able to understand several fields well enough to see where knowledge from one area can lead to understanding in others and vice versa.

You don't really understand a field without digging deep. A great example is how much students are willing to trust surveys before and after they do a large one. Another is mice running mazes, the numbers may look nice in theory, but they can hide a lot of problems.

I totally agree but the question is what constitutes a field.

I once read about a whole series of areas where solutions to issues in one field happened from understanding from understanding another field. Unfortunately ver few people span over multiple fields and can call themselves experts.

So the questions is whether we could spread out the expertise to more in between studies and letting go of literature as the only way to collect knowledge might be a great first step.

Honestly, University's or other places where people in diverse fields collaborate seem like the solution to this problem. Because, there are a lot of fields out there and 2^N sucks, but conversations take less effort than a deep dive. Bell labs comes to mind as a private sector version of this.

The problem is that this collaboration does not happen because people are mostly "trapped" within their own field. My point is simply to remove the idea of literature as the container of knowledge as if knowledge is like literature (i.e. once written saved as a piece of knowledge)

Or put another way.

Instead of modeling knowledge after the way our brain best deal with structure we should leave the structuring to the machines and start approaching it more like an organism that can be explored. I think we are bound to see something along those lines soon.

Expert generalists are precisely the thing that's missing, especially with how younger scientists are trained. It's harder to fund an expert generalist.

Freeman Dyson is a great example because at 89 he published the solution to the prisoner's dilemma, decades later than the original theoretical physics work.

Exactly!

The Ph.D. process weeds out anyone who has any desire to become an expert generalist.

> the solution to the PD

Dyson truly proposes a great solution lying in PD. That is a huge oversight for game theorists when they let an outsider do that.

However, it's not the ultimate solution. It's an extremely interesting aspect (a strategy, a style of play) of the PD that game theorists somehow have never seen and articulated in maths.

For the evolutionary fitness of the extortioner solution that Dyson discovers:

Evolution of extortion in Iterated Prisoner’s Dilemma games http://www.pnas.org/content/110/17/6913.abstract

Maybe it worked for him; I have found that people who tend to do the most important work in my field are dedicated and believe in their work, instead of perpetually looking for something new but unimportant to publish.

What was Einstein's single idea that he worked on? The photoelectric effect, for which he won the Nobel prize, or special and general relativity? Or statistical mechanics like Bose-Einstein statistics? Because it seems to me like he worked on many ideas during his life.

the nature of spacetime and gravitation seem to have followed him to his death.

Certainly. But it was not the only idea he worked on, as I commented.

Perhaps I'm reading too much into it, but I interpret "Most of the great scientists worked on a single idea their entire life ..." as working on a single idea, to the exclusion of others.

Otherwise the statement would be "Most of the great scientists worked on an idea their entire life"

FWIW, using the list from http://listverse.com/2009/02/24/top-10-most-influential-scie... :

Marie Curie: received Nobel Prizes in Physics for her work in radiation and in Chemistry for her work in radium and polonium. If the idea here is "radiation" then Einstein's idea was "physics" and Darwin's was "biology"

Alan Turing: died entirely too young. Best known for early work in computers. Then switched to mathematical biology, specifically morphogenesis.

Niels Bohr: perhaps "nature of the atom" or "quantum mechanics"

Max Planck: Black-body radiation and special relativity

Charles Darwin: evolution (not really a small idea)

Leonardo da Vinci: not applicable

Galileo Galilei: astronomy (I can't think of a smaller "idea" for his body of work)

Nikola Tesla: umm, "electricity"?

Albert Einstein: see above

Isaac Newton: optics, gravitation, physics .. and then the Royal Mint, counterfeiting, and a whole lot of alchemy.

It's hard to conclude that these people worked on a single "idea", though some worked mostly in a single field.

Most of these people didn't even work in a single field... Tesla, Newton, Planck, Curie, Turing, Galileo...

Yes, I believe that is my point.

I may have not stated it well, i did not mean an idea to the exclusion of others but rather that they but that they didn't stop working on every single idea they took upon for a long long time. From the list you give, there arent any scientists who put their research on a subject "to bed" , unless it proved a failure.

Umm, Newton? The latter part of his life was research in alchemy and biblical chronology.

If you say that Newton "worked on a single idea [his] entire life", then what was the idea?

In any case, the observation is incomplete. All great scientists slept at least once a month doesn't mean that sleeping at least once a month is a distinctive attribute of great scientists.

If most great scientists work on a given topic all of their scientific career, is that not mostly because most scientists do the same?

That's not true -- not even for Einstein. His work on relativity was arguably his greatest, but remember that he won the Nobel for his work on the photoelectric effect. He also did a lot of work on particle movement, i.e. Brownian motion.

Exactly.

We often have the idea of "solved problem" in science, but very few of them in software development (overgeneralizing a bit, any field of technology). This is probably because science is often about "what" (what is the fastest algorithm for matrix multiplication?), "whether" (does P = NP?), while technology is often about "how" (how to implement matrix multiplication efficiently?).

Once you solve a problem in science, you solve it for good. But in technology there can always be "better" ways to solve it, and things keep evolving.

When Darwin "solved" the question of evolution, he did not solve it for good. There has been a lot of work to make it a better, deeper, and more powerful mechanism for understanding biology.

This is true for just about all research.

Science doesn't solve problems in that sense, Darwin didn't solve the evolution question, he proposed a successful theory to explain it and that core idea remains true though modern biologists have gone much further than Darwin ever could as his time was before DNA. To use the word solved just wouldn't be correct in any sense of the word.

Sure, that's called engineering, and applies to far more than just technology. Engineering is the practical application of science.

can you give an example of any problem in science (not engineering) that is definitely solved?

All that is known in the entire field of mathematics. But yeah, mostly just mathematics. I think sorting algorithms is a solved problem, but you can say that's just math as well.

Even math systems are engineered somewhat: we can always invent a new and better math. It has happened a few times in history.

Well, Newton's laws of motion suffice for non-relativistic motion. Maxwell's equations, I think, suffice for typical electronic/magnetic fields. Judging from all the labs I did or TA'd back then, we have a pretty good handle on friction and inclined planes. While there's a lot of "science at the edges" that's ongoing, much of the everyday, classical things have a suitable solution.

As others have written, modern Engineering could not be done without the existence of such working solutions.

That's just simply not correct - scientific endeavour and the scientific method is an ongoing prospect. You never actually solve anything, instead you refine the parameters in which a given solution is deemed correct.

Newton "solved" the equations of gravity, Einstein "solved" them too, and now quantum mechanics is "solving" them yet again. None are wrong, but also none of them make the problem solved.

The idea of a "solved problem" in science is a dangerous one IMHO.

I should have used a more accurate word than "problem".

There are two sorts of problems in science: coming up with assumptions and answering questions under these assumptions. I was thinking about the second category. Modeling gravity falls in the first category, where problems are a lot like engineering problems - there is no final answer.

However, problems in the second category often have final answers (let's not bring Gödel's incompleteness theorem to the table) once you bring all the assumptions with you. Think about: in classical mechanics, what are the possible planet trajectories? This problem has already been solved, and since it is solved, it is solved for good. Later students can simply learn the solution by heart. You can, of course, insist to go through all the trouble of finding out the answer - that is what physical majors often do anyway - but the point is that you don't have to as long as you trust the science community.

In software development you don't have this luxury. First, you can almost never trust libraries to be 100% bug-free; second, even it really is correct, you always suffer from performance costs from invoking the library. There is no such thing as "performance" in scientific knowledge; indeed, a short proof is better than a long one as long as they are both correct, but they are just as useful in that they show the solution is correct.

I think that is true, which is why what this author should be suggesting is changing the incentives of what the 'output' of a scientist should be.

Literature is great because it measures how far you're progressing the measuring stake of knowledge in a field. Old distribution models like journals just help disseminate that "hey everyone, here's the new mile marker". You then get paid based on how far you manage to advance it.

I think while that should remain true, it can definitely use some rebalancing. It's incredibly lopsided and inefficient, and too many nodes in the process are hoarded and centralized. In this case, I think science should more follow art.

To me, the main problem with papers in their current shape is that they are required to be more or less self-contained. When one wants to state a result that improves a little bit the knowledge in a well established field, he has to waste time and space stating the definitions and preliminary results necessary to understand his result. This is counterproductive both for the author and the reader interested only in the small new bit of information.

If papers were collaborative, one could simply propose his improvements directly where they fit in the reference paper, without having to write one from scratch, and readers would be immediately aware of these follow-up results, without having to search in dozens of papers.

I don't know. I think the act of forming a paper up from scratch is an essential element in the process of research. It really isn't a waste of time and space in the sense that proper research will require these definitions and preliminary results be carefully reviewed by the author anyways. Why not have them type it out for us? Or for their own sake even.

A wiki style database of information could be an exciting resource for kicking off new ideas. When scientists conduct research within this database though, findings and methodology must be to a high enough standard. Marks of high quality research include well defined terms and contextualized prior results.

If you want to get straight to the new information, read the abstract up top, skim the middle and read the results and discussion.

I completely agree that serious researchers should review definitions and results they are basing their findings upon. But then, I'd find much more reliable an editable reference paper corrected and improved by a significant number of people working in the field, rather than an old-style paper, published decades ago, after a (botched ?) review by a couple of anonymous reviewers, and not revised since then.

This would address also a little bit the problem of notations: if everybody agree and work on the same piece of work, they are likely to adopt the same notations, providing a nice coherence to the user.

A wiki style database has many advantages when it comes to organizing and searching information. In my opinion, though, the editing process should be closer to GitHub's pull request (as advocated by the article), to ensure that everything is properly reviewed before publication.

Finally, a publication scheme like the one I described also address a recurrent issue with traditional citation-based papers: citations are one-sided. It is easy to see which papers one article depends upon, but the converse is hard (unless using specific tools, at least). With collaborative editing and wikipedia-style links between articles, the reader is immediately aware of the latest findings in the field, which simplifies tremendously the bibliographic research.

There is a tradeoff here. If you want terse, field-specific papers without introduction and/or definition of terms they will be essentially incomprehensible for those not knowledgeable about the field. If you want things to be more generally accessible, a brief (!) explanation of the point and basis of the paper is helpful.

That said, I would like to see papers that actually provide a return for the time spent reading them. Teach me something that I can use if I'm familiar with the discipline. Make sure it's actually generally applicable rather than an artifact of the data. Do proper statistics and/or testing of the idea. Show it's not just an LPU (least-publishable unit) that resulted from running tutorials from the vendor.

There's a tremendous push to publish as it's the currency of academic and much professional life. I've done methodology all my working life, and the good papers are joys to read. They provide insight for new techniques, things which I can understand and apply and build upon. There's test data, so I can check that the paper and any work I do based on the paper is robust.

They unfortunately quite rare. I think editors use me as a hatchet man for me-too papers, or that's all people write anymore. Yeah, the goal is to get one's students trained and employed and the grants obtained but please, please write things that are worth the time of reading them.

While not essential to the argument, it uses software development as an analogy for scientific publications. Unfortunately, that analogy isn't correct.

> Just as the software industry has moved from a "waterfall" process to an "agile" process—from monolithic releases shipped from warehouses of mass-produced disks to over-the-air differential updates—so must academic publishing move from its current read-only model and embrace a process as dynamic, up-to-date, and collaborative as science itself.

"Agile" is one name for a long line of software processes, which includes XP, RAD, and Unified Process. Iterative development stretches back to the dawn of software engineering. In modern practice, "agile" seems used for anything which is even a touch more flexible than the strictest of waterfall. Take your process, add a standup meeting or story cards, and poof, you're agile.

Nor is "agile" development tied so closely to 'over-the-air differential updates'. The development methodology has very little to do with deployment. Perhaps the easiest example is to look at the various language implementations (Python, Ruby, gcc, etc.). They don't have auto-update. So I guess the question is, are they agile?

> But academic literature makes no distinction between citations merely considered significant and ones additionally considered true. What academic literature needs goes deeper than the view of citations as kudos and shout-outs. It needs what software engineers have used for decades: dependency management.

> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing.

I have no idea of how to make this work. I give as an example the Kabsch algorithm algorithm for RMSD minimization of two molecular structures (see http://en.wikipedia.org/wiki/Kabsch_algorithm ). The original paper contained an error that sometimes resulted in a sign error. This was fixed in a subsequent paper.

Most cite the first paper, because it introduced the core concepts. Some cite the first paper but actually implement the correct solution from the second paper without even realizing there was a problem, because they combined the principles with their own mathematical understanding rather than duplicating the exact steps in the paper. Still others cite both papers.

How does this hypothetical dependency graph incorporate those details?

That paper is a simple one, because it really is a single algorithm for a single goal. Many papers introduce multiple concepts, solutions, or observations. It can well be that one of them was wrong while the others were correct, interesting, and influential. For example, in one paper I read the authors did not include a test case because, they said, the source material was invalid. I investigated, and discovered that it was due to a transcription error by the authors of the paper, and not a problem in the source material.

The addition of that one data point would have no material impact on the overall paper. But it's still wrong.

How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are?

Bear in mind that there's been over 50 years of attempts to do this, so I think there's no easy answer.

You make some very good points. I was hoping HN would pick up the story, but it didn't. It reminded me a lot of

http://marciovm.com/i-want-a-github-of-science/

edit: Oh, never mind. The story did pick up! I don't get HN's rankings at all. Usually if you don't pick up within the first 30 minutes, the story is forgotten forever.

> How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are?

How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are?

That's how we do it in programming. I think the analogies could be stretched to scientific publishing.

Could you be a bit more concrete as I am having a difficult time trying to understand how this would work?

To start with, if there is no activity in the bug tracker, doesn't that indicate success in the paper, because it contained no problems? Or should I put in a few silly errors (misplaced commas, typos in the citation list, etc.) in order to increase the bug activity?

You suggest that programming prestige is based on the activity of the bug tracker. Can you give some examples? As an admittedly extreme case, TeX has has very few bugs, and has no bug tracker, but is a well known project.

Bug trackers only work for active projects. If the main author writes a paper in solid state physics, is awarded a PhD, and gets a job working for Seagate on high density magnetic compounds for storage tapes, then why should the author care about maintaining the paper's bug tracker?

But getting back to the topic, the dependency graph for programming is extremely coarse grained. For example, here is a recent bug for Python - "integer overflow in itertools.permutations" - http://bugs.python.org/issue23363 .

Given your software dependency graph, can you tell me which programs that depend on Python are affected by that bug?

Because that's the sort of thing that Brian Christian (the author of this piece) wants for scientific publications:

> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. And it would tell us, at a click, which other ideas are likely to get swept away with the rubble of a particular theory. An academic publisher worth their salt would, for instance, not only be able to flag articles that have been retracted—that this is not currently standard practice is, again, inexcusable—but would be able to flag articles that depend in some meaningful way on the results of retracted work.

How do I indicate "in some meaningful" that my program depends on itertools.permutations() not having an integer overflow?

Every paper has problems. Yes, even typos should be mentioned. Why not? They might trip up someone.

Perhaps instead of structuring the whole thing around papers, it should be something like Wikipedia and editing articles in it.

I still don't see a concrete explanation of how this would work. Instead, I see you tossing off possible solutions that don't pass even a basic test of feasibility.

Is bug tracker activity a sign of a good paper? Or a poor paper?

The use-case I objected to, from the original article, is:

How would structuring it "something like Wikipedia" make this use-case more feasible? It seems instead like you are talking about a completely different topic.

Why do we need all things to indicate paper quality? A lot of bug tracker activity might be both a good or bad sign, but more important that's not relevant.

What one should aim for is improving research quality and productivity.

Isn't it possible to consider each little 'chunk' of knowledge, data, information as a node in the dependency graph, as opposed to one whole paper?

We could also have a hierarchy to represent layers and composition of knowledge.

Not to say it is easy, but it looks quite possible.

While it's certainly possible, in an abstract sense, what is a "chunk"? Who gets to decide? How do we resolve disagreements?

There is a vast set of reasons for why something is cited. "This argument fails if reference X is wrong" is different from "I cite X because my professor/grant reviewer wrote the paper" vs. "I don't want to repeat something I described in a previous paper" vs. "I used the same training set described in X but for different purposes" vs ...

Or, suppose I write a paper which analyzes the evolution of how the term "gene" is used over the 1900s. It's unlikely that every detail that might be analyzed will be dissected into its individual components. And if it were, that's an awful lot of work, with little chance automation support (especially if some of the reasons, like promoting one's earlier work, are gauche to state explicitly).

These the concrete problems that make it really hard to convert the "possible to consider" into something practical.

> Who gets to decide?

The authors under a community guideline. Editors should get to edit as appropriate.

> There is a vast set of reasons for why something is cited...

Adding reasons as a label to the citation link in the dependency graph would be very useful.

You seem to be describing a high-maintenance search engine, if I'm not mistaken.

The thing is, to my knowledge most academics keep up to date on most everything that is happening in their field; they don't need this high level overview because they don't communicate solely through journals.

It's only if you don't have the view that it's useful; which is obviously a problem for PhDs, but I'd question whether it's worth the necessary annotation investment (hours per paper for good classification, I'd imagine).

You are being entirely too abstract.

Once you try to embody these abstract ideas into concrete form, you'll discover that it does not work. Or rather, the gains are limited compared to the cost of creating, curating, and maintaining the extra metadata, and the error rate will be high.

"a label to the citation link in the dependency graph ..."

To start with, here's the Citation Typing Ontology: http://www.essepuntato.it/lode/http://purl.org/spar/cito .

I then write a paper which says that approach X is generally three times faster than Y, and http://purl.org/spar/cito/citesAsEvidence paper ABC.

Now, paper ABC is the one I mentioned earlier, which omitted a data point because of a transcription error by the authors. This is not a serious error, but it is a mistake. So you write a paper which http://purl.org/spar/cito/corrects paper ABC.

How does this automated system of scientific dependencies supposed to figure out that my paper is not invalidated by your correction to paper ABC?

"... would be very useful"

It may well be useful, but it can't be used to give what the OP wants. My comments concern the infeasibility of the proposed system, not the feasibility or usability of other projects which are in the same general area.

The two things that are most ridiculous about science in this day and age (imo, I work in the field)

1) Tons of papers get published without the underlying data. Why is there no easy system in place where I have to upload all data from experiments etc. and if the data isn't there to be verified -> questionable research at best (or in the current framework of science autoreject)

2) Paywalls and the entire publishing system. Most science is state funded in some way, shape or form and as such it should be available for free for all citizens.

I think publish and improve is an ok model but there's some issues like the author says it's pretty strange how the whole redacting a paper system works. It's also nontrivial to say "well I was wrong in my paper X as shown in papers Y and Z" (some sort of HTTP redirect to X/Z would be in order)

Maybe I'm just not aware of it but it would also be interesting to have a huge Dung-style argumentation system of papers for fields (X attacks Y, Z defends Y etc.). Might be a neat project if it doesn't exist.

These things take time and money to build and maintain though, and there's very little of that to go around unfortunately.

It seems the problem is that the author is trying to create a representation of knowledge, whereas research papers are more of a logging of work done.

His approach may be a perfect fit for an experiment in meta-research. You could run periodic reviews of articles released in specific topical areas and create summaries of the findings. Any time those findings change, its a git commit, and you can track this change over time. It's something like Wikipedia, but designed for research. I would not be surprised if this already exists.

I see a challenge in figuring out the document structure which will be the most conducive to distributed version control (pull requests, etc), and can provide some insights on a historicalbasis. For example, you could run these summaries retrospectively, say looking at DNA research in the 1960s, and do a separate commit for every key finding through the years. It seems to me that you would pretty much have to specify a specialized coding language for scientific knowledge for this structure to work.

This sounds like an update to the format of textbooks. They sort of work like this already: as new versions come out, they include the latest revisions in the field. On the other hand, they also serve their publishers by putting out new versions with little or no improvement to continue earning money.

Exactly. The problem is that papers aren't worth what the rest of the world think they are. Nobody who is working on the cutting edge of a given field give a damn about the papers that are published in the field's major journals. Papers are little more than permanent records of what people talked about at some conference several months before, and through other informal channels even earlier. By the time they're published, they're already old news. They may be worth some archival value, but that's about it.

Unfortunately, the rest of the world thinks of papers as the primary method by which scientists exchange ideas. This is a myth. Sure, there some scientists (mostly in developing countries, or those coming from a different field) who rely on papers to figure out what their colleagues are up to, but if so, that's only a reason to improve real-time communication, not a reason to turn papers into real-time communication tools.

This and the previous comment are /exactly/ right, especially: "... that's only a reason to improve real-time communication, not a reason to turn papers into real-time communication tools".

When all you have is a hammer, everything looks like a nail. The hubris on display here is: "it works for me, it must be right for everyone". What the author is missing is that the article-reference system is actually more flexible than a tree of dependencies. Consider the obvious example: citing Wikipedia. It's a pain, because Wikipedia changes, so you have to link to a certain revision and if you want to know the author you have to trawl the contributions, which nobody does. And this is true of Wikipedia because Wikipedia is a work in progress: it's not meant to be a presentation of knowledge but a repository of knowledge.

Science works like that too, but on the blackboards and notebooks and today blogs and comments of the investigators, which lack real start and end dates and are regularly crossed out or replaced. The work is refined for publication because publications are essentially learning materials, they're written and edited to be as comprehensible as possible for any active researcher in the field to read for education or leisure; likewise in order for citations to be useful the cited works must include some background information which goes a long way in keeping the sciences connected, because it's not rare for analogous problems to pop up in unrelated fields, and it is crucial that researchers from outside each others' fields can read each others' work in order for interdisciplinary collaboration of this form to be possible. But all of this background information is just background noise to the people actively working on a problem and that's why it isn't on the blackboard or in the blog post, and it's why terse notations like dummy indices became popular in scratch work. The divide between the didactic and generative arrangements of scientific work is better explained thus: papers are binaries, not source code. Binaries don't go in source control.

Author here. Delighted to see this near the top of HN this afternoon, and to see the healthy discussion it's provoked. Happy to answer any questions as best I can if you've got 'em.

Are you familiar with the field of bibliometrics?[0] It is a not-that smalish research area that works on quantitative research studies. Qualifying types of citations have been a pipe dream there for decades. There are a lot of scientific literature on the subject out there.

[0] Or sciencemetrics, informetrics and other assorted subfields

I'm aware of portions of that literature, but haven't studied it explicitly. What do you think are the most interesting things happening in that space at the moment?

For me one of the most salient things in that space -- albeit a different point than the one raised in the essay -- is the debate about what sorts of metrics are most appropriate (if any) for informing decisions about hiring, tenure, and the like. In some ways it's quite like the be-careful-what-you-wish-for problems any business has in choosing which metrics to care about. For instance, the h-index might shape a researcher's choice of whether to publish something as one larger paper or two smaller ones.

As Sam Altman put it, "It really is true that the company will build whatever the CEO decides to measure." I think a similar principle holds for research and scholarship.

Are you a scientist? If so, in what capacity? If not, where do your assumptions come from?

My problem with your post is that it seems to be based on a faulty understanding of the actual way things work (compared to some theory on it), and furthermore is a case of 'I'm a software engineer, I can solve anything' without actually understanding the issues. Much like engineer's armchair lawyering, which is always so far off the mark that it's impossible to even intelligently debate, because it's based on so much misinformation.

Speaking from the perspective of life science, I think the idea of the "article" and "review" as two separate complimentary parts of constructing knowledge actually makes a lot of sense.

An individual article is an argument, with evidence for how some process works. A review looks at all the papers, getting the more global view of "knowledge" the OP seems to want.

Perhaps in the spirit of the article we should have reviews that are ongoing, sort of like wiki pages for various research areas. These might capture a living, complete state of the field better than current review articles.

It's really unclear how this would be done though because:

1.) How fields and areas interact is constantly changing, how do you manage the merging and splitting of review articles? You end up with a confusing view of "what we know" once again. 2.) As others have noted, who gets to write these things? The cutting edge literature is full of big personalities in conflict with one another. It takes years to figure out who is actually right.

That said, at a minimum retractions should be much easier to track. I'll bet we'd save a whole bunch of wasted effort.

Well, now Wikipedia is the review. Depending on the field, you may just do all the real research on Wikipedia and then add in citations in your paper to look professional.

How about a Wikipedia with original research? "Sciencepedia"? Each article gets a section for each editor. If you want to contribute you can either submit a pull request to an editor you view favourably or you can write your own section. Sections can be voted up or down.

Wikipedia is not the review, not in the life sciences sense of the word.

A review, despite "reviewing" the literature, offers a unique perspective, and does have an argument, although one that is based solely on the aggregation of existing evidence and literature.

Wikipedia just describes parts of the field, reviews offer insight and perspective.

There is scholarpedia, which was initially crippled by the fact that each article (which was similar to a review) would often be limited to the (invited and authoritative) author's point of view. In fact they have iterated their policies . Still their contents are often adapted copies of review articles published in "real" journals.

I think that's the spirit of what he proposes. A possible way to organize the fluid current knowledge would be as a network of scientific questions, which depend on one another (like a bugzilla for science). Having a list of articles that answer positively vs negatively to every question would give an immediate view of the scientific consensus, would allow everyone to make their own judgement about the quality of research and importantly, would pinpoint gaps in knowledge thus pointing out possible research directions.

Another thing that complicates this in the life sciences is that techniques aren't standard. If I use 8 month old rats and draw blood in the morning, I will likely have different gene expression and protein content than your study that uses 4 month old rats of the same strain and draw blood at night.

Each article includes these details, and it is very hard to know if the result is more generally true, or only true for the conditions of my study.

It is for this reason that reviews are important and difficult to replace with some constantly-updating model of information release.

Hrm. But if each article includes these details but there is no way to generalize the findings, how is a review going to help? Scientist A doing the study claims X effect when using 8 month old rats and drawing blood in the morning. Scientist B comes along and says, "looks like sound science to me, conclusion is valid!"

But Scientist B has no idea if some gene expression and protein content are the result of the blood draw time, the age of the rats, or the temperature of the lab. How can ANY review be effective if there are no other studies on 8-month old rat blood drawn in the morning in a colder room?

The problem with comparative studies here is also a problem with peer reviews, and more generally, a problem with life sciences and the idea of controls v. variables.

Cool ideas. We will probably end up with a better world if someone implements them, but I would argue that these "agile" things are much less important in scientific publications than in software development. Science and technology are different.

Software artifacts are huge and uncontrollable. Today it is very rare for a software developer to know all the ins and outs of a software he/she develops, along with all its dependencies. There is simply way too much code, and it is impossible to control them manually.

On the other hand, scientific papers tend to contain very few core ideas. It is still reasonable today for a researcher to understand everything he/she publishes, along with every idea in all the papers he/she cites. Once you get the core idea, you get the whole paper. They merely act as mediums for conveying ideas and there are not that many of them. Often you can just "manage" them with your own brain.

Anyway, I resonate with the first few paragraphs about retraction a lot. It is frustrating to read a paper for a whole day, find out that it is wrong, and that someone already refuted it in another paper :-/

What about scientific papers that depend on software artifacts?

Either scientists somehow understands all the "ins and outs of a software he/she develops, along with all its dependencies" in order to publish a paper, where a programmer doesn't have the same understanding, or you are comparing two entirely different concepts as if they were the same.

> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing.

This is the problem with the academic form of knowledge transfer as a whole. It's too dependent on feeding of ego. And the "improvements" discussed here-in only reenforce that. If you boil knowledge down to meme, the necessity to "weight" and define pillars becomes less relevant. Without the ego's you get a remix culture where the Amen Brothers are more highly valued than say some one hit wonder... but not because we've measured them, and because we know that sample was from the Amen Brothers. It's more highly valued not because we know the source. In-fact few would know the music they are listening to has the Amen Brothers sample in it. It is more highly valued because it has taken so many forms and created actual value.

We don't need tools to allow for better attribution and knowledge measurement. We need to remove ownership from knowledge as ownership is a hindrance to the transfer and consumption of it.

Personally, what really gets me with the current publishing regime is how fashion-driven it is (at least in many parts of CS). Oftentimes, it feels like the most challenging aspect of producing publishable research at first is not rigor or novelty or writing a sufficiently readable complete paper but hitting something sufficiently "interesting"—largely a function of what happens to be popular at the moment.

Publishing has other issues, but they can be mitigated by releasing results in forms besides simple papers. (Again, I'm mostly talking about CS.) A lot of successful projects end up releasing blog posts, enhanced online resources, libraries or complete, living open source projects.

At least in CS, this is happening enough to be useful even if it is by no means universal. This even comes up in more theoretical fields—somebody recently pointed me to a website[1] on tree edit distance, which is a great example. (I really wish it had existed a few years ago when I needed the algorithm originally!)

[1]: http://www.inf.unibz.it/dis/projects/tree-edit-distance/

In biology, there are a large number of very specific niche journals such that a paper on a topic out of vogue is likely to land in a more specific journal. This is not without drawbacks, but it sort of works.

First of all, there exists "living review" journals, where authors are mandated to regularly update their texts every now and then to reflect new developments in the field covered. This seems like something the author ought to really comment on.

Secondly, while research and software development have many similarities, the idea that academic research is best conducted with methodologies created to allow efficient development of CRUD applications which nice interfaces is not very credible.

I like the concept, however I'm afraid "Going Agile" is not a solution for science, it's just more jargon.

Would be nice to see science testing several different methods though. Also, it feels like some areas of discipline might work better with different workflows. For instance theoretical physics may need a different process than structural engineering or chemistry. I think it's also weird that we currently use very similar processes across disciplines when there are surely better tools that could be made specifically for each discipline with interoperability between disciples handled on another level.

The right tools for the job would hopefully lead to more science faster.

One possibility is to keep the current publishing model but add an additional "meta" layer in a central repository to indicate the relations between the contributions of each paper.

The repository should be independent of an individual journal or academic field so that multidisciplinary collaboration would be easier and more visible. If designed well, cross-pollination of ideas between fields may multiply.

Each node in the dependency graph should not represent an entire paper, but a chunk of contribution (knowledge, information, or experimental data), with an option to delve deeper to see its source code/raw data. A hierarchy of compositional structures can represent broader units of knowledge. Citation links should allow for labels to indicate the reason(s) for citation: prior work, refutation, tool, methodology, etc.

This is a more practical transition than revamping the whole system and should still add much valuable information for researchers and consumers of research.

If realized, an automated system may be able to analyze the graph for additional insights into our knowledge structure and point to interesting research directions as well.

Sciencepedia, with hyperlinks.

Yes. Scientific knowledge structured as literature means scientific knowledge as subject fields which basically means a lot of lacking knowledge between these fields.

It's not that we should not have specific categories but rather that we should have a lot more than we have right now.

"Removing literature" as a framework for knowledge will mean the flourishing of knowledge that can only be created by combining several fields.

Baby steps. Can we just ban using two-columns in PDFs?

That single change will speed the rate at which new results can be read and understood.

Forget PDFs. JOVE and other video scientific journals should become the norm. Much less chance of fabrication and much better replication if you can demonstrate on video.

There has also been a big push to only count "recent" scientific literature as valid for reference in some scientific fields. I know that this helps keep any new papers "fresh", but I have always had a bitter aftertaste about the way that still-valid, but older, research is looked down upon.

I see his point. What's true of scientific literature is true of all literature. But the solution doesn't match the ambition of the article's title. Charts and graphs, and all the high tech tools available are certainly potentially beneficial, but what's needed is a new product and center of culture, and that's a function of ethics. Those are always defined in terms of the challenge of their day. But tools with out ethics are dangerous indeed.

I had a similar article written once on lawmaking. In general, many fields will benefit from the advances in software-defined collaboration.

https://www.linkedin.com/pulse/20140814084517-815484-lawmake...

The author clearly doesn't understand academia and scientific work.

> It is time for science to go agile.

Goodness gracious!

This solution is like trying to stir a pot of chili with an absinthe spoon.

I'm not sure why there's this holy grail of "the unified master version of" whatever.

Let me give an example. Say I write a paper on the shape of the non-dark-matter (stellar) density in the milky way by looking at y-type stars and I get an answer x. Now Bob comes along and looks at y2-type stars and gets answer x2. People have the idea that you just go to the first paper, add a footnote to y and x showing alternate values for y2 and x2...

But what that doesn't take into account is the fact that I used telescope a (a 10 meter hawaiian behemoth) and pointed it in one beam of the sky for 8 hours to get an ultra-deep pencil; but bob used telescope a2 (a modest 1.8 meter in la palma) that took an all sky survey and only goes very shallow. Now we add this in a footnote.

Next, there's a critical difference in the stars we studied. My y-type stars take 8 Gyr just to form, but Bob was using y2-type stars which live anywhere from 100 Myr to 15 Gyr. So I'm looking at the old stars and he's looking at all the stars. Since we know that different age stars live in different parts of the galaxy (old in the halo, young in the disk and bulge), our results are starting to look not as comparable as we thought... but it's minor, we'll add a footnote.

But then we realize that, since my old stars are giants and his all age stars are dwarfs, my stars are way brighter than his. Since my telescope is monstrous, and his is a small surveyor, my stars actually end up being observed to a distance 10 times that of his sample. In fact at these distances, the original model is a bad fit and we need to change from a power law to an Einasto profile. Bob can do that too, so our answers are easily comparable, but the Einasto law has more parameters so it would give a worse fit per parameter value than the power law he wanted to use originally... We add an appendix to the paper to explain this bit.

Then I notice that Bob's been using infrared data, and in the infrared there's a well known problem separating stars and galaxies in the data on telescope a2. In fact, Bob has to write a whole new section on some probabilistic tests and models he uses to adequately remove these galaxies from his y2 star sample. My telescope, observing in the optical at high resolution, has no such problem, so that section doesn't exist in my paper. Bob looks around awkwardly and stickies a hyperlink to his meta-analysis somewhere in my data section.

Then Jill comes along and says she doesn't agree at all with us; she got value x3 using the distribution of dwarf galaxies and if you believe in theory z, then _hers_ is the most accurate answer.

And we tell Jill to go write her own fucking paper.

The title made think this article was going to be about how to write science. Writing of all kind benefits if you incorporate elements of a story.

Stop writing filler text.

Reply on news.ycombinator.com