A commit history of BERT and its forks
amitness.com
amitness.com
We'll see how long that particular revolution in communication takes.
Once / if we have all that information in even a loosely structured but plaintext format (whatever that may be ... JSON? XML? semantic HTML? MHTML? TeX? as long as both images and text can live in the same file), there will be another revolution in accessibility, ease of distribution and accountability.
Who knows, we might even stop mentioning GitHub as a best practice for that and use something distributed, blockchain-like.
Imagine a future where citations of previous work looks like "As H.G.Wells et all mention in block 4f98aaf1 document #391, the foo is better than the bar." with hyperlinks of course. Peddling my own wares, it can be done with something like https://github.com/ivoras/daisy .
Which is why I think a lot of burnt out scientists spend their time there editing articles.
A few weeks ago I spun up wikisurvey.org to try and make a place for Wikipedia-like collaboration on research. There's nothing on there yet, but I'm hoping it will enable low-latency collaboration once we get the ball rolling.
I dunno, it’s easier for me to describe and to remember “Radin’s 801 paper” or “the ultimate papers” or such.
And it doesn't really incorporate the diff concept in any way atm, but I imagine it's up their alley at least.
The greatest concern with any web-based solution is reliability. A PDF might not be as easy to grasp as a web-based solution with live examples and graphs that make the content more approachable but it is still worth gold since I can back it up and have it available whenever I need it. Web-resources on the other hand can become unavailable at a moment's notice.
This is a really common way of reading scientific papers, but I think it is ultimately harmful: I believe it contributed to the replication crisis as well as terrible scientific journalism in the popular press. Current scientific papers certainly do not meet this standard and I'm not sure it's even possible or desirable to require that they do.
Ultimately, I think we just have to accept that scientific papers only really make sense as part of a large body of scientific literature, to which they are responding and which respond to them.
The problem is a lot of people would rather just pick one or a few papers, cite them to justify a claim with total certainty, and then move on. I worry about encouraging this tendency.
Research is non-linear by nature. One of the reasons research code is usually not well written is because of this same fact - there is no clear plan (or else it wouldn’t be research). Restricting research cultures to follow this linear progress model also imposes an artificial block that may be detrimental to productivity. Alternatively, this exercise could demand a second pass over research code which cleans it up and puts it in its “place”. This may or may not work because “clean up” is a non-trivial task which the researcher may find limited time and utility for.
By no means I’m against writing clean code. In fact, I’ve had multiple discussions advocating it but it’s hard to keep everyone aligned. The primary objective of research in a fast-paced field like Machine Learning is to get the idea out the door. Unfortunately, the exceptionally compounding benefits of writing clean thoughtful code from the beginning are realized much later in this timeline. By then, it’s too late and too hard to retro-fix code.
My understanding so far (having written code for multiple research projects) is that the only way to fix this culture is to deeply ingrain the compounding benefits of clean code in new researchers. By exposing junior researchers to the compounding benefits of “clean” code, we can gradually nudge the community into a more favorable culture with respect to this “commit history” of research.
But here's where the problem lies - the proposal in the article is representative of how the community has become so used to this nice side-effect of modularity of neural networks, giving the impression that a clean linear decomposition of contributions is even conceivable.
There are examples where this modularity isn't possible (or not optimal even if possible). The poster child, I would say, is probabilistic programming. It is easy to think about building general purpose inference algorithms in probabilistic programming. However, more often than not, inference heavily relies on context.
An example for this case is Gaussian Processes. In theory, it is easy to conceive a chain of ideas that connect threads of research within this tiny field. Unfortunately, in practice it turns out inference with GPs is more efficient when we exploit structural assumptions within the context of modeling choices and general purpose few-liner code changes would be terribly inefficient. You'd find the code will look vastly different to an untrained eye even though it would compose similar sounding (or looking) building blocks.
and it is rampant.
Communities must be wary of giving into the Semmelweis reflex. Often times, the perceived "outsiders" tend to have interesting perspectives.
Another example to consider is Fermat vs Descartes - Fermat was a relatively unpopular figure at a time when mathematics was confined to closed-group elitist figures like Descartes. Nevertheless, he provided more elegant perspectives to problems which Descartes thought couldn't be done better.
Swanking into the joint, assuming spherical cows, and berating the locals for being lazy roustabouts who just need to see the light is not going to make you friends, because it's demonstrably not useful. To the extent that there's human politics and gatekeeping, it's resolved by getting ones hands dirty in the lab before one holds forth ex cathedra about things one obviously doesn't understand. It's certainly possible to take valuable ideas from one field and apply them in a productive cross-disciplinary way. The people who do it well don't play this game.
Intuitions and practices developed to handle designed and often linearizable systems for which spec sheets of some form are available just aren't applicable without heavy modification in the biosciences, which routinely deal with eking out victory in systems which are not designed, are wildly nonlinear, and for which the partial specs which do exist are known to be at best lies of omission and often just complete misunderstandings which can be, and are, revised at a moment's notice.
-Next Sentence Prediction
+Sentence Order Prediction
+Cross-layer Parameter Sharing
+Factorized Embeddings
It's why we aren't having this discussion in terms of diffs versus comments we're replying to.
The intention in the above blog post isn't to suggest that we discard detailed research papers. It's just thinking about additional mediums we can use to make papers more accessible(a good example is distill.pub and paperswithcode). As mentioned in other threads below, PDF can be a limiting medium.
distill.pub heavily favors fancy and interactive visualization over actually meaningful research content. This is not to say that the research publicized on distill.pub is not meaningful, but that it is biased to research that can have fancy visualizations. So you end up seeing a lot of tweakable plots, image-augmentations, and attention weights visualizations. It is also further biased towards research groups that have the resources to create a range of D3 plots with sliders, carved out of actual research time.
For instance, I don't think BERT could ever make it into a distill.pub post. Despite completely upending the NLP field over the last 2 years, it has no fancy plots, multi-headed self-attention is too messy to visualize, and its setup is dead simple. You could maybe have one gif explaining how masked language modeling works. The best presentation of the significance of BERT is "here is a table of results showing BERT handily beating every other hand-tweaked implementation for every non-generation NLP task we could find with a dead-simple fine-tuning regime, and all it had was masked language modeling."
To give another exmaple: I think it's one of the reasons why a lot of junior researchers spend time trying to extract structure from attention and self-attention mechanisms. As someone who's spent some time looking into this topic, you'll find a ton of one-off analysis papers, and next to no insights that actually inform the field (other than super-trivial observations like "tokens tend to attend to themselves and adjacent tokens).
I totally agree that additional media could be good. I got caught up on the "most papers could be compressed to < 50 lines" line and misunderstood the premise you were presenting.
It would allow a kind of public review process of the paper and it would float the more interested papers for the community to the "first page".
Same thing for most other preprint sites that have integrated comment sections.
Or do you mean the river of news-style, the frontpage, maybe also with upvotes, so people can better manage attention to papers?
Let's say that for a subset of scientific papers you have the possibility of specifying both the premises and results in a way that can be composed.
Let's say for example that your result is that the rate of expansion of the universe is N. Other papers could cite this result let's say through a URL of the result. Other papers could then use this URL as premise for their results and we could create an automatic system that would notify all this papers if the result has changed after new data for any of the elements of the chain. Scientists could be notified that they should revise their own papers to see if their conclusions change with the new data so other papers depending on them can be notified. A paper could be even marked stale if after the change of one of the premises the authors have not confirmed that the conclusions are still valid, this staleness would propagate down the chain
A very simplified structure of the data would be sth like this:
{ premises: ['urlOfResult1', 'urlOfResult2',...], conclusions: [RateOfExpansion: > N] }
This is obviously terribly simplified, I suppose it'd take me a lot of time to clearly explain in more detail how such a system could work and I thought about it a long time ago.
It could be interesting to apply this to other fields, for example public policy: In this case let's say that we have created this law because of this piece of data. The data changes, we could be notified that perhaps the law could benefit from a new look.
{ premises: ['urlOfResultLeadIsNotDangerous'], conclusions: ['PeopleCanUseLeadInPipesAsMuchAsTheyWant'] }
Such a system could be made even more generic. Probably people have already worked in this kind of systems but I never took the time to investigate it. If someone knows of examples of this kind of systems applied I'd be very happy to know more
Instead of a centralized, URL-based solution, we do the fact comparisons locally, allowing each user to decide what source of facts he or she considers "canonical." The facts and their sources are recorded in that user's writing going forward so the next user has the same access to sources.
[0] http://mimix.io
Edit: Remove the question about contact info since you already answered in your comment in another thread :)
To accuse a research group of shingling means that you think that that group releases a lot of papers one after the other which have a lot of overlap between them and back-cite each other-- and that this is done to artificially boost publication count and citation count to make the group look prominent.
By that, I mean the date to use to interpret any wording like "current" or "recently" or "yet" used inside the paper.
For some reason preprints often have no visible date on them at all, and automated datestamps can be misleadingly recent, if someone makes a minor change without rewriting the whole thing.
https://en.wikipedia.org/wiki/Special:History/Wikipedia:No_o...