Of course. The obvious examples are probabilistic, e.g., Monte Carlo methods, as an increase in the problem space decreases the number of samples needed.
I’m surprised this is downvoted. I liked this paper (I’m mildly biased since I’ve been doing generative modeling work and am very excited about flows) but was disappointed to see MNIST and CIFAR-10.
I don’t know what you mean by “too research oriented.” A common complaint in LLM research is the poor quality of evaluation metrics. There’s no consensus. Everyone wants new benchmarks but designing useful metrics is very much an open problem.
It’s useful when you’re modeling a structure, e.g., P(x, y) or P(x | y), rather than a distribution. This is helpful in situations where generating samples is bad (CRFs) or your data imposes dependent features (HMMs). It appears from the README (and the authors’ previous work) this is useful for arbitrary structures too.
For me, the calculus is simple: is the platform maker ensuring their work is merged upstream? If not, what’s the point? The field moves way too fast to rely on forks. Relying on Intel’s comparatively unmaintained PyTorch fork (stuck on PyTorch 1.10) is foolish. Hell, Apple, despite their reputation, has done a better job working with the PyTorch organization than Intel.
There’s a Grand Canyon sized chasm between creating a “low pressure superconductor” and creating a room-temperature superconductor and, unfortunately, Evel Knievel wasn’t a co-author.
This method was frequently used prior to the ubiquity of dummy tokens. XLNet was the paper that introduced me to this idea. I believe it’s been in PyTorch since 2019/2020. I would not be surprised if someone finds an earlier reference.
I’m surprised by the pompousness in the OP. Especially about something that most people who do transformer research understand. I’m also surprised that so many in the replies are taking the position of “this is what research should look like” when this is clearly an example of why research doesn’t work like this. Peer review is good for many things and one of those things is saving yourself some embarrassment.
Yes (I’m a numerical analysis researcher and wrote a handful of ubiquitous mathematical packages)! The implementation of even primitive types can vary considerably! It’s way too much to hide. Nevertheless, I understand I’m biased. :)
No. That’s the issue. Without moderators and posters, there’s no subreddits. Without Reddit, Inc., there’s no subreddits. In my experience, when trapped in a causality dilemma, it’s preferable to be thankful for both the chicken and the egg and not ask too many questions.
Yes. It’s very important. Think of it as quantifying whether a model understood the objective. This is, as you’d expect, especially important for auto-regressive models.