Word2Vec received 'strong reject' four times at ICLR2013

If so, frankly I think it makes Mikolov sound pretty insecure.

However, there is another statement by Quoc, saying this is not true: https://twitter.com/quocleix/status/1736523075943125029

> We congratulate Tomas on winning the award. Regarding seq2seq, there are inaccuracies in his account. In particular, we all recall very specifically that he did not suggest the idea to us, and was in fact highly skeptical when we shared the end-to-end translation idea with him. Indeed, we worked very hard to make it work despite his skepticism.

So, word against word. I'm not accusing anyone of lying here, one of them probably just misremembers, but this leaves also a somewhat bad taste.

Cosine similarity of the sum of different word vectors sounds soooo dumb nowadays imo

https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints

I've also reviewed and rejected my share of papers where I could tell there is a seed of a great idea, but the paper as written just isn't good. It always brings me joy to see those papers eventually published because they're usually so much better.

I actually disagree, but maybe not for the reasons you're expecting. I actually disagree because the reviews are unreasonably reasonable. They are void of context.

It's tough to explain, but I think it's also something every person who has written research papers can understand. How there's a big bias between reading and writing and how our works are not written to communicate our works as best as possible, but in effect how to communicate to reviewers that they should accept our works. The subtle distinction is deceptively large and I think we all could be a bit more honest about the system. After all, we want to optimize it, right?

The problem I see is all a matter of context. Good ideas often appear trivial once we see them. Often we fool ourselves into thinking that we already knew this, but do not have meaningful evidence that this is true but may try to reason that x = y + z + epsilon, but almost any idea can be framed that way, even breakthroughs like Evolution, Quantum Mechanics, or Relativity. It is because we look back at giants from a distance but when looking at works now don't see giants, but a bunch of children standing on one another's shoulders standing in a trench coat. That is the reality of it all. That few works are giant leaps and bounds, but rather incremental. The ones that take the biggest leaps are rare, often rely on luck (ambiguous definition), and frequently (but not always, especially considering the former) take a lot of time. Something we certainly don't often have.

We're trained as scientists and that means to be trained in critiquing systems and letting questions spiral. Sometimes the spiraling of questions shows how absurd an idea is but other times it can show how ingenious it is. It's easier to recognize the former but often hard to distinguish the latter. It is always easy to ask for more datasets, more experiments, and such, but these are critiques that can apply to any work as no work is complete. This is especially true in cases of larger breakthroughs, because any paradigm shift (even small) will cause a cascade of questions and create a lot of curiosity. But as we've all written papers, we know that this can often be a never ending cycle and often is quite impractical. The complaint about Table 4 is a perfect example. It is quite a difficult situation. The complaint is perfectly reasonable in that the question and concerns are quite valid and do warrant further understanding. But at the same time they are unreasonable because the requisite work required to answer these is not appropriate for the timescale that we work on. Do you have the compute or time to retrain all prior works to your settings? To retrain all your works to their settings? Maybe it doesn't work there which may or may not be evidence that the other works are just as good or not. What it comes down to is asking if these questions being answered could be another work in their own right. I'm not in NLP as deep as I'm in CV, but I suspect that the answer is yes (as in there are works that have been published answering exactly those questions).

There are also reasonably unreasonable questions in other respects. Such as the question about cosine distance vs Euclidean. This is one that I see quite often as we rely too deeply on our understanding of lower dimensional geometries to influence our understanding of high dimensional geometries. Such things that seem obvious, like distance, are quite ambiguous there and our example is the specific reason for the curse of dimensionality (it becomes difficult to distinguish the furthest point from the nearest point). But this often leads us in quite the wrong direction. Honestly, it is a bit surprising that the cosine similarity works (as D->inf cos(x,y)-> 0 forall x,y in R^D because any random vector is expected to be orthogonal meaning that to get cos(x,y)=1 means y = x + epsilon with epsilon -> 0 as D->inf. But I digress), but it does. There definitely could be entire works exploring these geometries and determining different geodesics. It is entirely enough for a work to simply have something working, even if it doesn't quite yet make sense.

The thing is that science is exceptionally fuzzy. Research is people walking around in the dark and papers are them communicating what we have found. I think it is important for us to remember this framing because we should then characterize the viability/publishability of a work not as illuminating everything but if the things found are useful (which itself is not always known). Because you might uncover a cavern and then it becomes easy to say "report back when you've explored it", but such an effort may be impossible to do alone. It can be a dead end, one that can take decades to explore (we'll always learn something though) or it may lead to riches we've never seen before. We don't know, but that's really what we're all looking for (hopefully more about riches for humanity than one's self, but one should be rewarded for sure).

This is why honestly, I advocate for abandoning the journal/conference system and leverage our modern tools like OpenReview to accelerate communication. Because it enables us to be more open about our limitations, to discuss our failures, and write to our peers rather than our critics. Critics are of course important, but they can take over too easily because they are reasonable and correct, but oft missing context. For an example, see the many HN comments about a technology in its infancy where people will complain that it is not yet competitive with existing technologies and thus devalue the potential. Oft missing the context that it takes time to compete, the time and energy put in by so many before us to make what we have now, but also that there are limits and despite only being a demonstration the new thing does not have the same theoretical limits. The question is rather about if such research warrants more eyes and even a negative result can be good because it can communicate that we've found dead ends (which is something we actively discourage, needlessly forcing many researchers to re-explore these dead spaces). There's so much more that I can say and this is woefully incomplete but I can't fit a novel into our comments and I'm afraid the length as it is already results in poor communication to the given context of the platform. Thanks anyone who has taken this time.

If you ask people funding research I'm pretty sure they'd prefer to fund influential ideas than non-influential "high-quality" paper production.

The fact that the first doesn't lead more to the second is a failure of the system.

This is the same sort of value system that leads to accepting job candidates with neat haircuts and says the right shibboleths, versus the ones that make the right bottom line impact.

Basically, are "good" papers that are very rigorous but lead to nothing actually "good"? If your model of progress in science is that rigorous papers are a higher probability roll of the dice, and nonrigorous papers are low probability rolls of the dice, then we should just look for rigorous papers. And that a low-rigor paper word2vec actually make progress was "getting really lucky" and we should have not rated the paper well.

But I contend that word2vec was also very innovative, and that should be a positive factor for reviewers. In fact, I bet that innovative papers have a hard time being super rigorous because the definition of rigor in that field has yet to be settled yet. I'm basically contending that on the extreme margins, rigor is negatively correlated with innovation.

There's an article I came across awhile back, that I can't easily find now, that basically mapped out the history of our current peer review system. Peer review as we know it today was largely born in the 70s and a response to several funding crises in academia. Peer review was a strategy to make research appear more credible.

The most damning critique of peer-review of course is that it completely failed to stop (and arguably aided) the reproducibility crisis. We have an academic system where the prime motivation is the secure funding through the image of credibility, which from first principles is a recipe for wide spread fraud.

For example, a reviewer essentially insisting that nothing is worth publishing if it doesn't include a new architecture idea and SOTA results... God forbid we better understand and simplify the tools that already exist!

For example just see this from the review of f5bf:

"The main contribution of the paper comprises two new NLM architectures that facilitate training on massive data sets. The first model, CBOW, is essentially a standard feed-forward NLM without the intermediate projection layer (but with weight sharing + averaging before applying the non-linearity in the hidden layer). The second model, skip-gram, comprises a collection of simple feed-forward nets that predict the presence of a preceding or succeeding word from the current word. The models are trained on a massive Google News corpus, and tested on a semantic and syntactic question-answering task. The results of these experiments look promising.

...

(2) The description of the models that are developed is very minimal, making it hard to determine how different they are from, e.g., the models presented in [15]. It would be very helpful if the authors included some graphical representations and/or more mathematical details of their models. Given that the authors still almost have one page left, and that they use a lot of space for the (frankly, somewhat superfluous) equations for the number of parameters of each model, this should not be a problem."

These reviews in turn led to significant (though apparently not significant enough) modifications to the paper (https://openreview.net/forum?id=idpCdOWtqXd60&noteId=C8Vn84f...). These were some quality reviews and the paper benefited from going this review process, IMHO.

https://southerncrossreview.org/81/feynman-brazil.html

If you don't have arXiv access, find an endorser <https://info.arxiv.org/help/endorsement.html>, and send them a SHORT polite email (prioritise brevity over politeness) with your paper and the details. Something like:

> Hello,

> I wrote a paper for college in yyyy (attached) on automatic grammar correction, which got rejected from Venue for grammatical errors in the figures. I still want to publish it. Could you endorse my arXiv account, please?

> Also, could you suggest an appropriate venue to submit this work to?

> Yours sincerely,

> your name, etc

Follow the guidance on the arXiv website when asking for endorsement.

Papers are an exercise in communicating information to the paper’s readers. If the writing makes it very difficult for the audience to understand that information, the paper is of little use and not suitable for publication regardless of the quality of the ideas within.

It is not the reviewer’s job to rewrite the paper to make it comprehensible. Not only do reviewers not have time, it is not their job.

Writing is not easy, and writing technical papers is a genuinely difficult skill to learn. But it is necessary for the work to be useful.

To be honest, it sounds like the teacher who suggested you write the paper let you down and wasted your time. Either the work was worth their time to help you revise it in to publishable form, or they shouldn’t have suggested it in the first place.