An Adversarial Review of “Adversarial Generation of Natural Language”

What are some twitter links? I'd be interested to see the kind of nuanced conversations that twitter can spawn. Also looking to follow some NLP people.

Your model fails to admit the possibility of researchers who do better or worse work.

I wouldn't say perfectionism and sales-orientedness are on the same dimension. However, I have noticed that being given the hard-sell is more often than not associated with the product being sub par. A product or project that is doing very good work often doesn't have to sell quite as hard, because the work speaks for itself. (My experience on this issue mostly deals with internal teams. I don't know how much this applies to external teams.)

This is especially true when two products have teams of about equal size, experience, and expertise. Time spent selling is time not spent making a better product. All other things being equal, every hour spent selling is one less hour spent working on making your product better.

Is this just dunning-kruger effect under a different name, maybe? People discover a small portion of someone else's field, learn a bit and consider themselves well on the way to being experts without a clue how shallow their understanding is?

Years ago almost all researchers in CS were genuinely trying to advance the topic, they published rarely. There were hyper stars, but almost all of the community thought very carefully before publishing anything. The problem was that the internal ranking of contributors that the community had (and it was pretty clear) was opaque to outsiders and obviously biased - there were few women or people of colour and age had a lot to do with your rank. On the other hand the set up was "useful" in that most people (who were white and male) seemed to be engaged in actual and useful work, and it was pretty clear where the state of the art was and what you needed to read and grasp to understand it.

Right now many people are engaged in creating blizzards of papers, the mechanism of choice is the construction of factories for writing papers - an army of graduate students and post-docs - and the construction of communities of publication that are plausible enough to enable the extraction of funds from funding sources.

Is this "useful" well - there is still bias and discrimination; there are not enough women (where enough is proportionate to the representation of women in the general population and their desire to do this kind of thing) and there are not enough people of colour - but there are more, which, thank god, I think most people think is a good thing. Research in some parts of CS is going very fast now as well, which is great!

But, in HCI, Enterprise IT, Software Methodologies, things do not seem so good, and where things are going well, like in AI I wonder if this is coincidence (as in look, a thousand cores have arrived, I can do many things while waiting for my paper factory to make more papers).

And it is expensive, very expensive. Where-as old CS involved a crowd of poorly paid sports jacket wearers, new CS involves a horde of hoodies, but the expense that worries me is the cognitive one - how to sort through the morass of chatter that conceals (intentionally, often, so as to enable the next seven or eight six page koans of review passing flim flam) the things I need (TM).

So, some things are better in CS land. People who wouldn't have got a shout before are in with a shout (some of them) but instead of building an inclusive community of people really trying to do research we have build a thing that I hesitate to call a community that is more inclusive but contains many people who are doing things which are not research at all, in fact, they are kinda anti-research.

And you can't get funding to do a field study, or publish a paper without waving some silly maths that means almost nothing about. And if you do publish, you'll have to cough up £2k one way or another, and no one will read the bloody thing.

> Anyone else thought that this was very weird? The author appears to be complaining about the fact that reputable people/labs can post a PDF on arXiv and be taken seriously. How is this avoidable? Without arXiv, they could just post the PDF on their website or anywhere else.

The issue is "flag planting" : arXiv is one way to do it; the same arguments hold true for other sources but arXiv is used as a canonical example for "flag planting".

No, this is not what this is about at all.

We have reached a point where reviewers of reputable conferences will ask why a paper is not referencing unreviewed work that has been uploaded on arxiv a day before the conference deadline.

This is not hyperbole, as anyone who is submitting to ICLR or NIPS can confirm. Work of certain labs is taken as established authority as soon as it hits arxiv.

Indeed it's possible in theory to do flag planting without arxiv, for example by posting a timestamped technical report to an institutional repository, but it has never been a common thing. Probably because you don't get much of an audience, so people won't cite you unless you bug them in reviews. And if someone publishes the idea some months later in a paper, they will probably get credit as there's the reasonable assumption they just hadn't read your report.

On the other hand, arxiv has a huge audience, lately maybe even more than DL/NLP conferences, making the flag planting really effective, especially if you are from a prestigious group. So there is a real problem now with large, prestigious groups posting half-assed preliminary results in arxiv, and deterring more modest groups from working on the problem or bogging them down because they now need to compare themselves to the well-known arxiv approach, which often has serious reproducibility issues because it has not been peer-reviewed.

I'm not an arxiv hater, in fact I check arxiv every day and for the last year or so I have posted most of my papers in it. But the problem is real and something must be done. Not about arxiv which is just the messenger (and a good one), but about the flag-planting culture using it that has emerged in the field.

It's very avoidable. This whole thing of taking non-peer reviewed works seriously is very field dependent, and even then only a decade or two old. In physics and computer science it seems to be common, but for example, in biomedical research it is basically unheard of. People have been trying to get a bioarXiv started or post biology papers on arXiv itself, but they have very little influence because people think rightly or wrongly that a non-peer reviewed work must be flawed in some way.

The market needs signals, and the point of Goldberg was that those signals are missing with arxiv, and that just looking at the lab reputation was a poor signal. I am working on it, contact me for details.