Turing awardees republished key methods and ideas without credit

> Whenever you cite papers just because you're "politically supposed to", even though you don't think it's really intellectually useful for anyone to be aware of the cited paper, you're making the citations less useful for everybody.

The audacity of deciding on behalf of readers whether information is valuable to them when publishing an academic paper is quite an interesting concept to me.

I'm not published in journals, but I do have a "professional" writing background. There's only a few ways I can imagine to read into your statement. Either you think you're above citation, or so much smarter than the reader you're qualified to decide on their behalf whether it's relevant, or simply hiding it for nefarious reasons.

Any of these options, or even the perception of them, is precisely the type of behavior fostering anti intellectualism and prejudice for academia.

It's also quite ironic you repeatedly mention "politics" while your entire argument seems to hinge on seeking to deny empowering further those already in power. Isn't that exercising political capital?

My godfather is an academic that does some arcane nonsense (I mean that in the nicest way possible) involving topology and number theory. He told me a story about how he and his coauthor had a paper in review for 3 years because the reviewer insisted on adding citations to what they assumed were his own work because they were not particularly relevant. My godfather after a round or two didn’t have the patience for it and said whatever let’s just put them in by his coauthor refused hence the three years of back and forth.

>> Schmidhuber is only the worst of many abusers of the citation system. Everyone knows that anonymous reviewers are more likely to approve of people who cite their work. Someone is politically powerful? Better cite their paper, even though you don't really think it's a good paper and you didn't use it.

The reason that this is not the norm in academia is because there is a substantial number of researchers who care very, very much about academic integrity and even more so about getting their work properly recognised, just like good old You_again.

Btw, "everyone knows" is a running joke in Game of Thrones, it's what the Dothraki always say to show that they're pre-scientific barbarians who believe whatever they like, or at least that's how I get the joke.

Whether Schmidhuber is a bad actor taking advantage of the "system" in general is separate from the specific case I was responding to (not sure if that was your comment or somebody else's.)

But as I said before, we should take the long view. Globalized scientific research ought to have a more neutral way of assessing citations. The 20th century way of doing it may not be the right way any more. I don't see anyone else suggesting a problem of this scope.

> Schmidhuber is cited far more often than he should be

You haven't even read the paper, have you? Otherwise you'd see that it's Hinton and Bengio who are cited far more often than they should be. Just look at disputes B1, B2, B5, H2, H4, and H5 to see how they republished parts of his work again and again without citing it. No honest scientist can approve of something like that.

Sometimes this just happens by mistake. The literature is vast and people rediscover things fairly frequently that they honestly didn’t know existed. The related work section is a good faith effort to contextualize your work in the space of extant works. You will not know every work even seminal ones (it happens, especially if it came out a few decades beforehand), and you don’t have space to include everyone. Sometimes you’re coming from an adjacent field that doesn’t literature from other sources. I study a lot of control theory, and optimal control theory is something that shows up in engineering and economics. An economist writing on the topic may not be familiar with all the engineering literature. Recently, with the machine learning explosion, I’ve come across reinforcement learning papers from time to time where they rediscovered something from optimal control and are proposing it as a whole new thing because, well, they don’t know a lot about the control theory literature. Personally, I don’t think it’s a large problem because it brings the idea to a new audience and it’s impossible to know everything.

Another aspect that makes it difficult is that different people even in the same field use different terms for the same thing. As an example, engineering optimal control folks like to optimize cost functions, economists might refer to that as the utility instead, and RL folks like to use rewards (which is just a negative of a cost and functionally equivalent). That makes it difficult to run a search. I’ve come across a whole slew of new papers to read on topics just by changing out key terms for synonyms.

A reviewer may suggest other works to include in your discussion. Sometimes the recommendations are appropriate, other times it’s a backhanded way to fish for citations and the works suggested aren’t actually relevant. I would say harping over not including one is poor form unless it is so similar that is important for you to explain the novel aspect in your paper.

So which papers are you talking about?

I agree a significant amount of work (and often insight too) is needed to translate an architecture idea into something that works in practice, and there are certainly plenty of ideas that are obvious in the abstract. But I also think it's important to avoid dismissing work only on the basis that it doesn't involve "real life datasets".

Deep learning is a relatively unexplored field and there are many open mathematical and scientific questions to ask that involve only model equations or contrived datasets. Novel theoretical results are not just about some architecture idea but about proving facts that can be useful for understanding how the model class would perform in different scenarios. Which in turn can help shape the search space for applied work.

Additionally, I don't think credit assignment should be so discrete. 100% agree that vomiting out vague ideas shouldn't grant claims to credit, but academic science much too often gives only a single author the "real" credit.

Incidentally, in other fields the person who actually makes it work very well may not be the person that receives this credit. Like biology can involve a lot of hard manual work (that isn't really intellectual) in order to realize a project plan. It varies how much of the credit those people receive, and I'm not even sure how much they should receive. This topic is extremely nuanced.

In many fields, this is how citations would work

"Introductory theoretical work in GAN was done by Schmidhuber [1], but it was not until large experimental efforts [2,3,4] on image generations that the power of GANs was revealed."

I don't buy that this is standard practice in the ML community, and even if it is it's BS. If the basic idea/principle has been published previously but in a different context you should cite it and say why the solution is not directly applicable or has not been evaluated in the current context. Anything else is unprofessional.

> Basically ideas are a dime a dozen.

There's a spectrum of ideas, from groundbreaking to "dime a dozen". In tech startups, and in almost all of computer science, most ideas are a dime a dozen, and the value is in the execution.

But clearly, some ideas are groundbreaking. Einstein rightfully gets the credit for an on-paper hypothesis that wasn't proved until decades later via a chain of critical discoveries and experimental innovations by other people. It's legit to call it Einstin's relativity, and not Mossbauer/Hay's relativity.

This is nothing to do with ideas. Schmidhuber is complaining that his published work was plagiarised. In machine learning research, in order to publish your work you have to show that your proposed approach works and to do that you have to beat some benchmarks and establish a new state of the art, otherwise there's no publication. That takes work and that's the work that Schmidhuber claims was inappropriately left uncited. In fact that's exactly the kind of work that Hinton, LeCun and Bengio have always done. That's what machine learning researchers do.

This ... idea that Schmidhuber is an ideas man who's never done any real work is Hinton's allegation, and it's clearly designed to misrepresent both Schmidhuber and his work in order to discredit his complaints. And I'm sorry to say that people on HN have fallen for it hook, line and sinker, I guess because that's what social media says.

Btw, the point I make, that you don't get published in machine learning without beating some benchmarks and establishing a new state of the art, I can attribute that to none other than Hinton himself, in an interview with Wired, whence I quote, by the by:

>> What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement.

https://www.wired.com/story/googles-ai-guru-computers-think-...

So that's the guy accusing the other guy of being nothing but an ideas man and that you don't need to cite someone who first came up with an idea, saying that "new ideas" are important.

But that's just Hinton presenting things just the way he likes. Now ideas are important, now they're not, as he pleases.

I don't know enough about this area of research to have an opinion on this particular topic, but I've noticed a trend with this sort of thing with my colleagues. They both claim at different times, depending on whether it benefits them, that

(1) Ideas are a dime a dozen, and making it work or bringing it to fruition, is the important thing

and

(2) The idea is the important thing; the specific implementation by someone doesn't matter, as they're just doing what the idea creator or discoverer laid out for others to follow.

Sometimes I feel like there's a fundamental paradox there that arises a lot in numerous areas of work, business, and economics.

This sounds a bit like a justification of plagiarism. In science, you must cite the original work.

But if other people identify and further develop one or two out of his models, he should still be cited. If they did not use his work at all, coming ith one or two models independently, then that's a different situation. It's a bit of an honor code thing as well, it's hard to prove if somebody has a read a paper or not. But then there's a more stringent standard where one cites as part of surveying preexisting work, which can result in a vast list.

>atrophying his skills too much to do anything now that the industry is moving so quikcly.

His recent papers are still cutting edge. He's already solved self-improving AI: https://arxiv.org/abs/2202.05780 , it just needs to be scaled up.

There's nothing wrong with being a communicator to the public.

It's not his field, what do you expect?

It's hard not to think of the beef when you think of him now though, even if you already knew some of his work. To be clear I don't think that should take away from what he did accomplish, but it's like a Pavlovian association at this point to also have his behavior come to mind.

I should not have said "becomes famous for...". I meant that today, his name is mostly associated with that, when people talk about him.

That particular claim seems to be true, yeah.