Comment by YeGoblynQueenne | Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

YeGoblynQueenne

That's it? We need better frameworks?

I thought this was going to be about this "stuck in a rut":

GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.

https://www.wired.com/story/googles-ai-guru-computers-think-...

Which is the granddaddy of the other one. You don't really need better frameworks that make it easier to explore innovative ideas if you can't ever hope to publish those innovative ideas, even if you manage to make them work.

NB: "GH" is Geoff Hinton.

Both GH and TFA are raising a pretty classic "structure of scientific progress" trade-off: communities that are well organized around specific {theories, datasets, metrics, hardware, software} greatly accelerate progress within the box, and exclude progress outside it.

The way around has always been the hard work of creating your own box - ideally starting with questions, then theories, then methods, and so on.

What's not available is instant attention/accolades/funding for work that presents beautiful methods without first building a community that cares about the problems those methods solve and agrees on a definition of success. So, folks with radically new ideas should focus on rallying others to agree on that definition. I see a lot of good work these days introducing new problem definitions and datasets for that purpose.

This is applicable to almost every research area, where the publication of peer-reviewed (PR) papers is de-facto a standard. I have a background in physics and a few papers published in Q1 journals, so I heard these complaints about "peer-review pressure" many times. But the truth is somewhere in between: 1. yes, sometimes you get a stubborn reviewer and it is extremely difficult to get through, I've experienced this several times; 2. but PR still deliver a required filter against a flood of low-quality papers.

In essence, I found the note above as too emotional, since currently we have nothing better than PR of papers, review of code, etc., in order to guarantee the minimal level of quality.

You get that "flood of low-quality papers" on Arxiv but most researchers filter through papers daily and are generally capable of isolating the relevant and interesting from the dull and unimportant. Probably a lot of citations to papers are somewhat dishonest with the Arxiv version being the only version that was read. From another researcher's standpoint, the peer review is not very important: you already know if you got scooped, you already know if you find something interesting enough to study as well.

Peer review seems to have a purpose but it does not seem to be a quality gate keeping purpose. Perhaps enforcing orthodoxy is the actual purpose.

Why should the majority have to bother with ‘radically new’ when ‘same old’ is still chugging along with superlinear growth?

The majority doesn't have to but journals should. The majority will pick it up if and when someone has proven its utility above and beyond current techniques.

However, it should be noted that most "radically new" ideas are neither radical nor new. And it would be more or less accurate to label a vast swathe of them as "nonsense" although I'd prefer to use a term with a less pejorative connotation.

The difficulty comes in sorting out the "nonsense" from the genuinely new ideas. This takes expertise and time. It's easier to use a quick heuristic to weed out those most likely to be "nonsense" so you can spend the time on better prospects. Of course that heuristic is unlikely to be perfect.

Admittedly I can't speak to ML specifically but I assume the same applies to it as it does to many other areas.

Ideas are cheap. I have ideas, you have ideas, everyone has ideas. What counts is beating the state of the art, that turns heads and raises eyebrows (such as ResNet, AlphaZero, WaveNet, BigGAN and Transformer). Usually those results are based on lots of compute and training data, though.

you can train some weak small baseline Transformer on your workstation GPU, and then check if your creative idea outperforms it. If it does, then it would be worth trying to pursue larger calculations.

Ideas are cheap, but the skills to engineer ML pipes and the knowledge requirements to grok models are not. And the cost of testing some ideas not is not cheap either.

Lots of people have legit skills, high level math, pro software engineering, actual ML chops (I got mine), and probably significant learning in other sciences. But there is no door to walk through on that alone, only strange VC corridors and FAANGY career mazes.

There should be a wide door to support people with real skills taking chances. If 100K for so many silly startups is worth the gamble, so is the gamble on people who have gone the extra distance with their learning and experience.

To summarize: if you disapprove of my paper you’re either stupid or stressed.

Sounds like: /r/iamverysmart

Don't people just read the arXiv? I think Hinton is being a bit old fashioned here. A paper doesn't need to be published to a journal at all to be influential.

How do you filter arXiv? There's way too many papers on arXiv, it's impossible to keep track of everything, especially since the quality on arXiv is ... heterogenous.

You use Google to find a paper with a title that sounds like what you're looking for and then if it's on arxiv you read it.

That's interesting, this method has Google take the place of peer review. Are there concerns with the Google algorithm being unsuited to this purpose, or SEO?

Personally, I only trust arxiv preprints when I know something about the authors.

Google searches all the journals I need. I try searching journal databases (like web of science) and they always miss papers I know exist. I type the title into Google and I usually find it, and if I don't I ALWAYS find it on Google scholar. They even have nifty little links to download the citations for endnote etc.

I find this doesn't work well.

t's quite easy to produce a catchy looking title and abstract. Unlike mere Googling, top conferences and journals add a layer of peer review that reduces the probability of low-quality content. I've been burned too often by random papers on the internet.

Check out: http://www.arxiv-sanity.com/

Reply on news.ycombinator.com