Hacker News Reader

Comment by jcattle | Hacker News Reader

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.

And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.

When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.

But still, 30 years later here we are, standing on top of the shoulders of this useless research.

View on HN

Like half of what Schmidhuber is always complaining about is that (except for LSTMs) people aren't standing on the shoulders of his research very much. They try to solve some of the same problems people have always wanted to solve, try some of the same approaches people always tend to try, and then tinker until it works. At no point do they consult Schmidhuber's decade-old papers where he tried something kind of similar but didn't get very impressive results, and hence they also do not think to cite him. Then he comes out of the woodwork to assert priority.

What you're describing is people who fail to follow the most basic principles of academic research. (Check existing academic literature, mention and give credit to prior work.) This would be fine if these people didn't claim to be doing scientific research, didn't boast their academic credentials, didn't publish their finding as original work and didn't demand credit for their work in academia. Of course, they do all of these things. They benefit from a system they're actively denigrating (and in some ways degrading).

To put it more simply, people with academic credentials should not demand acknowledgement of their current intellectual work while denigrating and ridiculing the importance of very similar work done in the past.

You can be influenced downstream by papers you haven't personally read.

Shane Legg was in Schmidhuber's lab at IDSIA before being one of the founders of DeepMind, so he probably read the papers personally and knows what influenced him or not...

Of course, but if you haven't read them you also shouldn't cite them.

And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.

"if you haven't read them you also shouldn't cite them" -- this is wildly incorrect in an academic context. If I'm using ResNets, I should cite the original ResNet paper, even if I haven't read it. If I'm using Transformers, I should cite the original Transformer paper, even if I haven't read it. If my work is a direct extension of method B, and method B is a direct extension of method A, I should cite the source of A, even if I haven't read it.

You can't claim independence from past work simply because you didn't look directly at it. The job of an academic researcher is to know the landscape of relevant ideas, where they come from, where they're going, and to hopefully contribute a few new good ones.

Citation chains should extend back from your work, along a reasonable line conceptual inheritance, back to a reasonable point of origin. Schmidhuber has different definitions for both of these reasonables than the bulk of the ML research community, to a point that makes him difficult to satisfy.

It's worth pointing out that sometimes, some papers just become part of the general context of things and are no longer explicitly cited. Or people cite textbooks or general survey papers instead.

For example, take a look at Albert Einstein's Google Scholar profile. He's not the top cited physicist. Not even close. It's because other researchers don't explicitly cite his papers. https://scholar.google.com/citations?user=qc6CJjYAAAAJ&hl=en...

Same with Tim Berners-Lee and the World Wide Web. Imagine if his original paper were cited every time someone deployed a web site.

You make it sound like all original ideas from academia must be cited all the time, even if that was not the source of someone’s inspiration.

If I’m in the private sector, and I rediscover something from first principles, it is not my responsibility to go search all academia to see if someone’s done it before so I can cite their work.

If I rely on a code library that doesn’t explicitly cite papers it was built on, it is also not my responsibility to go find all the papers that it might’ve been built from and cite those papers.

Your Paper C does not need to cite Paper A unless you are discussing some aspect of it that Paper B is not. Otherwise you inherit the A citation via B.

Spamming citations is unnecessary.

You should read those papers then

> this is wildly incorrect in an academic context. If I'm using ResNets, I should cite the original ResNet paper, even if I haven't read it.

Eh, I think the correct answer is: read it, then cite it.

You're not really supposed to cite something without reading it, as it might say something different than you think. But sure, citing it w/o reading it is better than not citing it at all.

> this is wildly incorrect in an academic context.

Worked for me in my academic career ¯\_(ツ)_/¯ Of course if you suspect a work is poignant, you should read it, then cite it.

> Of course, but if you haven't read them you also shouldn't cite them.

But if you build on them you should have read them. I don't know about the specifics and I don't know if Schmidhuber is out of line or not, and citations and impact factors are a terrible mess, but generally speaking, you are responsible for finding and reading and citing any related work that needs to be cited, and if you work on neural networks in an academic context you probably have been forced to read that particular one at some point. Citation obligations don't just disappear because you don't want to do the research.

It's not just the papers. It's also the students and their students, many of whom ended up working in the top labs today.

Every western academic nearly systematically ignores eastern science and philosophy: classicism means "western European". Never mind Europe only flourished intellectually post Islam, which imported the science and engineering of China and India, critically including printing and zero[0]. IMHO this is why distaste for academia grows: it's based on appeals to authority which are demonstrably farcically misplaced. Alternatively stated: the emperor has no clothes, much less silk or paper!

Just as the Dewey Decimal System really only served the purpose of providing the facetious nominal linearization of an arbitrary depth ontological oversimplification, so too humans are much more like random pattern matching machines than festidious sense-makers glued to absolutes derived from false appeals to static mono-perspective ontological hierarchies. The same is becoming lived experience in the LLM age, although the tiktokked youth apparently cannot string ten words together or focus longer than three seconds to attest, I'd wager they can feel it. Are we losing something by rejecting the habit of rigorously manually tending to spurious and temporary ontologies? Yes. Is it necessarily a loss in the long term? Probably not, in the same way we no longer write long-form letters or leave calling cards. Are we gaining something in response? Yes, at a minimum much stronger cross-pollination between ivory towers by fearless exploratory pragmatists who disrespect the would-be scope of nominal professions in favor of holistic thinking... both AI and human.

[0] https://en.wikipedia.org/wiki/Science_and_Civilisation_in_Ch...

I think most of criticism of academia is about the rampant fraud and unreproducible results, due to the way the incentives are structured.

It's like the old saying "only 10% of my marketing budget is making a difference, I just don't know which 10%"

You don't know ahead of time, where the breakthrough will come from.

There is ton of research that sits on the shelf, and then years later, it gets re-combined with some other useless research, and boom, some big breakthrough.

This current attitude of all research is worthless, so it should be cancelled, is shooting our future selves in the face.

I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life easier. So while the bedrock is based on academic research and we thank them for that, scaling anything requires a lot of work in uncharted territories.

But that isn't the purpose of academia -- the purpose of it is to discover new phenomena not to make products. It is true that there is a lot of work to turn a new advance into a product whether it is software or turning biological knowledge into a drug, but without discovery of new phenomena new products will come to a halt. While it is true that some corporate labs, most famously Bell Labs in its heyday, but also for example IBM's T.J. Watson and Xerox's PARC did do basic research besides product-focused work, this is pretty rare because it is hard to justify the cost of something that may only be practical in decades and often help your competitors as much as yourself.

Well, yeah. That's why we have "research & development" as a term.

What you're referring to is the "development" part of that. In some sense: the job you have _exists precisely because it's not part of the research phase_, and it's equally as valuable as the research part. Research is the proof of concept; development is scaling up and making production-ready and finding small efficiencies and so on.

From an industry perspective, it's tempting to conflate these, because that's what industry research labs are designed to do: integrated R&D. But that is not at all how academic research labs work.

> My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers?

Soon we will also blame academia for not providing iOS and android apps

I jest but database design is its own sub field of computer science, maybe look into their papers?

I did that too. Ending up building my own reverse index with a fixed-size vocabulary. But that's my issue, you start building one product and you end-up building ten in the process to solve all edge cases because no one bothered to research how things scale.

Well it sounds like you did?

The practical issue of academia is epistemological. It's about learning how a phenomenon came to exists. If you are looking for efficiency the field of academia related to learning how to do so is computational complexity and it works quite well.

The goal of academia isn't to be practical, "only" learning.

The science is on them, the engineering is on you.

Yeah, that's your job.

Where is "this crowd" that you are talking about?

The closest to that that I've seen is that traditional academia approaches are too far removed from practical applications for highly applied fields like software engineering, or too slow for fast-moving fields like modern day ML (thus, all the preprints).

> Where is "this crowd" that you are talking about?

In the same room with Illuminati, Yeti, and Chupacabra.

I think most people forget the graph-like nature of scientific research. You don't have n useful papers and m useless ones by themselves, you have an interconnection of those. There may be isolated cliques of uselessness, but there isn't a clear correlation between academia and private research.

Many ideas come from philosophy, which many find useless.

Heraclitus discovered change back in ancient Greek, I don't know where we would be in scientific research without that (deliberately ignoring the debate about the originality of what we know about Heraclitus work). I bet his contemporaries found his "research" useless.

> From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.

Well... that's "starve the beast" in action. A lot of things we take for granted, that underpin our modern ways of life, came to be due to government investing. Laser, radar, microwaves, the early Internet, that all was military R&D.

"Unfortunately" (well, for the rich and the MIC, at least) there is no way for people to siphon off money in government-funded research, so once the libertarian/small-state BS completely took over following the collapse of the USSR, a lot of that got torn down or supplemented with enough bureaucracy to make Germans cry... and that's why reusable rockets were not invented at NASA but at SpaceX instead.

Reusable rockets were not invented by nasa because their mission was exploration not commercialization.

Cheap reusable rockets allow for a lot more research for a lot less money.

Unfortunately, as the early history of SpaceX shows, it required a lot of failures to learn from to design the current crop of rockets. And that's the advantage that private R&D has... as long as the person in charge has money, failure is an option, because in anything publicly funded, any failure will relentlessly be blamed on the currently governing party by the opposition.

Reusable rockets were "invented" by Lars Blackmore when he was working at JPL (Jet Propulsion Lab). I say invented because like anything in the evolution of engineering, credit is messy.

https://en.wikipedia.org/wiki/Jet_Propulsion_Laboratory

> Founded in 1936 by California Institute of Technology (Caltech) researchers, the laboratory is now owned and sponsored by NASA and administered and managed by Caltech.

Minimum-Landing-Error Powered-Descent Guidance for Mars Landing Using Convex Optimization http://larsjamesblackmore.com/BlackmoreEtAlJGCD10.pdf

Elon originally wanted parachutes and was convinced by Lars to go with self landing rockets.

Private labs feed off academia. Without academics to staff them, they'd get a lot less far.

I used to work at Nokia Research when they still made phones. Probably the closest thing Europe had to Silicon Valley twenty years ago. Except it was in Helsinki. Lots of stuff got invented there. Nokia didn't really manage to capitalize on its own inventions of course. Or rather it got caught up in its own clumsy attempts throwing babies out of the window by the bucket load. But others sure did. A lot of modern smart phones still have tech in them that Nokia pioneered before either Google or Apple shipped a smart phone.

At the time there was a lot of talk about the demise of industrial research labs. Bell labs (now actually owned by Nokia!), Xerox PARC, IBM, and all the other big US labs that produced amazing stuff are former shadows of themselves. There is some truth in that

But you could argue that Google and Apple picked up some of the slack. And the current AI boom came out of Google cherry picking all the best universities for their AI talent and putting them all together in a research group that then got free reign. Like Nokia, that involved a lot of ejecting of babies with the bath water. But it seems to have spawned lots of new startups that can trace their roots back to that research group in Google.

This is a straw-man if I ever saw one.

Practically no one is against hard science research, properly conducted. The issues are rampant fraud / p-hacking / unreproducible garbage mixed with an unhealthy dose of ideological monoculture and indoctrination, garnished with rising tuition prices while sitting on huge endowments in case of the Ivy Leagues.

> Practically no one is against hard science research, properly conducted.

As long as you do that with your own money (or money got freely given from other people), sure.

If you use taxpayer money, that's a different game.

There is a long list of grievances I have regarding the (mis-) use of taxpayer money, and funding the hard sciences is way, way down. I can’t even see it from where I stand.

Sure, there's always a worse programme you can point at.

Yes all good points showing issues that academia has at the moment.

However I often see this going from "there's issues" to discounting academia altogether and positioning private labs as a good or only alternative.

After all, most people in the open science collaboration which published the seminal paper kicking off the replication crisis were from academia.

Yes there is no substitute for academia. Monopolist's research labs get close (Bell Labs etc), but they tend to be more "applied".

> Practically no one is against hard science research, properly conducted

Eh, I grew up conservative evangelical, and they were pretty much always going to have a problem with research in evolution and astronomy. Same goes for the fossil fuel industry w.r.t. climate science.

When the scientific evidence interferes with religious doctrine or industrial paycheck, then yeah, folks are still going to have a problem with hard science research.

and you still need tons of money

I feel like you're constructing a strawman to argue against. I visit this site almost daily and the prevailing sentiment is usually the polar opposite of what you're suggesting.

If sentiment on HN were as you say, how could your pro-academia and anti-big tech comment be sitting at the top as the most upvoted comment?

Reply on news.ycombinator.com