Hacker News Reader

Comment by ggambetta | Hacker News Reader

> And the public funded the research that made it possible. The transformer architecture,

Errr pretty sure that was Google?

The engineers at google did not sit in a blank room and come up with these ideas out of nowhere. They read the literature to figure out what the hell to do. If you look at the Attention is All You Need whitepaper, you will find there are 32 references, like most ~10 page manuscripts.

The author never claims otherwise?

Did the public fund googles research?

The foundations of it, yes: "all of these were publicly or quasi-publicly funded"

This is genius. Whenever a company does some fundamental discovery, you can point at some grant they once got for something vaguely adjacent and say "see! quasi-publicly funded!" and your worldview is saved.

The author is correct. It is incredibly simple to trace how public research spending creates scientific advancements and how private companies add on the last 1-3% to commercialize the research.

If you want to learn, go trace how deep learning was funded. It started off with USPS.

My entire argument is that techno-libertarians can enthusiastically say that all great innovations were done inside companies, and progressives/marxists/etc can enthusiastically say that well actually, many of those developments started with publicly funded research projects and public-private partnerships, and both are completely right at the same time because reality is messy. It doesn't prove nor disprove anything about whether governments or companies are better at innovation, or deserve more of the credit, or the upsides.

This is right, and it's more or less what I was arguing in the piece. Google built the transformer. That's not in dispute. The point is that they built it inside a research ecosystem that traces back through decades of publicly funded work--DARPA, NSF, university labs, the whole pipeline from perceptrons through backpropagation through the GPU computing that made large-scale training feasible.

Both things are true at the same time, and acknowledging the public foundation doesn't diminish the private achievement. It does raise a question about who should benefit from what gets built on top of it, which is what the piece is concerned with.

Governments can be really good at setting incentives other than immediate profit. That’s where a lot of basic research and the growth of applied engineering toward future products happens.

It's not vaguely adjacent, the actual foundations of that research were directly publicly funded and wouldn't be possible without it - the author is not talking about how their PageRank algorithm got funded nor money that Google received.

Do you have a reference here, or are you just going to continue to baldly state it as a fact? I’m looking at “Attention is All You Need” and don’t see any grant numbers or anything like that.

When I see that paper, I see 32 references at the end. I'm not opening each one of those but there is probably significant grant funding behind that exploratory research. When you actually look at their results its only about a half page. Most of that document is merely providing background and context out of the academic literature. Conclusion is literally just a 1 sentence paragraph, followed by a 2 sentence paragraph, followed by a 3 sentence paragraph, then a link to the git repo.

The public “funded” these models in the sense of contributing to their training data.

The public of the US, China, Japan, the UK, and Canada at the very least contributed to every precursor to transformers in the history of AI. Public and private universities, government grants to them, government projects, prizes to government contests, direct government investment into and subsidies for companies doing commercial research, tax breaks to operators designed to encourage the growth of the field, and large government contracts to use things the government might not even have good uses for yet all add up to government assisted funding.

https://en.wikipedia.org/wiki/History_of_artificial_intellig...

Reply on news.ycombinator.com