Hacker News Reader

Comment by visarga | Hacker News Reader

That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection.

I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.

View on HN

Before there were neural language models, there were n-gram models, skip n-gram models, latent dirichlet allocation models, a whole zoo of non-neural machine learning models. Google used some of them to power old Google Translate, which was still a very impressive piece of technology.

> That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection.

... pull the other one.

Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.

The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.

For a time while I was working at Google I nursed the idea of transferring to the natural language processing / language modeling areas, but I only have a bachelors in linguistics whereas it seems like they strongly prefer PhDs. Linguistics PhDs can pretty much teach linguistics or go into another field. It would not be hard for Google to find a couple hundred of them and entice them to join up.

At the time Google Books started, “big data” was in vogue and there were data-driven ML approaches to NLP e.g. word2vec, that would have benefited from a large corpus

Reply on news.ycombinator.com