Hacker News Reader

Comment by gen220 | Hacker News Reader

Yea to be clear I think >70% of the information is not profession specific.

I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.

The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.

My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.

View on HN

The human genome is 800MB, (and 500MB with compression according to GPT) so that part is easy.

I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".

Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.

Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.

> A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for

What makes it hard to search for? If you tell me you want to open a lemonade stand in East Bumbling Falls, I won't know that city’s municipal ordinances. But I will know that cities tend to have ordinances. So I’d search broadly for what codes that city has. Then I’d go through each one looking for anything that might affect a lemonade stand.

Usually at least here it's tens of thousands of laws like "replace article 3 paragraph 2 with:" and it's not like you can ctrl f "lemonade stand" and get something useful.

A strong llm without that knowledge can probably spend millions of tokens parsing every single ordinance and give you an answer if you directly ask about that.

But both a local human or an "all knowing" llm when you just tell it about the stand will be like "hmm wasn't there a law about low calorie beverages being tax exempt? Let me check" and after a very precise search come back with a tip that if you sell sugar free lemonade that's tax exempt.

I mean would you get an accountant from another country just because he can use search?

Reply on news.ycombinator.com