Comment by etaioinshrdlu | Hacker News Reader

Comment by etaioinshrdlu | Hacker News Reader

I think the total economic impact of AI will be greatest for tasks that output high-dimensional data, such as GANs. For the simple reason that it can replace a lot more human labor. A great many jobs could be augmented with such tech.

Furthermore, I think the results from GPT-2 and similar language models show that researchers have found a scalable technique for sequence understanding. They are likely to just work better and better as you throw more data and training time at them. Imagine what GPT-2 could do if trained on 1000x more data and had 1000x more parameters. It would probably show deep understanding in a great variety of ideas and if prompted properly would probably pass a lot of Turing tests. There is evidence that this type of model learns somewhat generally, that is, structures it learns in one domain do help it learn faster in other domains. I am not sure exactly what would be possible with such a model, but I suspect it would be extremely impressive and meaningful economically.

I think we are likely to see that type of progress in the next year or two, and for there to be no AI winter.

While I don't think there's going to be an AI winter either, I don't think GPT-2 will achieve sentience or anything close to it.

And that's for the same reason that no matter how much data they feed Tesla's self-driving AI, it will still try to kill you now and then. The problem space is just too big. All the people I know in this space don't think it will be solved for at least a decade and maybe not even then.

But I do suspect the 2020s will see the creation of agents combining classical algorithms with deep neural networks to do amazing things in domains that are closed and constant. But they're all going to be glorified (yet wonderful) unitaskers.

The only thing that worries me is that I don't trust FAANG to do the right thing ever anymore, and it's amazing to me that so many have opted into the panopticon of things in exchange for the ability to order stuff and turn their gadgets on and off.

Does GPT-2 really "understand" anything? I feel like this is pretty quickly going to devolve into a semantic argument, but having interacted with some trained GPT-2 models, it seems to produce only what Orwell would have called duckspeak[0]. There's very clearly no mind behind the words, so it's hard for me to credit it with understanding.

[0] http://www.orwelltoday.com/duckspeak.shtml

I think the only time a system can be truly be said to understand something is when its answers are derived from logic (such as old school symbolic AI). No matter how good current statistical approaches get, they won't meet that bar.

However, I do believe we see evidence of approximate logical reasoning in these models, as well as the concept of abstraction.

Furthermore we can take statements generated with statistical techniques and validate them mechanically with older techniques. This is basically what recent work in automated theorem proving using deep learning is about.

Generating logical statements using heuristics and then validating them mechanically also sounds like a reasonable approximation of what a human often does, speaking as a human.

> Generating logical statements using heuristics and then validating them mechanically also sounds like a reasonable approximation of what a human often does, speaking as a human.

I think I agree with that, but I might add that humans who understand a topic well can also make novel connections and uncover further implications that might seem illogical at first glance. This process of "insight" seems poorly understood by everyone, but I think it goes beyond validating heuristic intuition.

If something acts as though it understands a thing then it does. What else could understand mean? Behold the damage that Searl hath wrought, lol.

Maybe it makes more sense to talk about predicting the behavior of a system independent of the composition of the system.

> If something acts as though it understands a thing then it does.

I've had the misfortune of working on teams where the hiring manager subscribed to this philosophy.

Reply on news.ycombinator.com