A stochastic parrot may be a good model of what a human does when you force them to react and give their first impressions on something. But there is more to cognition than that.

People have some emergent machinery for symbolic reasoning in their brains. When they don't just react, they can continue working on an idea and improve it.

When it comes to humans, there is no fundamental difference between training a model and using it. People continue to learn all the time, which allows them to build models of new things and to update their existing models.

People can also interact with the outside world on their own initiative. When you combine this with the ability to learn and reason, it may transform the internal models from arbitrary mathematical constructs to increasingly accurate models of the outside world.

Nothing in this discussion is particularly new. It's just the Chinese Room Argument from decades ago, except that this time it's not a thought experiment. While a static Chinese Room can fool humans and be useful for various purposes, a general intelligence also needs other mechanisms.

Many people don't use System 2 very often, even if they're capable of such thought. That includes me. Most of what I say is probably regurgitated opinions I've read on Twitter without much first principles thinking, which scares me. I think reading a decent amount of books helps a bit. When you encounter books with deep convincing arguments that nonetheless disagree it forces you to use System 2.

Fundamentally, if humanity is given data set A, it will eventually generate dataset A' of 'higher quality' than A. Colloquially that might be called 'progress' or maybe even 'understanding' or 'learning'. That's different than regurgitation.

I don't have strict definitions for any of the quoted terms because I don't think we really understand them yet.

It seems very possible that human generative content shares much in common with 'stochastic parroting'. But I'd hypothesize there's a filtering process, sometimes at the individual level, sometimes at a group or even civilization level that (so-far-over-long-enough-timescales) tends to cull some information that has low 'value' and propagates other data that has higher 'value'.

When an AI system can do the same, that's a big inflection point as you can give it energy and time and it will self-improve indefinitely.

It's pretty clear humanity has this capability. It's not clear whether AI does (or ever will). Or, perhaps it might already be there, and we just don't understand enough to know for sure.

Yes definitely agreed, but most of us are not math model (I hope).

We are able to understand right from wrong, we are able to answers with what is true or what we believe is true. We are able to lie or deny the truth. We are able to say we don't know because we think we don't know and not because we have been programmed not to answer, etc.

Overall I think people saying that, are just saying the difference is that we have a conscience and AI doesn't. Which in a lot of case does matter.

Humans are just fancy autocomplete

You should be terrified, then, that humans walk among you.

> likelihood

Irrelevant since people do have the module to reflect and vet thoughts.

I think the sorts of people who are prone to philosophizing about AI certainly consider this. I also think that whether or not we are isn't actually all that relevant to criticisms being presented.