As for your analogy, I'm not sure we know enough about human intelligence core mechanisms to be able to dismiss NN as being fundamentally incapable of it.
Of course if you statistically sample from that in convincing ways it will convince you it has the properties of the systems (ie., people) which created its training data.
But on careful inspection, it seems obvious it doesnt.
Bugs bunny is funny because the writing staff were funny; bugs himself doesnt exist.
Magic “reasoning fairy dust” missing from the formula? I get the argument and I think I agree. See Dreyfus and things like “the world is the model”.
Thing is, the world could contain all intelligent patterns and we are just picking up on them. Composing them instead of creating them. This makes us automatons like AI, but who cares if the end result is the same?
Data is produced by intelligent agents, it isn't just "out there to be sampled from". That would mean all future questions already have their answers in some training data: they do not.
See for example this exact tweet: pre-2021 coding challenges are excellent, post-2021 are poor. Why? Because post-2021 didnt exist to sample from when the system was built.
this to me is the proof it is able to correctly infer meaning, and is clearly a sign of intelligence. (something a drunk human has trouble doing, for example).
Well yeah, it's been trained to produce output that would look like conversation.
This requires at least "some" conceptualisation of the things you're talking about. It's not just statistics.
Excellent analogy, and I appreciate analogies (perhaps even a bit too much). Will be using this one. Thank you!
If there's one field of expertise I trust programmers to not have a clue about it's how human intelligence works.