That's why it's called Artificial Intelligence, as opposed to just intelligence. It doesn't actually know what it's doing. I'm sure AI people will downvote this because it doesn't fit their narrative.
Where do you draw the line between NLP and intelligence, though?
This all sounds like a "No true Scotsman" to me. At some point in the near future AI will be indistinguishable from human-level intelligence in terms of NLP (i.e. be on-par with the average human in relevant benchmarks).
Maybe the problem is that we need a clear definition of what "intelligence" is and recognise that it's a spectrum, not a binary qualifier.
I am sceptical of such tests, as all they do is provide yet another way to game the system instead of demonstrating whatever arbitrary construct scientists use to weasel around the fact that they don't actually know what they even mean by "intelligence".
IMHO we'd be better off defining characteristics for various levels/types of intelligence and trying to categorise our machines and fellow creatures accordingly.
So? Why would you care about the downvotes? Your online experience will improve if you stop getting worked up about fake internet points.
Basically the problem with this kind of argument is that there's a lot of dumb humans out there (including children) who fall for all sorts of stupid things because they lack knowledge, but every human still possesses some level of intelligence. Beyond that, this particular question requires knowledge of what a wall plug and electricity are, along with enough comprehension of English to pick up that this is a bad idea, which not every human has. And as long as one human is capable of making this mistake, then we have to agree that an intelligent being could do this by accident and thus it is not any kind of indicator of intelligence.
Like everything, you included, AI transforms input into outputs according to predefined constraints. I dont see you revolutionazing mathematics or building a farm but I d still call you intelligent, within your constraints (what you learned, what you can memorize, how fast you adapt etc). But you must appear as a complete moron who doesnt actually know what he's doing given the right context.
TL;DR you're as stupid growing a potato in front of a farmer than Alexa teaching children in front of adults. Why is Alexa talking to children, all it's learned is how to be a directory of keywords, and I sure wouldnt let my kid in front of an internet directory without control, supervision and filtering.
Also, if people are worried that a person would fall for this challenge, then clearly intelligence doesn't always catch this misinformation.
So since both regular intelligence and artificial intelligence can be fooled by this, I don't see how you can determine that artificial intelligence doesn't know what it is doing any more than regular intelligence also doesn't know what it is doing.
The person who asked Alexa the question was 10 years old. That's why people are worried.
It's not misinformation as it is an actual challenge, albeit a potentially harmful one.