In fields I have less experience with it seems feasible. In fields I am an expert in, I know it's dangerous. That makes me worry about the applicability of the former and people's critical evaluation ability of the whole idea.
I err on the side of "run away".
But the Rubicon is still crossed. There is a general purpose computer system that understands human language and can write real sounding human language. That's a sea change.
What you're referring to isn't a bug. It's inherent to the way LLMs work. It can't "go away" in an LLM model because...
> understands human language
...they don't. They are prediction machines. They don't "understand" anything.
The 'bug' presented above is a simple case of not understanding correctly. Larger models, models with MOE, models with truth guages, better selection functions, etc will make this better in the future.
> ...they don't. They are prediction machines. They don't "understand" anything.
Implementation detail.
I've got some oceanfront property in Wyoming to sell you.