It's hard to observe your own mind, or indeed others, it's intangible and generally difficult to observe thinking.

Because of this drawback, LLMs are actually a decent model for this sort of process since we can observe how they operate. I'm not claiming they're actually intelligent like we are, but rather that they model the process of drawing connections and making associations close enough to how we think to the point where it's an useful analogy.

I think, if you adopt this mindset, acutely any process that involves precise instruction could be dismissed. Or more generally, your standards for analogy may be so stringent as to render that concept invalid.

There are parallels. You feed an LLM context data and then tell it what to focus on so it can pull relevant data.

Maybe the entire process isn't like feeding an LLM, but that step is. Relevance identification is an interesting part of the process. The LLM can do a decent job of making connections, but it doesn't know what is relevant. In the longer time frame of the thinking process, we constantly throw out data as irrelevant or identify previously unknown relevant data that needs to be added. It's a part of the process completely outside of the LLM.

I'm as AI-wary as they come, but - it's a completely factual, relevant, and helpful statement. To say that any relation or comparison with AI ruins the comparandum is as absurd as saying that anything coloured red is bad because the Nazis used the colour red in their iconography.