Must we always see this restated every time? It's getting a bit stale always seeing these kinds of comments on articles about LLM.
It's a relevant comment in this instance because we're discussing concepts you need to be both trained and practiced in to reason about, and that our discipline has traditionally been blind to. Plenty of people working with LLM context issues who've never been exposed to the idea of 'subtext' or could tell you why it would matter to their direction of effort.
how much of the training data had thinking traces that dont make sense to people as being actually a description of why the output should be that way?
A naked "natural intelligences do this too" might be a bit too short to be useful. But if we can add when/where, cite papers, or show ways in which the parallel operates, then it might be useful.
Compare, eg, talking about a robot arm, and someone goes "a natural arm does this too". You can tell about the fact that it has the same degrees of freedom in the same places, or how this pertains to inverse kinematics, or etc...
Same way here, "this happens to be how natural intelligences seem to solve this too! According to Foo, Bar, Baz et al (2026) the gadget is always twiddled beforehand in macaque apes. " or "Same for natural intelligence: I've noticed I use the same general algorithm myself. I've always considered this the correct way to do translation between languages". --
A more concrete example of a useful answer here.
Natural intelligence does this too! When given the question "explain your reasoning" humans are indeed quite prone to post-hoc confabulation. [1]
[1] https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20s... "Telling More Than We Can Know: Verbal Reports on Mental Processes" (this citation is quite old and may have been superseded, mostly just to illustrate how the rule might work)