How do you define reasoning in a measurable way?
One of the biggest things I've learned after the event of LLMs is that humans definitions of intelligence/thinking/reasoning/consciousness/etc are very poorly defined. Not just across society at large, but the sciences themselves.
Something like this is actually a stance in the tradition of inferentialism (see the term sapience). Though "reasoning" isn't like, turing machine computability in this space; from what I understand, it's some abstract notion of the "space of reasons". I don't really understand it, honestly.
There's some merit to this, IMO. When an LLM goes wrong, do you blame the person or the LLM? As in, would you throw said LLM in jail, and hold the LLM accountable? Not right now, at least. I'm not sure if that's what is meant by the "space of reasons", but the intuition is that 'reason' can mean a lot of different things, pragmatically speaking. Reason as a legible audit trail is one of those ways.
But that's arguably getting into the social aspect of 'reason' (important!) and not like, what STEM people traditionally think of as 'reason'.
what would be the difference between not reasoning and reasoning not like a human?
I ask it to highlight that most people have no clear answer to what we mean by reasoning. It's a notoriously hard term to define with an precision, and even harder to define by precision in a way that would exclude LLM's, which is usually what the people I ask it of want to do.
I don't think your steelman version really "works" for that purpose because the immediate response would be that it makes "reasoning" largely irrelevant if you define the term that restrictively.