Capabilities vs. "is this a good idea?" — seems like you're saying two things, one of which I strongly agree with and the other I strongly disagree with.
On "should we":
I agree completely.
I've previously made a similar example with sociopaths, when people are incredulous as to the risks of an un-aligned AI.
I've also previously averred that the first time we attempt a brain upload, we're likely to know so little about how our minds function that the upload will have every possible mental health disorder simultaneously, including many that we've got no words for because they're incompatible with life.
On capabilities:
> But I want human teachers behaving truly human to students, not something that can act "disturbingly capable of emotions" or fake it.
If you can't tell, does it matter? Students have been learning from increasingly capable simulacra of human intelligence and emotion since no later than the invention of writing, possibly even earlier with art.
LLMs are good enough to be interesting, and good enough to fool a lot of people a lot of the time, even if last year's models weren't passing Turing test scenarios where people were actively looking for them. (But also, in those tests, humans rated other humans as human only 2/3rds of the time: https://arxiv.org/html/2405.08007v1). Although the newer models are generally better, I don't want to claim if they're specifically, at this task, better than the old models — still, that's how close they were on the only formal test I know of, rather than the mere vibes of listening to them synthesising a voice and me being surprised they'll um and err now.
> AGI is nowhere near this (and there's no good definition of what AGI is, anyway).
I made the parenthetical point myself. LLMs, however, are absolutely there.
Think of it this way: neither one of us can tell by the other's words that the other is not an LLM, all we can do is trust because we're on a forum where it would be rude to use one.
> whether the sociopath can pass knowledge well and can fake as if they had normal human emotions won't sway my opinion.
Sure, but you also wouldn't be able to determine if you're looking at a sociopath or someone with normal human emotions.
> Real human connection is what makes us human.
Para-social relationships say otherwise. Fictional characters, let alone the actors portraying them. The electoral popularity of sociopaths in democracies. Shallow demagogues. Art from generations long-dead. These things give the illusion of a real connection — to the extent there's any connection, it's all one-directional.
> AIs are not human by definition. Therefore, AIs cannot replace humans in those situations where human connection matters.
(Parenthetically: would an uploaded human mind, count as human? I don't know).
But it doesn't matter if they're human or not. Dogs and cats often provide a similar connection (both growing up as a companion, and a being to look after as an adult, though the dog owners I know really don't like anyone treating dogs as "fur babies"), as do TV shows: to the extent that the "connection" part matters, it doesn't need to be human; to the extent that the "human" part matters, it doesn't have to be real.
> And we shouldn't want them to either, because otherwise what's the point of existence? Asimov's fictional planet of Solaria was a dystopia, not something to aspire to.
The trouble with fiction is that it's as much of a shallow echo as current LLMs themselves: yes, it can produce an emotional affect, but for every Asimovian Solaria there's a Banks' Culture where the AI have us thrive.
Both are unsound examples upon which to base actual expectations: even if we happen to develop down one or other of those paths, they are written to be interesting not plausible, and the similarity is mere coincidence.
> [1] or the pop culture understanding of them, anyway. Let's not get caught up in definitions.
I tend to agree, though I feel there's an irony here somewhere — shared human connection on the basis of common understanding of a shared meme about the mental processes of a sub-group of humans that's being Othered in the same way as LLMs… but I can't put it into words. And it would be super ironic to ask an LLM to try to do so for me, so I won't.
--
Don't mistake the map for the territory; but also don't forget you can learn the territory from the map.
Any given AI may be a mere simulacra, a P-zombie mimicking our inner world without having one of its own, but that doesn't mean the model isn't good enough to learn from.