I'm sorry, what is "AI" about it? That's just basic human psychology. How is this different from being manipulated in the same manner by a human?
I'm sorry, what is "AI" about it? That's just basic human psychology. How is this different from being manipulated in the same manner by a human?
I would view this from a practical perspective: Institutions are considering moving their dumb questionnaires behind chat-bots, because they think it'll somehow be more efficient, and here's this research showing that in at least one important case there's a big unexpected danger.
While it is possible that hiring an unbiased human intern to "guide people through the paperwork" would give a similar effect... as a practical matter no institution wants to pay for that, so it's not an option that's on the table.
That said, I wouldn't rule out the idea that an LLM could be worse than an average human helper, since their correlation-following may cause them to introduce stuff into the conversation where a human wouldn't think to or would know better. [0]
Tangentially, this reminds me of the video “Never Talk to the Police”: https://www.youtube.com/watch?v=d-7o9xYp7eE
The most important video any American will ever watch IMO
Now it wasn't an in-person law enforcement encounter, and I wasn't the suspect but the victim. The agent on the phone was only tasked with taking down the report and forwarding it.
She listened to my narrative and took down the facts. Then she began to relate it back to me, and at every turn, she gave me the wrong details and altered the story.
So I found myself correcting her again and again and ironing out the actual facts so she had them right. And I came to realize that her mistakes were probably not accidents, but she was intentionally prompting me to reinforce the same narrative as I'd stated it, because if someone is lying, fabricating, or embellishing the truth, they won't be able to repeatedly insist on the facts as retained in their memory.
Conversely, I've had interactions with authority figures, who seem to intentionally misspeak as a test. They want to see if I will challenge the veracity of what they said, or if I can accept that their knowledge counts for more, and perhaps I shouldn't openly question them for every trivial matter.
So if the police succeed in implanting false memories, then maybe someone just had a shitty memory to begin with. If someone's involved in a crime, even as an eyewitness, it's important to work with their perspective, because testimonies consist of a lot of subjective information, and different people have different capacities for recall, so if you ask 3 eyewitnesses what happened, you may get 3 different but true stories, then you reconcile them. Just ask Matthew, Mark, and Luke.
I think I misunderstood you the first time I read this, so let me verify my revised understanding:
You're saying that she purposely feeding back false information to check to see whether you were a reliable narrator? If you fail to correct misinformation, then you have a loose relationship with the truth (either because you're lying, or confused, or perhaps have a mental illness).
Why else would she tell me a falsified version of my own narrative and bring those errors to my attention?
Legally for them, that would be a terrible idea. If she's tasked with receiving allegations then their legal team will be interested in knowing exactly what is believed and what accusations are on the table. They definitely do not want some clerical worker faking a story and masking issues that have a real chance of being substantiated or argued in a court case someday. If I'm lying or fabricating then they'd also want a reliable record in their favor in court. The reporting office is motivated for accuracy and that's exactly why she challenged my facts, so that I could reinforce them through repetition and clarification.
I think it’s good to have research like this pointing out these flawed uses of AI before they’re inevitably used as a means of laundering accountability. It’ll happen anyway.
Is not the ability to screw people's memories up at hither to undreampt of low cost and scale something a bit different?
It can be done at scale for very little. I'm not touching AI because I can't know its biases, it can be enshittified and advertise without the user noticing.