It's an intriguing idea, but prompt injection and misleading the AI seem like fairly trivial things to do right?

I dont think so. The LLMs are trained to play it safe and avoid responsibility for hard decisions. No commercial model will say "No, I wont connect your call, I dont think you are bleeding to death at this moment, I believe you are a scammer". Nope.