>Has anyone set something like that up?
Yes! I did "Captcha for phone calls" for a school project earlier this year. It would play a quick message along the lines of "Hey, to ring my phone just enter these numbers: XYZ", and redirect the call to my phone if the captcha was correct.
The challenge was redirecting calls to the challenge even if they were originally directed to my phone number - the solution was to write a custom call handler (Android) that rejected calls if they weren't in my phone book, and then set up call-forwarding on rejected calls to go to a Twilio number with the challenge. There was a nice edge case where if I actually missed the call, it would just redirect you to the challenge again... but it was good enough for the class.
While testing, I accidentally actually got someone to use it! A guy repairing my watch called me, got directed to the challenge, and actually followed directions! Which was surprising to me. I also caught some obvious spam callers (same first 6 digits), and unsure if there were any other real people who did call and got confused or just hung up.
I've been meaning to clean up the code and publish it, but haven't gotten around to it - there's lots of hardcoded numbers in there.
(Disclosure: currently employed by Twilio)