You could turn the verification around. Instead of texting a value to a phone number and asking you to toe it in, you say “Text this number to 40404: 123456”
Then, wait until someone texts that number in, and salt/hash the caller ID number and compare it to what you’ve got stored. If there’s a match, then you’re authenticated.
Probably lots of issues with this from ux perspective…