Caller ID is an untrusted advisory best effort service. It has absolutely no integrity of any kind today. There's an American law against spoofing it in theory, but it's not very enforcable at the moment. Understand it's literally a string that you can set to any value, including complete nonsense, and calls will often work.
Heck, we constantly receive calls with invalid phone numbers. And it's not always on purpose, sometimes it's just crummy or small-time carriers in rural areas (no disrespect intended, this is just an honest statement from experience). If we ingress filtered all invalid numbers, it would almost certainly be devistating to our customers.
This is like with email. First we need an integrity mechanism, then we can start doing reputation and identifying bad actors.