Two reasons:

1) Because the whole system was designed back in the day of one single integrated phone company (Ma Bell/ATT in the US), so having any form of authentication was unnecessary, because only the one single phone company was ever responsible for handling anything phone related.

2) Because caller-id is an 'add on' that has no relationship to the underlying phone number "address" that is actually used to route phone calls. It is just an extra text string sent along from the call initiator for the purpose of appearing on a display at the destination end.

Of course you can spoof anybody's address. Depending on the protocol, you may not be able to get it working. But if it's UDP, you can definitely spoof someone's IP address.