Yeah, an important but little-understood fact about ZIP codes is that they are not an area, but a collection of points (≈mailboxes) and polylines (delivery routes); this is because they were created by the post office, who care about logistics, not land. You can try to impute geographical location from them and this mostly works, but there are fun edge cases such as an area where the streets going in one direction are in one ZIP code and all of the cross streets in a different one. The code for an address can also be changed at USPS’s convenience (making comparisons over time difficult); in extremely rural areas “your ZIP code” may be whichever town you prefer to pick your mail up from.
(My favorite fun fact about ZIP codes: 12345 is the General Electric headquarters and factory in Schenectady, New York. They get a lot of mail addressed to Santa Claus.)