A road in a random neighborhood is still important to me even if I can't cut through it: The mere fact that I can easily travel to any destination connected by it is very valuable, even if I don't frequently use that ability for any single specific road segment. In addition to my own use, its very useful to me that other people I interact with can reach those properties (so that they can live near by and engage in commerce and friendship with me).
A complete road network has greater value than the sum of its components in isolation, and this is true even for components even where you can't use them for non-destination travel.
Following your logic, any sub-graph of roads that can't be transited to other subgraphs should entirely be paid for by the connected properties-- and that clearly is not how we do things.
Moreover, the "if I can, I should" thinking strongly incentives making neighborhoods into single-access navigational cul-de-sacs, which creates worse traffic for everyone when the local traffic for the neighborhood gets funneled into a single choke point and causes congestion.
A month or two ago I saw an image on reddit from florida where someone showed that it was an 8-mile drive to their backdoor neighbor due to a lack of connectivity between neighborhoods. ... that's the kind of garbage roadway network design we'd get if we can only use raw connectivity control the load and character of use of roadways.