You're right in that, usually high housing prices are a symptom of scarcity, not a cause.
But from what I understand, in Palo Alto, much of that scarcity is artificial. The city, in the process of fending off the rapid, misshapen development of the rest of the Santa Clara Valley, passed a number of zoning laws which restricted height, imposed 'setbacks', and limited the subdivision of lots. While this did work to preserve the community to some extent, the population density is forced downwards.
At some cost to the residents who already live in Palo Alto. Berkeley was lain on an older pattern, with much influence from works such as 'a pattern language', and latin american towns. Consequently there's a high enough pedestrian density to support a town that's much more pleasant to walk in.