Furthermore, if you want cheaper housing, build more of it. I don't think there's an easier way around it.
1) The terminal transit station should be built so as to exit right into the central business district. The Caltrain station is too far from SOMA and lacks decent public transit. Meanwhile, Chicago's three transit stations exit right at the perimeter of the Loop (within walking distance of nearly any office building, and with a transit ring around the perimeter of the CBD). New York's two transit stations exit right into Midtown.
2) You have to allow high-density construction near the transit stations, so people can live within walking distance of commuter transit. Look at the Menlo Park Caltrain station. There's nothing around it. There are far smaller Westchester towns that have a substantial downtown core around their Metro North stations.
With housing prices being what they are in Silicon Valley, there should be 30+ story buildings ringing the Caltrain stations in Menlo Park, Palo Alto, Mountain View, etc.
Why would you see big residential buildings in Menlo Park or Palo Alto? SF, unlike most cities, has a lot of people commuting outwards towards the suburbs from the city each day. It's SF where you need the extra housing, not SV.
+1
My position is a bit stronger: It all starts with zoning.
There was a geek news item recently that note the difference in USA and Canadian sprawl, which the authors believed started with how parcels of farm land were measured out. In the USA, parcels were squarish, requiring roads everywhere, requiring everyone to have automobiles. Whereas Canada made theirs long and narrow, to optimize farm-to-market transportation. Subtly different initial conditions causing very different land use.
In the jurisdiction I live, better transit has been actively thwarted for decades by the suburbans. For example, years ago, we got an Oregon-style Growth Management Act, which tries to slow sprawl, preserving farm lands and habitat. The suburbans frame it as anti-growth. But in fact GMO is pro urban and opposition is anti-urban.
I am fascinated by the increasing urbanization of the under 30 demographic: It's happening despite the pro-sprawl incentives and policies. An example of society way out in front of policy.
Locally, I think this is perfectly captured by the head quarters of Microsoft and Amazon. In the 90s, Microsoft created a campus atmosphere in former second growth woods (Bellevue/Redmond). Having nearly everything newly minted university graduates would need right there on campus. It was very desirable.
Now in the 2010s, Amazon is transforming an urban area, South Lake Union. The area now has housing, hip food, great access to parks, culture, etc. In fact, in order to attract the young talent they need, I can't imagine Amazon locating anywhere other than an urban environment.
My wife and I were just talking about this the other day. The "campus" tendency of old-line tech firms seems to be the result of wanting a smooth transition between college and work for new employees. But the newest batch of college kids is more urban and don't necessarily want their office to resemble a college campus or the suburbs where they grew up.
Basically the same principle except in this case the transport system was the river and the land usage is optimized to give as many people access to it as possible. Interesting to see the idea cropping up again in reference to modern public transit.
What I'd most like to see from Caltrain is more frequent trains. It's annoying to arrive only a few seconds late (due to a Muni delay) and then have to wait an hour for the next one. The 9:36 southbound they introduced in October helps. I'd like to see more of this.
Edit: Couldn't find actual completion date, so edited to reflect that fact.
Edit 2: Was referring to this: http://en.wikipedia.org/wiki/Central_Subway
Edit 3 (Last one!): This page from a while back says service begins 2019 http://www.sfcta.org/delivering-transportation-improvements/...
Pro tip: Get off at 4th and King and walk to Caltrain if you are cutting it close. You'll beat MUNI most of the time.