As you mention there is plenty of cheap and large offices as you go south along the peninsula, but startups seem to consistently choose more expensive SOMA offices over these. Only startups that need a warehouse or other specialized facilities start their search in the south bay.
I think the reality is that its a combination of factors driving people to SF, from 'cool factor'/perception to quality of life issues (employee proximity) to 'cheaper-than-palo-alto' benchmarking.
By the way, if you're reading this thread and looking for office space, the south bay is a perfectly good place to look. Yes, it's not as glamorous, but it's both functional, frugal, and actually quite nice. The graveyard doesn't consist of dumpy factories, but pleasantly maintained offices with good facilities. You can even live nearby, as long as calm neighborhood is an acceptable condition for you.