This is done because otherwise, the towns get yelled at. Heavily. There are lots of interest groups involved in this kind of thing.
The bus provides an appealing (relative to the Caltrain) option that's appealing to people who could also easily afford to drive. (They're more convenient than the train or driving, to boot, whereas taking the train is much less convenient.) In doing so, they take tens of thousands of cars off the road each morning.
Hence, not very many private entities want anything to do with the market.
Variable fuel prices doesn't really change any of the other private mass transit systems, like air, ship, and train (or even taxi?).
Environmental requirements are essentially irrelevant, especially compared the the fuel that will be saved.
High capital investment? I could start with a single van and one frequently travelled road. Or provide the service exclusively to one employer (ala Google style).
Demand challenges meet multi linear optimization. Our fire stations, electrical grids, gasoline stations, highways, office buildings, and eateries all have demand challenges.
Why would the profit be especially lower? If it were too low, entities would leave the market, if it were too high entities would enter the market.