Yes, flying is quick, but the more people who fly the larger the airports you need, and large airports need to be situated outside of urban centers.
Getting the Eurostar from London to Paris would take you no more than three hours (2:15 for the journey, checking in 45 minutes before departure). Flying from London to Paris would take two hours (1:15 for the journey, again checking in 45 minutes before departure), but you've got to get to and from the airport, whereas the train is taking you directly into the city.
Across medium distances (SF to LA, Paris to London, etc) high speed rail typically provides better passenger comfort, lower CO2 emissions, and travel times as fast as flying once you factor in transit.