The Millbrae stop is a very short transfer to SFO.
BART's kind of remedied the situation, but for a while their operational costs were so high that there was no direct SFO-Millbrae service during the day. You had to backtrack all the way to San Bruno. By 2019 there was a "shuttle", and now the red (but not yellow!) line serves SFO and Millbrae.
In addition the Caltrain/BART connection at Millbrae requires a lot of walking. It's tedious and poorly designed.
The BART connection does suck, but I dont know how it look 2 hours to get to the Caltrain Station.