He said Chinatown, not SF Chinatown, I assumed he was referring to LA's Chinatown, which definitely exists.
SF Chinatown is just a tourist trap these days. Many more Chinese live in the Richmond neighborhood and throughout the bay area. Also, many of the Cantonese speakers in the Bay Area (usually south bay) are actually from Vietnam (ethnic Chinese refuges who left during the Chinese/Vietnam war in the late 70s/early 80s). I assume that is quite similar to the situation in LA, but have no direct knowledge.
There is pre-existing immigration from british areas, but they have mostly diffused throughout the country and are probably on their 4th or 5th generations by now. If you meet someone in the states who speaks Cantonese, they are more likely to descend from Vietnam, or somewhere else in Southeast Asia, rather than Hong Kong. Though I once met a cantonese speaker who immigrated from...Belize.
But I guess you are right...over a few hundred years and many intermediate destinations they did eventually come from Guangzhou/Hong Kong :)