I agree. I was once in a car with a Brit and Canadian and had a lot of trouble to understand them as a non-native English speaker. They could understand each other perfectly well. So, it must be not the specifics of the British or Canadian accent, but more about the fast speaking like you say.