The average American would likely fail to identify all but the most exaggerated Canadian accent (adding "eh?" to the end of every sentence). That's why we're constantly shocked when Hollywood blockbuster actors are Canadian.
My limited experience is that differences inside these two countries (e.g. New Foundland accent to Winnipeg accent, or Texas to Boston) are way bigger than averaged difference between countries (i.e. Manitoba to Minnesota). Most "Canadian" English is broadly similar to most "American" English, especially the standardized TV / Movie kind.
Then splash in some Quebecoise and that will really diversify things. Much of rural Quebec may not even speak English altogether. But you can tell the difference between Quebec French and France French pretty easily, with the latter sounding smoother. I've never been out west but I assume that BC and Alberta is the most 'normal' of the Canadian differing accents.
If Linus Tech Tips is anything to go by, this tracks. All of their hosts have very "generic American" sounding accents. They are very very close to Washington state which doesn't have a strong accent in my experience. Contrast this with Minnesota or northern Michigan and how "similar" they sound to the stereotypical Canadian accent.
Linus has a bit of a Valley Girl sometimes and wholly acknowledges it.
I had an easier time communicating even in smaller towns in Quebec, where it seemed that most people were capable of speaking a little English, even if they spoke French all the time.