It's primarily the constant, repeated threats of annexation ("51st state"). Americans don't seem to understand how much these have tanked the relationship between the two countries.
It's an easy word to say, and the implication that it would be peaceful or non-violent because of economic pressures ignores the fact that Canadians don't want it.