You just have to make sure you don't accidentally overstay your visa in Canada with either country since they probably check this by entry/exit dates on a specific passport.