The right time to refuse a search is always

When someone says the "Well, if you've got nothing to hide..." argument around me, I immediate ask for their wallet/purse, keys to car & house and cell phone (with password/unlock pattern). The usual reply is something to the effect of "Why would I give those to you?" I respond with, "Because you have nothing to hide, right?"

Most of the time, the conversation quickly devolves due to the fact that I'm not the Police. Which is obviously irrelevant; but many people can't see the forest of their rights through the trees of an "authority figure".

Occasionally someone gets my point and conceeds that they, like everyone else, have things that are private.

In the US, at least, the 4th amendment protections are significantly weaker at the border due to assumption of routine searches being "reasonable."

No clue what Canada's stance is on this, but worth knowing before you stand your ground with the border guard.

http://en.wikipedia.org/wiki/Border_search_exception

Two relevant articles for those wishing to be more articulate about these points. We all should be concerned about citizens not realizing the problems derived from the "nothing to hide" argument:

http://www.wired.com/2013/06/why-i-have-nothing-to-hide-is-t...

http://m.chronicle.com/article/Why-Privacy-Matters-Even-if/1...