A handshake is a good example: when you go give one to someone, you don't actually touch them, you extend your hand and wait for their explicit (if non-verbal) consent.

As for patting backs, do avoid doing so. Many of us don't like it even if it's totally non-sexual.