It's a security risk even if you don't use it. Someone just seeing your card briefly could copy it as the stripe has the same info that's printed on it.

Incidentally, this discussion is filled with people who have no idea how payments are authorized.