I'd recommend reporting your card likely stolen and getting a new number. The slight inconvenience is worth the peace of mind.
How much of your time spent chasing down and changing a CC number is worth a maximal risk of $500 with a vero low probability of occurring?
EDIT: Thanks to nenolod for the correction!
I'm pretty sure you're off by a factor of ten. If someone makes fraudulent charges with your credit card, your maximum liability is $50, and you'll only have to pay that if the charges occur after you've reported the card stolen. (There are different rules for debit cards, that doesn't apply here.)