If someone really wanted to find out, they would be able to.

You could misdirect someone just passively asking, however as soon as the questions got more pointed, you'd have a hard time keeping up the charade.

Something as simple as the clothes you wear or the food you ate or the mannerisms you had could "give you up".