I interview a lot of people. When candidates ask me this, they often say "I'm coming from work, and I don't want to over-dress and tip them off that I am interviewing. May I dress casually?"

I never say "no." This is always acceptable to me.