"Being polite would be to say "Just visiting for fun".
thats not being polite, that is answering the question and thereby opening yourself up to accusations that you were being misleading.
"None of your business" was a simple statement of fact, neither rude nor polite.