>>“None of your business,” I said.
That's not being polite. Being polite would be to say "Just visiting for fun".
It sounds like he didn't want to get through passport control.
Much of life is about jumping through silly arbitrary hoops to get what you want. In the case of getting through customs+immigration, you just keep quiet, don't cause a fuss, and tell them what they'd like to hear.