I get asked where I am from all the time, because of my strange accent. However I've never before been asked about what visa I came in on.

The way he talked about the Visas, gave me the impression of someone who has been conditioned to believe that "those Visas" are evil.

Or maybe I completely misread it, and he was just trying to make conversation as you said, to calm me down. But choosing the Visa subject seemed quite odd. Usually when I mention my home country people chose a subject like history, mythology, holidays or current affairs.