I told him he had no such authority, unlike with the room full of Mexicans around me he was used to harrassing, and even if I didn't answer his (non-citizenship related) questions he'd still have to let me in.
By god did that piss him off, and I spent many hours detained, but in I went. After hearing his irate and false claims that my passport would be revoked, not allowed in, etc etc.