I've seen immigration officers go on power trips before, and it ain't pretty. That much discretion with little if any opportunity for the potential victims to retaliate goes to some people's heads. And then they just need an excuse.
I told him he had no such authority, unlike with the room full of Mexicans around me he was used to harrassing, and even if I didn't answer his (non-citizenship related) questions he'd still have to let me in.
By god did that piss him off, and I spent many hours detained, but in I went. After hearing his irate and false claims that my passport would be revoked, not allowed in, etc etc.
I got put on CBPs shit list many years before that because I helped some people in a scary country which made some dumb uninformed government worker sad face. They do that stuff no matter how nice I am. Finally I learned they will treat me like shit and break my things even when I am nice so I should at least be assertive about my rights in the process.