> because accepting this framing doesn't really seem to leave a whole lot of legal wiggle room.

No, it’s not a problem. The government can’t grant or withdraw a privilege for impermissible reasons.