A registrar only controls NS records (and glue A/AAAA) required for delegation. It seems impossible to enforce the no address record policy.
Right, so it would be necessary to force the NS records to point to registrar-owned nameservers which would be the point of enforcement. This is of course bad for a variety of reasons, not least how convention-violating it is.
Personally, I wouldn't want to do that. If I register a domain, I should be able to point it anywhere you want.
Plus, you'll need cooperation from all the registrars out there that are selling domains in your .nonpublic TLD. Unless you're the only one.
Thanks for your comments - my goal with the writeup was to attempt to sound out why the idea of ".nonpublic" would or would not work, and I think it's clear that it's not really viable. That's a pity, as the current state of affairs isn't great. It was interesting to explore though.