I like the analogy with ports, I also do the same - ufw deny all :)