AllowedIPs can be disabled if you want; just set it to 0.0.0.0/0. AllowedIPs is needed because netfilter can't "see" which public key an inbound packet is coming from, so by the time a packet gets to netfilter it's too late to accept/reject based on which peer sent it to us.