> How do you avoid this

IPv6 of course.

> or is it just not important

Port knocking not a security feature anyway.