Over the few months I've had it running I've needed to progressively create failsafes for IP addresses that I know are trustworthy so I don't lock myself out. I've also started tiering the importance of blocking based on different sets of ports which are being probed. I've also discovered that there's a significant amount of Uninvited Activity coming from "security" companies in their pro-active scanning of the entire IPv4 space - which I don't trust at all and ban with prejudice.
It's messy and I need to update and add many explanations, but it's on Github if anyone wants a laugh: https://github.com/UninvitedActivity/UninvitedActivity
(I'm aware of various limitations and footguns inherent in this un-subtle approach but, as another commenter elsewhere alluded to, "it makes me feel better". I also think that a fair bit of processing volume can be taken off IDS' if a heap of "known garbage" traffic is blocked prior - it's all about tiers).