I have found nginx bad bot blocker [1] very handy for this. I have it setup to respond with 444 and it allows me to add my own rules very easily.

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...