I am using Linkchecker: https://github.com/linkchecker/linkchecker/

Unfortunately, many web servers will block bots, either immediately or after a few requests.