Does your spider face issues with cloudflare mentioned by Gigablast founder here.
This might explain why GigaBlast has a problem:
Because of bugs in the original Gigablast spidering code, the Findx crawler ended up on a blacklist in Project Honeypot as being “badly behaved” (fixed in our fork). That meant quite a bit of trouble for us because CDN providers, which are a very powerful hubs for internet traffic, put a lot of weight on this blacklist. Some of the most popular websites and services on the internet run through services like Cloudflare and other CDNs – so if you are in bad standing with them, suddenly a large part of the internet is not available, and we weren’t able index it.
extract from: https://web.archive.org/web/20190921180535/https://privacore...
Mojeek follows the robots.txt protocol so if a site doesn't want to be crawled by MojeekBot we respect that wish. There is also a generous crawl delay between pages on the same host.
Generally a 'badly behaved bot' will ignore robots.txt or hit a site too hard with requests.
Our bot uses a specific user agent which you can verify via DNS. https://www.mojeek.com/bot.html
What's the order of magnitude of this delay? milliseconds? hundreds of milliseconds? seconds? I'm curious what's considered 'polite' in this realm and how the various parties come to form opinions on this.
User-agent: bingbot
Allow : /
Crawl-delay: 10
Does this mean your spider is a fork of Gigablast? Is there some additional interesting technical information about how your code/infrastructure is set up?
https://blog.mojeek.com/2019/12/100-server-build-and-install...
We'll be writing about our tech stack in our next FAQs series; 3 of 4, this is 1 of 4:
https://blog.mojeek.com/2020/11/frequently-asked-questions-a...