Also, if a target site is behind Cloudflare then you probably won’t be able to masquerade as any of the popular bots - they block fake google/yandex bots.