"Ask HN: How to deal with GDPR / cookie notices in the context of a crawler?"

https://news.ycombinator.com/item?id=17795750

Might give an idea.