I agree, when you see that kind of page, you auto close.
17 karma · joined May 24, 2017
Thats what I would use (If target machines are windows)
I still havent tried Net core on Unix machines so I cant say much.
If you pack it into exe you lose the cross-platform thing, so you might aswell use any other language.
https://en.wikipedia.org/wiki/Robots_exclusion_standard
But basically scrapping is a loophole, you are not doing nothing wrong because everything is accessible with browser. Still web admins get upset.
For tools I use now selenium with phantomJS, but you can also use scrappy.
As long as you don´t claim alot of bandwith that other users could use should be fine