To add to the sibling posts:

Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and opening the URLs they contain; that stopped working well a long time ago. Modern sites are essentially complex GUI applications that run inside...a browser. You need either a browser, or some browser-equivalent thing therefore in order to run them and properly crawl them.