Hacker News Reader

Comment by danudey | Hacker News Reader

This comment made me go and try out Invoke-WebRequest because I'd forgotten what it was like.

Being able to do simple web scraping like this in a "shell" is pretty great:

    $hn = Invoke-WebRequest https://news.ycombinator.com/
    $hn.AllElements | Where {$_.TagName -eq "a" } | Where { $_.Class -eq "StoryLink" } | select "innerHtml"

View on HN

IWR and ConvertFrom-Json are what got me hooked on powershell. No more learning complex switches for curl or crazy JQ syntax. It's all built in and makes sense.

FWIW, I already have a command line tool that can do that installed in regular Unix: https://github.com/ericchiang/pup The docs even use the same example of scraping HN.

How does Invoke-WebRequest decide if it’s looking at HTML or something else?

Well, you use Invoke-RestMethod for JSON/XML stuff.

Or decide yourself for Invoke-WebRequest to pipe it with whatever the data: ConvertFrom-{JSON,XML,CSV}

With Invoke-WebRequest you get back headers, raw response, etc - and you get to decide whats next. With the RestMethod - just the content as an object.

There is an param to declare response type parsing and if you want to use different parsing. Older versions of powershell use IE and webrequest will randomly barf. It really sucked. You can also use a rest request method that uses json.net to deserialize I believe .

Reply on news.ycombinator.com