Hacker News Reader

Comment by jxy | Hacker News Reader

If you have your shell ready,

    curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \
        | json_pp | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'

View on HN

Nice!

To run this with macOS, I had to use the GNU version of sed. I installed it with

    $ brew install gnu-sed

And it is then called with 'gsed' instead of 'sed'.

As an avid Perl programmer, I had json_pp in my $PATH - for everyone else - it is here: https://metacpan.org/pod/JSON::PP

You can install it with cpanm:

    $ cpanm JSON::PP

If you don't have cpanm, you can install it with

    $ curl -L https://cpanmin.us | perl - App::cpanminus

The modified command line from above then becomes:

    $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \
        | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'

Here is a little Bash function to encapsulate this:

    $ function c() { curl -sS "https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=$1" | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'; }

Which then allows you to use it like this:

    $ c hacker
    hacker news
    hacker typer
    hackerrank
    hackerman
    hackers movie
    hacker fare
    hackerone
    hackers cast
    hacker pschorr

For spaces in your query, use a '+':

    $ c New+York
    new york times
    new york and company
    new york giants
    new york post
    new york daily news
    new york weather

Improved version using jq. This properly URL encodes the query parameter and uses a much simpler sed command. Additionally the API returns UTF-8 encoded data when the user agent is specified. Requires curl >= 7.18.0.

    function c() {
        url='https://www.google.com/complete/search?client=hp&hl=en&xhr=t'
        # NB: user-agent must be specified to get back UTF-8 data!
        curl -H 'user-agent: Mozilla/5.0' -sSG --data-urlencode "q=$*" "$url" |
            jq -r .[1][][0] |
            sed 's,</\?b>,,g'
    }

Example:

    $ c ':)' ':('
    ) ( meaning
    ) ( emoticon
    ) ( ͡° ͜ʖ ͡°)
    ) ( emoticon meaning

You can use BSD sed by just inserting a semicolon into the sed pattern:

  sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}'

(Edited to remove previous advice about inserting a newline)

Would think jq[0] is more suitable if you're going to install a cli json parser.

[0]: https://stedolan.github.io/jq/

Here is the command with jq instead of json_pp

    $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'

(see my other comment about gsed vs sed)

You don't need sed, you can do it all in jq:

    curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq -r '.[1] | .[][0] | sub("</?b>"; ""; "gi")'

Here's the above modified to work with BSD sed:

  curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}'

(Edited to remove previous advice about inserting a newline)

I have learned the hard way never to type stuff like this into a shell because I might not like the results. It looks fine, but never again. ;)

What happened?

This was like 13 years ago, but it in essence rm -rvf / ... but it was a obfuscated one liner that some dick put out there as a joke. Result was having to restore a site and lose a few weeks of posts.

It's generally an easy way to introduce malicious shell scripts into remote machines operated by people who don't know about this.

There's a blog dedicated to shaming sites that do this: https://curlpipesh.tumblr.com/#_=_

Have you ever downloaded and installed a program through a web browser? Is there a practical difference between that and piping to sh, besides a GUI?

If you install through a GUI or a package manager, it will often check a cryptographic signature to validate that you downloaded the genuine thing, and you weren't man-in-the-middle'd with some malware. Also, it will check that nothing was corrupted in transit.

But generally speaking, you're right. You're taking a risk every time you run executable code from the internet, and if you don't want to live in a paranoid state of fear constantly, you have to balance security and convenience. If you check that it says "https" in the beginning and that you trust the source, then you're probably fine.

But they aren't piping to sh..

The infamous Barmin's patch doesn't have any `sh` mentions either:

``` echo "test... test... test..." | perl -e '$??s:;s:s;;$?::s;;=]=>%-{<-|}<&|`{;;y; -/:-@[-`{-};`-{/" -;;s;;$_;see' ```

Yeah, but if there's a code execution vulnerability in any of the tools in the pipeline it's as good as piping to sh.

Yup! Using curl at all is downright dangerous.

Well, it's all about risk assessment. Curl has had RCE security bugs before[1]. That doesn't mean curl is "downright dangerous" it just means that "it's safe because it doesn't pipe to sh" is also not a correct thing to say.

[1]: http://blog.volema.com/curl-rce.html

Incidentally, cURL just had a major security audit. The dev is doing due diligence to avoid RCE vulnerabilities.

Piping to jq and sed is just fine. I'm not sure where sh comes into this...

Unless there's a bug in curl, jq or sed that leads to an RCE, which is what my point was.

I suppose 'devoply pattern-matched "curl -sS <sth> | <sth else>" to "don't do that". I think that because that's exactly what I did now; it took me a few seconds to realize that - this time - the invocation is mostly harmless.

Reply on news.ycombinator.com