Comment by malux85 | Hacker News Reader

Comment by malux85 | Hacker News Reader

Parent Full thread

It's not just the grounds that they're on a system where that's all they have. The command line tools are largely written in pure C. They are blazingly fast if used correctly, and robust to all sorts of failures you didn't even think of, they are ready to go and they are QUICK to both use and develop for.

Here's an example:

I was working at a consultancy one day as a data engineer, and across the desk was a junior data scientist. We were preparing for a meetup and the manager said "this website has a bunch of email addresses on it I'd love to have them in a CSV" (they were in a format where it wasn't easy to copy + paste them)

The junior data scientist said "I can do that" and the manager said "only if it takes less than 2 minutes", the junior, thinking he'd have to: setup a python environment, find a parsing lib, write code to pull the page, parse the dom, search for the email addresses, write them to a file,

said it could be done in an hour or two.

I viewed the code on the page, and saw the html wasn't minified, and the "a" tags were 1 per line, so I :

use wget to pull the webpage cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt

^ unix wizards will tell me a better way of doing this, but it took me literally 60 seconds to go from "I wonder if" by the manager to "here's your CSV"

THATS WHY WE USE COMMAND LINE TOOLS, they are a swiss army knife

See the classic of the genre, "Command-line Tools can be 235x Faster than your Hadoop Cluster"

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

This one too - 10 pages of Pascal to six shell commands https://leancrew.com/all-this/2011/12/more-shell-less-egg/

> unix wizards will tell me a better way of doing this

Great story. I won't claim to be a wizard, but here's mine:

    grep -Eo "mailto:[^\"']*" webpage.html | sed 's/^mailto://' | sort -u

The pattern means "mailto: and then everything up to a single or double quote", and the -o flag means "only show the matches". The main benefit of this approach is that it works even if you have multiple email addresses on a line.

You are a wizard, today I learn, thank you!

I get that it wasn't the point of your post, but Bert Bos' html-xml-utils¹ are a great set of tools for these tasks. `hxwls` lists links, so `hxwls <URL or file> | grep mailto | ...`.

There is a whole heap of tools in the package, and they're definitely worth a look if you like to poke at the web from a term. Plus they handle a lot of the nasty corner cases that will pop up if you try to treat HTML as simple ASCII text.

1. https://www.w3.org/Tools/HTML-XML-utils/

In that particular situation I imagine that a smart line of Javascript in the developer console would also have worked. Without the email file (so you must copy the result from the dev console), but it is considerably easier than using curl or wget when the page is behind a login page.

I came up with:

document.documentElement.innerHTML.split('\n').forEach( (line) => line.includes('mailto:') ? console.log(line) : 0 )

But please push this further (is there a shorter way to say document.documentElement.innerHTML ?)

Not minimized but generates a file 'emails.txt':

a = document.createElement('a');

a.download = 'emails.txt';

a.href = `data:text/plain,${Array.from(document.querySelectorAll('a')).map(a => a.href).filter(s => ~s.indexOf('mailto')).join(',')}`;

document.body.appendChild(a);

a.click();

That's a really cool trick for creating a file you can download.

something like this:

`document.getElementsByTagName('a').map(elem => elem.href)`

this works at least in chrome console (to get link urls, not emails)

[...($$('a'))].map(e => e.href)

we shared files on a mounted NAS. getting him the CSV was just a "cp"

can you share the smart line of javascript? You have 60 seconds, go.

Parent mentioned it was behind a login page - in which case using wget becomes more of a hassle.

I wasn't disputing that, I was curious to see the line of javascript

On top of the other recommendations, you could replace the separate "wget the page" and "cat it into the pipe" steps with a single "curl https://url.to/webpage.html" (assuming you have curl on your machine, which is practically guaranteed nowadays).

More to the point, though: the junior data scientist might not have been entirely on the wrong foot there. There's a tradeoff between "I need to do this thing quickly" v. "I need to do this thing repeatably and robustly"; whipping up a quick shell one-liner to do it falls into the former category, and also can drive some ideas for implementing the latter category (where you'd almost certainly want to be properly parsing the HMTL, properly accounting for commas in the email addresses (and yes, email addresses can have commas per RFC 5321), etc.).

> cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt

This is great! I can't believe I've never run into "cut" before. In similar spots, my workflow would have been filtering the lines with grep and then opening the file in VI to finish the task.

Another alternative would be to do the transformation with sed or awk, or if you require more complex regexes, perl with the "-i" and "-e (single line of code)" options.

Cut has a mirror image program, "paste", but I've not yet hit a use case for it.

I used it once for counting n-grams while making a markov chain generator

yip, that's why learning vi is essential for comuputational survivalism. most unix systems has it

We played games like this back in university, doing all sorts of simple off-the-cuff data transformation and extraction. It taught me an eternal respect for the Unix command line as a quick way to hack up data into a usable shape, and it still helps me regularly to this day.

It's amazing how handy the console and presence of jquery in basically every webpage is. Got a list of expandable elements and don't feel like clicking every_single_one? $(".whatever").click() (not tested but you get the idea) does wonders.

Yea...knowing basic Linux commands can save you hours of writing scripts in as little as a week for me. Also it saves sanity as having implicit loops is nice.

Reply on news.ycombinator.com