Computational survivalist | Hacker News Reader

Computational survivalist

johndcook.com

90 pointsweinzierl62 comments

It's not just the grounds that they're on a system where that's all they have. The command line tools are largely written in pure C. They are blazingly fast if used correctly, and robust to all sorts of failures you didn't even think of, they are ready to go and they are QUICK to both use and develop for.

Here's an example:

I was working at a consultancy one day as a data engineer, and across the desk was a junior data scientist. We were preparing for a meetup and the manager said "this website has a bunch of email addresses on it I'd love to have them in a CSV" (they were in a format where it wasn't easy to copy + paste them)

The junior data scientist said "I can do that" and the manager said "only if it takes less than 2 minutes", the junior, thinking he'd have to: setup a python environment, find a parsing lib, write code to pull the page, parse the dom, search for the email addresses, write them to a file,

said it could be done in an hour or two.

I viewed the code on the page, and saw the html wasn't minified, and the "a" tags were 1 per line, so I :

use wget to pull the webpage cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt

^ unix wizards will tell me a better way of doing this, but it took me literally 60 seconds to go from "I wonder if" by the manager to "here's your CSV"

THATS WHY WE USE COMMAND LINE TOOLS, they are a swiss army knife

See the classic of the genre, "Command-line Tools can be 235x Faster than your Hadoop Cluster"

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

This one too - 10 pages of Pascal to six shell commands https://leancrew.com/all-this/2011/12/more-shell-less-egg/

> unix wizards will tell me a better way of doing this

Great story. I won't claim to be a wizard, but here's mine:

    grep -Eo "mailto:[^\"']*" webpage.html | sed 's/^mailto://' | sort -u

The pattern means "mailto: and then everything up to a single or double quote", and the -o flag means "only show the matches". The main benefit of this approach is that it works even if you have multiple email addresses on a line.

You are a wizard, today I learn, thank you!

I get that it wasn't the point of your post, but Bert Bos' html-xml-utils¹ are a great set of tools for these tasks. `hxwls` lists links, so `hxwls <URL or file> | grep mailto | ...`.

There is a whole heap of tools in the package, and they're definitely worth a look if you like to poke at the web from a term. Plus they handle a lot of the nasty corner cases that will pop up if you try to treat HTML as simple ASCII text.

1. https://www.w3.org/Tools/HTML-XML-utils/

In that particular situation I imagine that a smart line of Javascript in the developer console would also have worked. Without the email file (so you must copy the result from the dev console), but it is considerably easier than using curl or wget when the page is behind a login page.

I came up with:

document.documentElement.innerHTML.split('\n').forEach( (line) => line.includes('mailto:') ? console.log(line) : 0 )

But please push this further (is there a shorter way to say document.documentElement.innerHTML ?)

Not minimized but generates a file 'emails.txt':

a = document.createElement('a');

a.download = 'emails.txt';

a.href = `data:text/plain,${Array.from(document.querySelectorAll('a')).map(a => a.href).filter(s => ~s.indexOf('mailto')).join(',')}`;

document.body.appendChild(a);

a.click();

That's a really cool trick for creating a file you can download.

something like this:

`document.getElementsByTagName('a').map(elem => elem.href)`

this works at least in chrome console (to get link urls, not emails)

[...($$('a'))].map(e => e.href)

we shared files on a mounted NAS. getting him the CSV was just a "cp"

can you share the smart line of javascript? You have 60 seconds, go.

Parent mentioned it was behind a login page - in which case using wget becomes more of a hassle.

I wasn't disputing that, I was curious to see the line of javascript

On top of the other recommendations, you could replace the separate "wget the page" and "cat it into the pipe" steps with a single "curl https://url.to/webpage.html" (assuming you have curl on your machine, which is practically guaranteed nowadays).

More to the point, though: the junior data scientist might not have been entirely on the wrong foot there. There's a tradeoff between "I need to do this thing quickly" v. "I need to do this thing repeatably and robustly"; whipping up a quick shell one-liner to do it falls into the former category, and also can drive some ideas for implementing the latter category (where you'd almost certainly want to be properly parsing the HMTL, properly accounting for commas in the email addresses (and yes, email addresses can have commas per RFC 5321), etc.).

> cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt

This is great! I can't believe I've never run into "cut" before. In similar spots, my workflow would have been filtering the lines with grep and then opening the file in VI to finish the task.

Another alternative would be to do the transformation with sed or awk, or if you require more complex regexes, perl with the "-i" and "-e (single line of code)" options.

Cut has a mirror image program, "paste", but I've not yet hit a use case for it.

I used it once for counting n-grams while making a markov chain generator

yip, that's why learning vi is essential for comuputational survivalism. most unix systems has it

We played games like this back in university, doing all sorts of simple off-the-cuff data transformation and extraction. It taught me an eternal respect for the Unix command line as a quick way to hack up data into a usable shape, and it still helps me regularly to this day.

It's amazing how handy the console and presence of jquery in basically every webpage is. Got a list of expandable elements and don't feel like clicking every_single_one? $(".whatever").click() (not tested but you get the idea) does wonders.

Yea...knowing basic Linux commands can save you hours of writing scripts in as little as a week for me. Also it saves sanity as having implicit loops is nice.

I hate this attitude by clients/employers. "I have a job for you, here are the tools you may use" ... Gives me a Windows 10 laptop and the only way to get into the network is some Windows only Cisco vpn app. So here I am, expressing all my enthusiasm, all my creativity through Windows on an HP (1280x768 display :S) which I consider to be a very narrow channel for my creativity and it kills my enthusiasm. I want a self chosen laptop that runs Linux and a distro chosen by me. And I want a nice Logitech keyboard/mouse. I want my tools to serve me, not the IT department.

I mean, even my friend the electrician gets money ech year to buy tools. And he buys Makita and DeWalt, not that 10 euro for 50 tools package at Ikea.

Yeah I'm not forced to work there, anyone saying that is right. And I dream of starting my own business just because of this.

I work in an environment that is actively hostile to us. We've got McAfee agents crippling our machines, IBM management software doing periodic forced reboots with no warning, MITM proxies decrypting and capturing all TLS traffic, self signed CAs with a misconfigured keychain on our machines and no access to fix it, random policy changes that screw with our browsers, web filters that prevent us from accessing common websites, netskope agents that crash multiple times a day, etc. I'm sure there are more. If there's a way to spy on us, our machines have it. Obviously, we can't install any unapproved software, so I am grateful for the built in tools MacOS provides. Of course, if you need to reach out to the internet, you've gotta do a dance with certificates and proxies...

I think about quitting a lot, but I am pretty numb to it. I'm sure it could be worse, too.

What's keeping you from looking for somewhere else? Despair?

I like the people I work with, I am still growing despite the environment, and I am in the middle of a major project and don't want to quit before it is done.

Are you me or a fellow worker at my office? \s

Seriously, that kinda bs is time consuming and soul deadening.

While I agree up to a certain level, this opinion needs to stay within its limits.

I like the analogy of a photographer: Talents aside, a pro will always make better pictures using a cheap low-budget camera than any amateur will be able to do with the newest and best camera available.

IMHO the same holds true for any IT related job.

> Talents aside, a pro will always make better pictures using a cheap low-budget camera than any amateur will be able to do with the newest and best camera available.

Unfortunately it doesn't work like this.

Photography, as a low threshold occupation has tons of professional hacks who are bad with any equipment. At the same time there are countless amateurs who are top talent.

And it's not just a recent trend. Aside from photojournalism, formal portraiture and weddings, photography was always dominated and advanced by talented amateurs. There's no such thing as "professional landscape photographer" for example, even if some of them are able to make a modest living off their works after years of effort. Ansel Adams is a great example.

Also, no professional photographer would use a toy camera for anything other than a gag/vanity/show-off project. They infallibly use reliable, quality equipment day in day out.

> Me: "Talents aside, [...]

> You: "It doesn't work like this. [...] talented amateurs [...]

Uhm, okay.

Well, anyway, my point is pretty valid. If photography is not as much dependent on skills from your perspective (I wasn't talking about weddings and landscapes anyway) take something else. It really applies to everything.

Maybe music is a better fit? As someone that even actively plays piano I am still 100% confident that someone like Beethoven would run a better show on a $40 Casio keyboard than I would on a Yamaha C5.

I can see how he totally would. On the other hand, given a choice, he would likely prefer a better instrument.

The point is that a "consumer" camera in the hands of a talented photographer can easily produce better images than the "best camera in existence" in the hands of a lousy photographer.

Yes but that's quite a truism. An average developer on an $300 720p laptop would outdo someone who can't code, but they still would be more productive with something better.

I spent a few summers making a living photographing weddings. It was really hard.

For what it's worth, a lot of the best photos out there are from amateurs (probably the vast majority, there's just more amateur photographers).

The pro, however, should _always_ be able to at least get _good_ photos, while an amateur might, or might not, who knows? This is why people hire a pro for a wedding - you know that there's exactly one chance to get that first kiss, etc. and need to have someone reliable. My favourite wedding photos are from weddings I didn't get paid for but attended as a guest, though I wasn't worrying about running through a standard shot list (and the group portraits, which I never liked compared to photos of people having fun at the reception)

But if you want to utilize the pro to her/his full ability... Why would you for the sake of "security" rob an IT pro of the freedom to choose the tools she/he wants?

Small confession: I'm currently working from home, Win10 laptop runs out of juice, I forgot the charger. Managed to set up a reverse SSH tunnel from our company cluster to my home server, proceed to work via SSH and SSHFS but now on my personal Arch Linux laptop. Loving it, feeling a bit unethical though.

> But if you want to utilize the pro to her/his full ability... Why would you for the sake of "security" rob an IT pro of the freedom to choose the tools she/he wants?

An "IT pro" should know about general risk management in IT departments or reasons why you don't give everyone full admin rights or every software they demand or everyone a machine with totally different specs from the other ones (read as: their favorite one). It becomes an unmanageable chaos at some point, too.

Who will be responsible if anything bad happens? Do you even want to be that one? (Note: bad can really be anything on a scale from 0 to hell and beyond in IT)

I don't know, why would you want to give a specialist different tools from people who do a different job? Why would you let a carpenter handle a power drill, or let your electrician access the fuse box?

I'd say there's a fundamental difference I think between the examples. Windows and the software that comes with it isn't undesirable because it's cheap, but because it's locked down.

A similar, but different, concept that I've seen is "slow computing." It also involves the use of terminal applications over GUIs but with the goal of having an environment that changes less often, is easier to investigate and modify the source code, and is easier to understand exactly what is happening with your data.

After so many frustrating experiences with modern GUI apps significantly changing interfaces and functionality, phoning home and creating privacy concerns, or being just plain buggy, I've moved to "slow computing" and have found it a huge relief. And incidentally, not slow at all -- I'm far more productive.

The annoying thing is that GUIs don't have to be made this way, the complaints you have are relatively recent inventions.

> "command line tools on the grounds that someday they may be in an environment where that’s all they have. I think of this as a sort of computational survivalism."

I don't think this is a helpful characterization, because this invokes post-apocalyptic associations of a world where there are somehow only text terminals left. In the article itself, the author goes on to describe a scenario where he used generic text/command line tools to achieve functionality that had not been explicitly granted to him by a restrictive software policy at work.

As opposed to survivalism, I think it's more fitting to describe it as user empowerment. I'd argue that the premier usage scenario for these tools is not as a fallback in hard times, instead it's an important type of toolset integral to the craft.

>post-apocalyptic associations of a world where there are somehow only text terminals left

an SSH session?

I'm confused by the intent behind this question. The point of my post was that being able to work in terminal sessions is not survivalism, it's standard tools knowledge.

I am pleasantly surprised that this wasn't about the daft Collapse OS.

I understand where he is coming from and why he may have gotten that impression but it is more like "accubus ergonomics" essentially for reasons I will explain at the end.

The point of the command line is that the shell is very transportable, efficient, and powerful. Compare updating a syatem - you can just write or pass about a single shellscript to do what woukd otherwise require clicking through a ton of menus, watching bars grow, and waiting for it to need clicked. Not that big a deal for one person one system, a bigger deal for 5000.

Its main weakness is the required experience and knowledge to use so if you already have those capabilities the added costs are marginal.

This the name - it is like dealing with decimals or integers with an accubus vs a calculator and scratch paper. Surprisingly despite a millenia gap the calculator is slower - the user needs to read symbols and enter them one at a time with a hunt and peckish sort of mechanism. Meanwhile a skilled user will have its operation essentially in muscle memory and only need to read the end state. The accubus is rare in comparison because it generally isn't worth it on its own vs the easier alternatives. Plus the paper has a less volatile storage format than bead position.

Command line operation isn't for everyone and in most cases and that is okay but it still has a valid purpose.

abacus?

Agree with minimalism. The older I get, the more I value mastery of few things over a superficial grasp of many.

In some cases, this can feel a bit regressive. Stepping back to a bone-head simple pattern with fewer moving parts and less dependencies on complicated "others" I cannot see.

Part of why I hate WordPress with a passion. It's complex mess of plugins and themes, slapped together without the tight moderation and insistence on good style that we see in a place like the Python standard library.

This is not much of a computational survivalism, if you need to install extra software on target machine.

I mean, you might as well install Python.

Do your task with PowerShell or DOS batch, and then this might cut it as computational survivalism on that box.

EDIT: Here, the solution to the author's problem without needing to pull a *nix tool chain in.

  ((Get-Content "<file name>").ToCharArray() `
  | Where-Object { $_ -eq '<the character that you need>' } `
  | Measure-Object).Count

I mostly really like PowerShell, and it works for Windows where the CLR is now part of the OS. I do wish there was a Linux equivalent but lighter weight, but so far all I've seen are attempts to shoe-horn existing unix programs into that model. I wish there was a ground-up rewrite, an entire environment that used a PowerShell-like object model with no legacy dependencies.

Nu Shell looks promising (inspired by PowerShell). https://github.com/nushell/nushell

Eh, it looks decent, but it isn't really what I want. For one, its cross-platform nature means it has eschewed greater integration with the system. One of the great strengths of PowerShell is how it treats pretty much every system interface as the same kind of object as everything else. Linux has these weird text based interfaces in /sys/ and /proc/ that it'd be nice to wrap.

Secondly it looks like it depends on a fair amount of typical userland being available. It'd prefer something I could slap on top of busybox and be done. Something that only depended on the kernel.

But, maybe with some tweaking...

Take a look at Object Shell: https://github.com/geophile/osh

Requires Python, hardly meets my personal criteria for 'lightweight'.

Have you seen elvish[1]? Structured data shell that's just a single Go binary.

[1]: https://elv.sh

Closer, but there's no way to interface with it from the outside world. If I want to add a tool I have to awkwardly pass data to and from it using to/from-json on the shell side.

Yes, you are right. It would be a cool thing to have - an object bash. 8O

Reminds me of this great talk from Dave Beazley where "I talk about my experience serving as an expert-witness in a patent infringement lawsuit. In this work I was locked away in a hidden vault with 1.5 TB of C++ source code and a Python interpreter. Great fun ensued."

http://pyvideo.org/video/2645/discovering-python

This argument seems to pop up with respect to proprietary software, where folks can be reluctant to acquire a dependence on a commercial vendor making acceptable offerings in the future.

In more extreme cases, I've seen it as an argument against Microsoft software like Windows and Word.

In less extreme cases, I've adopted this attitude with respect to some tightly restricted engineering (CAD) software used back in grad school. I mean, it was really useful software, but I didn't feel that I could reasonably rely on having access to it whenever I'd need it.

Notice how the author had looked up how to do his example in Stack Overflow. I found that foraging through different man pages looking for a specific thing to do only once is just very time consuming. Luckily looking up information on places like Stack Exchange or even different blogs and forums (which I usually land on after using a search engine) is much more efficient.

Yes, the information contained might not be completely up-to-date or accurate, but at least it sets you up on the right path to search further.

This is also why I try to avoid customization and try to use everything with the “out of the box” defaults. Not absolutely, but as much as possible.

Agreed. "After the apocalypse, I can't use this editor or shell because my macros or aliases are on github and the net was blown up"

Now everything has gone back to serial, after the apocalypse you should be able to toggle in the boot loader by scraping two wires together...really fast!

Reply on news.ycombinator.com