Command line tools every web dev needs to know | Hacker News Reader

Command line tools every web dev needs to know

coderholic.com

232 pointscountessa68 comments

quite relevant: http://www.commandlinefu.com a great collection of cli uses and abuses

one of my favorites is querying dns to get wiki excerpts

    dig +short txt bmw.wp.dg.cx
    "(BMW), is an independent German automobile manufacturing company founded in 1916. It also produces motorcycles, is the owner of the MINI brand and is the parent company of Rolls-Royce Motor Cars. http://en.wikipedia.org/wiki/BMW"

http://www.commandlinefu.com/commands/view/2829/query-wikipe...

I've got something like this in my .bashrc so I can access commandlinefu from within the shell:

  cmdfu(){ wget -qO - "http://www.commandlinefu.com/commands/matching/$@/$(echo -n "$@" | openssl base64)/plaintext"; }

It's like man but for one-liners.

Wow this is so cool, I'm adding this to my .bashrc as well! Thanks for the tip :)

Interesting. Can you explain how the command works? I did look at the commandlinefu API page but didn't get it. I know UNIX and also specifically what "$@" means in UNIX but didn't get the meaning of the two uses of it in the commandlinefu URL you show, nor what the openssl part does.

It seems that the API bizarrely expects:

    [URL]/matching/ssh/c3No - Search results for the query
    'ssh' (note that the final segment is a base64-encoding
    of the search query)

So it's passing the command args through 'openssl base64' to encode them as required by the api.

     $ echo -n ssh | openssl base64
     c3No

It's not immediately clear to me why they require both - base64 encoding might be useful to allow non-printable/special-purpose chars in the URL, but we already have a perfectly good URL-encoding scheme for that.

And requiring both (it doesn't seem to work with only one) is just silly.

commandlinefu.com author here. Agreed, the API is odd and needs rewriting - I was trying to work around the idiosyncrasies of CodeIgniter which didn't support query parameters (hilarious) at the time of writing. The first segment (eg 'ssh' in /matching/ssh/c3No') isn't actually used - it's just there for SEO. You can insert junk there if you want - both aren't required. I'll update the docs to make this clearer.

There is a Django-rewrite of commandlinefu on the way including a saner API.

Thanks to all who answered.

The openssl part is just using openssl to base64-encode "$@":

  $ echo 'test' | openssl base64
  dGVzdAo=

Which is the same as:

  $ echo 'test' | base64
  dGVzdAo=

The URL is the same as:

  args="$@"
  b64=$(echo -n "$@" | openssl base64)
  url="http://www.commandlinefu.com/commands/matching/$args/$b64/plaintext"

For these purposes though, "$@" could probably be replaced with "$*". Technically the URL is invalid since "$@" could introduce non-escaped spaces into the URL.

Well, I only copied it myself, but it's not that complicated. From the API docs we learn that we need our search terms both in verbatim and base64 encoded. The first $@ is the verbatim part, and the second one is the one involved in the base64 part.

$(<command>) gets replaced by the output of <command>, in our case that means the output of the openssl base64 call, which takes stdin, applies the base64 encoding and prints it out on stdout. echo -n $@ just serves to translate our parameters into something the openssl base64 call can work with (-n means it won't introduce a superfluous newline). I'm not sure why the script uses openssl base64 instead of just base64 (a GNU coreutils utility).

I agree with the sibling that the API requirement to include both verbatim and base64-encoded search terms are fairly bizarre.

We are interviewing for experienced developers at the moment and the lack of general unix knowledge is really surprising.

A little bit of shell scripting goes a long way in automating away dev work.

I agree whole-fully. I know many "experts" in various areas such as programming, web dev, security, and a few others.

Rare are the ones who know how to use these basic commandline tools.

The last time I've shown ngrep to a sysadmin he was like "omagad amazing" - always feels nice :)

But, that's not really an obscure tool. I initially thought everyone knew that, like most know nmap.

Turns out most know nmap and ssh because they're really that well known (generally they don't know most options)

Makes me sad! Command's line easy and fun!

Also, typing.

I am frequently amazed to see professional software developers with years of experience in the field, who only type using their two index fingers. You probably do not need to type 120 WPM to be an amazing programmer, but typing without looking at the keyboard is absolutely essential.

Why? What projects have slipped due to programmer's inability to type in all the necessary code on time?

I don't know, but typing that slowly means that there is a barrier between the thought and the deed. It like if you wrote a poem, then started speaking the letters one by one in Morse code --- and you refuse to learn Morse code so you have to look up each letter's code as you go.

My former coworker would use the mouse to copy and paste even 3-letter variable names rather than typing them in again. He'd also scroll his terminal window up and copy and paste old commands (again, all with the mouse) rather than using the up arrow, even after I told him about command line history and tab completion in case he didn't know, and would frequently mis-copy and miss the first letter off or have a stray $ at the front, causing him to start his mousing over from scratch rather than using the keyboard to correct it.

I am not the most patient of people and had to bite my tongue a few times when instructing him. I can understand being more comfortable with the mouse than keyboard shortcuts but programming is not the best profession for anyone keyboard-averse to that level.

Of course some people genuinely find the keyboard physically uncomfortable and there's no reason they couldn't still be excellent programmers - but it seems like it would impede the "flow".

A pointing device is often quicker than the keyboard without sufficient shortcuts, however, and this includes too many IDEs, including, for all practical purposes, every one you rarely use.

I'm not sure it makes a big difference. You're going to have to waste a HUGE amount of time doing things the inefficient way (whichever it is for the task at hand) before it will be the equal of arriving late for even just one day, or taking an extra 5 minutes at lunch.

Things I suspect you'd be better off getting good at than keyboard shortcuts:

* Regular expression search and replace

* The command line (default Windows command prompt is fine), and the POSIX-style command line tools (default Windows command line tools are NOT fine)

* Some kind of popular scripting-type language (perl/python/ruby/whatever/etc.), if you don't know one already

I like my keyboard shortcuts, but the above have saved me much, much more time over the years.

We must have worked with the same guy! This is a guy would literally right-click -> copy, right-click -> paste everything. It was so frustrating watching him do even the simplest of tasks.

Then there was the day when he had to merge a huge branch back in TFS. I try not to speak much of that day...

If typing is hard then you're less likely to go back over your code and correct typos and minor errors. At any rate, hunt-and-peck typists tend not to spend time going back over the code and formatting it nicely.

I think its more a question of needing to look at the keyboard and identify the keys you want to press. It breaks the developer's flow.

I used to be one of those. Until one day a colleague noticed and made a remark. Then I looked at other colleagues and realized my problem. I used http://www.gnu.org/software/gtypist/ to fix it.

I'm still trying, but it's demanding more discipline from me than I thought I'd need at first. It's hard to get rid of bad habits when they've already become muscle memory.

I actually learned to type mostly by playing text based games (muds [telnet elephant.org]). Having to type fast enough to not get killed by the monsters helped me learn. Of course alot of players just set up scripts or triggers in mud clients. I mostly used straight telnet to play.

I found http://play.typeracer.com/ a fun way to get my typing speed up. I originally just went there to play around with my shiny new mechanical keyboard (which turned out to be very nice to use) and then it replaced minesweeper for short mental breaks.

Admittedly, I was typing pretty fast before I started--I think I went from something like 80 wpm to 110. But it was certainly fun!

As an aside, I've found short mental breaks like that really help when I get stuck on some little issue or bug and am not making any progress. It also keeps me from getting too frustrated!

I'm in college taking some CS classes and the actual material is great, but the framework code they give us for projects is really poorly written Python. I've spent more time grappling with Python and their code than I have with AI or compiler concepts! Taking a break whenever I encountered yet another stupid bug helped me avoid spending hours going in circles.

I used to go to http://keybr.com/ but having a cli program is great. Thanks

This probably matters a lot less than younger people think. If you grew up touch-typing you got used to being able to put data in as fast (or almost as fast) as you could think it. If the input rate is slowed down, you feel like you're going to lose thoughts before they get captured.

Folks who did not grow up touch typing, though, simply learned to construct a longer-lasting mental model, which they can then write out however slowly. Think of it as a bigger buffer. In rare cases it can become enormous, and people write entire novels in their head before ever putting ink on paper (famous example: "The Diving Bell and the Butterfly", which was written at a rate of about one word every 2 minutes).

Or, they learned to create and capture design decisions in higher-density media (like diagrams or pseudocode), which, again, can then be typed in more slowly.

It's probably much better to judge developers by the quality of their output than how fast they type--especially since most software spends way longer in production and maintenance than it does being developed.

Having a slow typing speed still means that one spends a lot of time typing, rather than doing other things, even if one can still accomplish the same mental processes without having to record it "live".

Do you really type at full speed all the time though? Most of my time is spent thinking, and when I do type something it tends to get pipelined with more thinking. Typing faster wouldn't help much.

In fact, I'd wager that if someone is typing constantly at 100+ wpm they're probably writing bad code. Or very simple code, I guess. Or they're just transcribing UML or whatever, but nobody here does that anymore do they?

How many lines of code per day do you actually write? 10? 100? 1000?

If we assume something like 30 characters per line, and maybe five characters to a word, you'd be writing 600 "words" in a day where you'd write 100 lines, and as many as 6000 "words" in a 1000 loc day.

100 wpm for 600 words means you're saving 6 minutes per day over the 50 wpm guy. Not much, I doubt anyone would care about 6 mins one way or the other. 100 wpm for 6000 words means you're saving an hour over the 50 wpm guy. It's rare you write 1000 loc. I don't know if I've ever done it, myself.

But that's just code. How often do you write non-code related work? Documentation, email, notes, comments, spec, etc. 6000 words per day is probably not uncommon for a developer when you include the noncode writing a dev will do during a day, and when a 100 wpm guy gets done a whole hour before the 50 wpm guy, that's an undeniable advantage.

If you have two candidates who you're certain are equally good developers, then obviously the one who types faster will be more marginally more productive. But I wouldn't assume typing speed or style is an accurate proxy for programming ability.

I think it's rather a proxy of how long the person spends writing on a computer. For a programmer more time spent writing generally means more time programming, which means more experience.

I don't think time gains because of faster typing don't even factor into a decision, but broader knowledge/experience does.

It's perhaps one of the effective filters while hiring. Not just the typing speed, but the general use of command line tools and keyboard shortcuts. Most good developers that I know are masters of the keyboard. They rarely ever touch the mouse.

Do you allow them to install their favorite editor?

If someone watched me on Visual Studio without letting me install Viemu first, I'd look like an idiot.

Yep. Whatever editor/IDE they wanna use.

Another useful one; Fire up a instant web-server from local folder on http://localhost:8080:

> python -m SimpleHTTPServer

Also in case you need to use another port (such as 5000):

> python -m SimpleHTTPServer 5000

Note that it has changed in python3

  > python3 -m http.server
  Serving HTTP on 0.0.0.0 port 8000 ...

  > python3 -m http.server 8080
  Serving HTTP on 0.0.0.0 port 8080 ...

Or if you use php (5.4 needed)

> php -S localhost:8080

smtp server one liner :

    $ sudo python -m smtpd -n -c DebuggingServer localhost:25

> With the following command we test google with 20 concurrent connections for 30 seconds

Even though this is Google and they probably don't have a problem with this, I don't think you should benchmark servers that don't belong to you. It's not nice and could be mistaken for a DOS attack.

Google may actually give you a short-term IP ban (a day or so), so spamming their servers with spurious connections may not be in your best interest -- doubly so if you're at a workplace.

However, if I dislike the IT guys, this sounds like an awesome practical joke.

Ted, is that you?

Also check out HTTPie: https://github.com/jkbr/httpie/ I much prefer it over curl.

$ http GET http://yourmom.ca/

see also burl: https://github.com/visionmedia/burl been good for testing requests against json apis

see also Curlish: http://packages.python.org/curlish/

What's the advatage of HTTPie? Friendlier syntax?

Yesterday I used the ipfw command to run our product in "slow motion" to spot loading order and if animations started at the right places. Discovered alot of stuff that needs fixing. The ipfw command is for trottling bandwidth, and great for product designers like myself :) I'm sure it has other uses as well.

OS X has a useful Network Link Conditioner utility. It has built-in profiles such as lossy 3G connection, average wifi connection, good wifi connection etc.

Cool, where is it?

You have to have Developer Tools installed. I am at work right now, not on a mac, but I will try from memory. I believe you get it from Xcode > Developer tools > Get more tools. I don't recall the exact menu tree but I am sure you can google it.

After that, it's integrated into the System Preferences panel. You simply pick which network conditions you want to simulate.

I think it's also possible to launch it from the xcode Utilities folder w/o installation. Sorry I can't be more coherent w/o my machine in front of me.

    $ curl -I news.ycombinator.com
    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Cache-Control: private
    Connection: close

Keep in mind that this will fire a HEAD request to the server which, depending on web server you use in development, may not return the exact header you'd get with GET (IIRC, Jetty used to do this).

I like to use something like this in such situation:

    $ curl -s -D /dev/stderr news.ycombinator.com >/dev/null

I rarely find I use curl's -s without its -S, e.g. -sS. It seems curl's design is flawed in its definition of -s; perhaps it's historic and it was too late to change it.

The junk goes to stderr, so it won't impact anything but your screen when used from the command line. And sometimes (large transfers) you want that information. I think the default is sane. When you're using curl in a script you use -s, when you're using it interactively you want the diagnostics.

Yes, sometimes when it's a particularly large transfer I want a progress meter but I normally don't so it's in the way; perhaps when we all had modems. And from a script I'd still want to use -sS instead of -s because although the script is obviously checking curl's exit value the user wants to see curl's specific complaint before the script's own knock-on diagnostic.

    $ curl -s does.not.exist; echo $?
    6
    $ curl -sS does.not.exist; echo $?
    curl: (6) Couldn't resolve host 'does.not.exist'
    6
    $

You can also use the -i option which prints out the headers and the response body (which is perhaps a bit too noisy).

    curl -i http://news.ycombinator.com

    curl -v https://github.com

which dumps everything. Including, handily, the HTTPS handshake.

Among (too) many other things, the OpenSSL "openssl" command does Base64 en/decoding, calculates MD5 and SHA* hashes, creates, displays, converts, and verifies certificates, certificate chains, and CRLs, and even works as a convenient "micro-CA" for testing purposes.

I'm honoured that this post has made it to the front page a second time! Here are the comments from the first time around: http://news.ycombinator.com/item?id=2880800

That ngrep tool is pretty nifty. Has anyone done something like that with dtrace? (I know about tcpsnoop - something at a higher level)

If you're getting started with the command line, you might find helpful an entire series on Quick Left's blog for beginners and experienced devs alike: http://quickleft.com/blog/tag/command-line

swaks is another very nice tool for email-testing (isn't email also part of the web nowadays)

Cheers for that, will come in handy very soon. For those wondering

Deliver a standard test email, requiring CRAM-MD5 authentication as user me@example.com. An "X-Test" header will be added to the email body. The authentication password will be prompted for.

  swaks --to user@example.com --from me@example.com --auth CRAM-MD5 --auth-user me@example.com --header-X-Test "test email"

http://linux.die.net/man/1/swaks

Being able to use DNS tools is a requirement too.

Another alternative to ifconfig.me is icanhazip.com The main difference being that icanhazip.com supports ipv6.

if you use json a lot, this is super useful for prettifying stuff in command line.

echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

ant ?

why?

Reply on news.ycombinator.com