Command line tools for web developers | Hacker News Reader

Command line tools for web developers

coderholic.com

322 pointscoderholic68 comments

Ever wanted to know the progress of a long running operation? Copying a file, importing a mysql db, etc. Pipe Viewer (pv) is what you need: http://www.ivarch.com/programs/pv.shtml.

It lets you monitor the progress of any piped command and gives you time elapsed, speed, time left, and a progress bar (wget style).

Copy a file:

  pv sourcefile > destfile

Import mysql db:

  pv dump.sql| mysql -uroot -p db

More tricks: http://blog.urfix.com/9-tricks-pv-pipe-viewer/

edit: To install on OS X just do

  brew install pv

Pipe Viewer dramatically increased my productivity for large scale data processing. In particular, it lets you quickly know whether something will take 5 minutes, or 2 hours, so you can plan accordingly. It's painful watching people try to do this without pv.

For me the most invaluable command line tool that I have ever used is S3 Tools:

http://s3tools.org/s3tools

It provides a great suite of tools for interacting with S3, and is best used on an EC2 instance you are connected to via SSH. It is also ridiculously fast, much faster than trying to interact with S3 from a local FTP browser, or even from Amazon's own S3 dashboard. For example on my computer using Amazon's own web facing dashboard it will take about 30-45 minutes to make 14000 files on S3 public, versus via the command line tool downloaded and running on one of my EC2 instances it can make those files public within minutes.

I assume this is because it is local network traffic. Anyway, if you are ever in a bind and need to move a bunch of files to S3, I highly recommend S3 Tools. It has saved me many times.

Along those lines wget is the most powerful command line tool I've ever used, with so much capability. It is simply incredible when combined with S3 tools, allowing you to easily grab gigabytes of images off of a personal server or staging location and upload them to S3 very quickly.

And if you need to do more than move files around you can manage even more aspects of AWS including EC2 instances from the command line using this powerful command line tool:

http://www.timkay.com/aws/

Its great for writing automated bash scripts to manage your instances via cron jobs.

My main complaint with s3cmd is you can't easily throttle it, and on a busy machine its fast speeds can cause issues.

Good point. I guess in the past I have spun up a micro instance to run it, so I didn't have to run my S3 operations using one of my main production servers, and hence I didn't run into any problems with its unthrottled speed.

Perhaps lowering its priority would help.

nice s3cmd ...

Have you tried trickle[1]? It can limit the bandwidth of most programs I've tried it on.

  trickle -u 10 -d 20 ncftp

This runs ncftp capped at 10 KB/s up, 20 KB/s down.

[1] http://monkey.org/~marius/pages/?page=trickle

No -- but I will -- thx! I've tried nice, ionice, cpulimit and cstream.

I prefer curl'ing icanhazip.com, and I make it a quick alias in my .bashrc:

    alias myip="curl icanhazip.com"

Apachebench (ab) is a decent alternative to siege, and knowing how to use tcpdump and netcat comes in handy for debugging. Other than that, my new favorite command line tool over the past several months has to be vagrant, which lets you script and streamline VM creation and builds from the command line. If I need to completely reproduce my production environment on a test box, it's my utility of choice.

That only returned my IPv6 address so I can't get my IPv4 address with that.

Try ipv4.icanhazip.com and ipv6.icanhazip.com .

I use http://whatismyip.org/ frequently.

Theres also http://ifconfig.me/ which acts similarly. The page has a list of curl commands you can issue and what your output will be.

My most used cli tool outside of the default nuts and bolts is dtrx, the best and easiest file extractor for *nix. No more fiddling with flags or looking up, handles issues with putting lots of things in different directories or the wrong permissions on files. It has saved me a ton of time over the years.

http://brettcsmith.org/2007/dtrx/

Wow! Just wow! Bookmarked! A mere upvote is not enough for this! Thank you!!!

Questions:

- Is there a site pointing out amazing command line tools like this? (their existence as opposed to usage examples like at http://www.commandlinefu.com/)

- Is there a site listing the OS X equivalents for Linux command line tools? (e.g. what's the OS X equivalent for dos2unix / flip?)

(edit: merge 2 comments into 1)

Yep, check out http://onethingwell.org which is exactly that. Software (incl cli) that do one thing well, as according to the UNIX philosophy. It's nicely tagged as well, so you can click the 'osx' tag and get tools just for OS X and so forth.

I actually found dtrx on hacker news though a few years ago, but it's also on onethingwell I believe.

Amazing, cheers.

Actually, for extracting tar files, using xf is enough.

    tar xf archive.tar.gz
    tar xf archive.tar.bz2

Still, this will be useful for other formats, I think. Thanks.

Don't you need the z and j flags, respectively? The way you write it works for uncompressed tars.

Did you check? GNU tar will guess based on file suffix.

That's cool, although most of the time when I use tar, it's in a command line like

    curl http://example.com/foo.tar.gz | tar zxf -

lftp is a practical SFTP, FTPS and FTP transfer program, including automatic upload/download resumption and synchronization (mirror) mode. Good for both interactive use and scripting.

lftp can also continue downloads and run multiple concurrent download threads, acting as a download accelerator.

for eg.

  lftp :~> pget -c -n 10 http://warez.com/large_file.mp4

will start 10 concurrent download processes (n) and continue (c). lftp is full of hidden gems.

Synchronization (mirror) mode is absolutely invaluable when you only have access to a machine via FTP.

curl -I and wget -S are particularly helpful when debugging redirects.

Sometimes I migrate URL schemes, and set up permanent redirects in my .htaccess files. Testing them in a browser is a real pain, because browsers cache the redirect (which is the point of having a permanent redirect), so even if you change the .htaccess, you still get the old response. And pressing the refresh button is no help, because that reloads the destination page, not the source of the redirect.

That's when a non-caching command line client saves your day.

http://freshmeat.net/projects/apachetop/

I find it amazingly useful.

screen is another tool I use a lot, specially when connecting to remote servers over ssh.

Screen is a bit like a window manager for consoles. I use it to start multiple (software) servers without daemon mode. I can then switch between their outputs with screen.

Screen also detaches the console from your ssh session. In my case this means that the servers keep running if I loose my ssh connection to the (hardware) server.

It's a very handy tool and definitely belongs in the articles list of tools.

    screen -d -m processName

is practically the easiest way to run and daemonize a process.

    nohup processName

But then the output goes into a black hole after you close your shell

You can always redirect stdout/stderr to a file, e.g. nohup someapp > someapp.log

The output goes into nohup.out automatically.

screen is one of those things that's been on my todo list for way too long, along with its alternatives(?), tmux and byobu. Anyone that uses/used all three and can offer a comparison?

Consider the more lightweight dtach and dvtm. Following a well-known tradition, they each do one thing only, and do it well ;-).

dtach is very tempting. Is there a nice way to keep all the scrollback history?

byobu is something like a theme or packaged config for screen, so it doesn't really need to be treated separately.

The biggest difference between tmux and screen is that tmux is a lot more flexible with regard to laying out groups of sub-terminals within the main terminal. screen is mostly limited to one-at-a-time, horizontal splitting or (in very recent versions) vertical splitting, while tmux lets you go nuts: http://tmux.sourceforge.net/tmux3.png

Also, GNU screen is very old and stable, while tmux is (so far) still new and flexible. For example, tmux very quickly added support for handling Unicode characters beyond U+FFFF, a feat that (so far as I know) screen still can't manage. That's only one example, but I'm sure there'll be more as time goes on.

EDIT: One other thing that tmux does that makes it better than screen: when I start up a tmux session from within an X11 session, tmux clears the $DISPLAY variable so that processes running inside tmux don't try to connect to the original X server - which may very well have gone away at that point. It's a small thing, but incredibly annoying when it happens.

I have used screen for many years and decided to finally give tmux a try about 2 months ago. I really enjoy it. It took some adjusting, but overall I like it better.

Things like this are largely due to personal preferences, though. All I can tell you is that I am happier with tmux than I was with screen. It is more modern, and the split screening capabilities are better.

I switched from screen to tmux a while back. The key thing for me is that configuration and manipulations/operations in general are more user-friendly. Otherwise, in terms of the basic use cases, they are essentially the same.

I switched from screen to tmux about three months ago. It's easier to configure, does vertical splits, and is under more active development (apparently).

GNU screen is probably the most indispensable command-line tool I've learned since vi.

(Sort of) in the same vein, I've recently started using xmonad as my window manager and so far it's a lot more comfortable than the Ubuntu default. You may need to learn a teeny but of Haskell to get up and running, but so far I've been OK copy-pasting from sample configs and muddling through.

When I bought a new Macbook and wasn't able to run linux, XMonad was really the only thing I missed. No matter how much Apple focuses on it's interface and making it easy to use, it's still incredibly slow and unintuitive to me.

Agreed using 'screen -x' is a trick that routinely blows minds. (and is occasionally even useful)

I hate that the default escape key sequence clobbers over Ctrl-A though, so the first thing I always have to do when I log into a new server or account is this:

$ echo 'escape ^uU' > ~/.screenrc

Or I quickly start tearing my hair out and screaming profanities every time I try to do something.

I prefer ` (backtick) as my escape key sequence for both screen and tmux.

If you do this, though, be careful what you paste - if it has backticks in it, screen will interpret them as escape characters.

It's been mentioned on Hacker News a few times before, but my project PageKite (and showoff.io and localtunnel) is designed to help out with quick web demos and collaboration.

Getting started is now two lines:

  curl http://pagekite.net/pk/pagekite-0.4.py >pagekite.py

  python pagekite.py 80 SOMENAME.pagekite.me

... answer a few questions and whatever is running on port 80 will be visible as https://SOMENAME.pagekite.me/ within moments, almost no matter what kind of network connection you have. :-)

There are also .deb and .rpm packages available for heavier users.

Oh-my-zsh with git and git flow plugins work well so far. https://github.com/robbyrussell/oh-my-zsh

Are there any graphical interfaces for load testing, in lieu of siege?

You might want to have a look at JMeter (http://jakarta.apache.org/jmeter/).

`curl ifconfig.me` is pretty awesome

But I didn't find anything else to be interesting.

ngrep was news to me. I think I've heard of it once before, recently, but didn't really realize what it was. Having not tried it, it sounds like a really nice option when something like Wireshark or Ethereal would be overkill or just too much effort to bother with.

Coderholic.com is Timing out, Anyone else having issues?

Database Error

Perhaps everyone is trying out siege.

Works for me, but here's a cache just in case: http://liveweb.archive.org/http://www.coderholic.com/invalua...

Note: ngrep can also be installed using brew on OS X.

Also available via MacPorts.

This was worth it for me for the reminder about siege alone, as 'ab' seems to have some serious issues in Lion.

also useful:

wget post-file.it --post-file /path/to/file

iMacros firefox addon for web automation via command line (when curl is not enough): http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Sup...

siege is a very useful tool for benchmarking. I use it a lot.

How come sending concurrent requests, as explained in the article, doesn't flag you as a DoS offender?

I was surprised that he ran siege against www.google.co.uk without some kind of "don't do this" disclaimer to new users. Running it against other people's websites is pretty poor form.

Google has rate limiting protection from a single IP. If you make too many requests, they send you to a captcha page.

If you run it against a public site, that may be what happens. However it's intended for stressing your own sites in a test environment, in which case you have full control over the config. Setting up your test environment to block your stress tester as a DoS offender would just be silly.

If it brings down the site, you might get into trouble. It's totally consider DoS.

Does siege support benchmarking gzipped HTTP responses? This is something ab doesn't seem to do and it's starting to annoy me.

Reply on news.ycombinator.com