How expensive is SF rent? Use Unix commands to find out | Hacker News Reader

How expensive is SF rent? Use Unix commands to find out

4 pointsjph10 comments

How expensive is SF rent? Here's a coding challenge that uses Unix command line tools to find out.

Hacker friends suggested I post the result and code here because I'm writing open source statistics software and this is a practical example of results and code.

The results for a one bedroom in SF: mean $3348, median $3475, interquartile range $1514, and standard deviation $1130.

How to use Unix commands to find out: use `curl` to fetch Craigslist apartment listings, then `sed`, `grep`, and `tr` to filter and parse the data, then `num` to calculate the statistics.

    curl -s http://sfbay.craigslist.org/search/sfc/apa\?min_price\=1\&bedrooms\=1 |
    sed 's/\(<p class="row"\)/\n\1/g' |
    grep ' 1br ' |
    tr ">" "\n" |
    sed -n 's/^\$\([0-9]\+\).*/\1/p' |
    num mean median interquartile-range standard-deviation OFMT="%.0d"

The software I'm personally writing is the `num` command to calculate the statistics. The software is free at http://www.numcommand.com

@echo "very expensive"

What happens if the price overflows the limit of a 32-bit integer?

If the price overflows 32-bit, then everyone in SF moves. :)

For this script, the fetch returns at most 100 rows, so won't overflow.

More broadly, the Num command runs using awk, and number handling depends on your system's awk implementation and C libraries. Current awk implementations use an internal representation of all numbers, including integers, that uses double-precision floating-point numbers. I know GNU awk supports arbitrary precision arithmetic, and also can use --bignum, GNU MPFR and GNU MP (GMP); these will be great additions to the Num command for the next major version upgrade.

I had a perl script to do mean, median and std dev; it'd be great to have this functionality available through the OS vendor as a package.

Yes. I'm writing the Num command specifically to become an OS vendor package. (For example I'm writing Num using Awk, which is available on every current POSIX system.)

Love it!

sed: RE error: illegal byte sequence gawk: cmd. line:1691: fatal: division by zero attempted

Can you contact me about this? I'll diagnose it. My direct email is joel@joelparkerhenderson.com. I really appreciate it.

FreeBSD 11-CURRENT

   num: command not found
   tr: Illegal byte sequence

To install the num command, use http://www.numcommand.com

Reply on news.ycombinator.com