Hacker News Reader

Comment by jph | Hacker News Reader

This is great work, and runs fast. The documentation is well done and has plenty of examples.

Here's an example of datamash and R with timing.

    time datamash sstdev 1 < data.txt
    288891.28552648
    0.76s user 0.01s system 99% cpu 0.775 total

    time R --vanilla --slave -e \
    "x <- read.table('data.txt', header=F); sd(x\$V1);"
    288891.3
    2.68s user 0.06s system 99% cpu 2.761 total

(The data.txt file is 1 million lines, each line a random number 1 to 1 million. The timing is on a MacBook Pro Retina 13" 2014)

View on HN

For this example (1 column data), you get much closer results using R's scan function rather than read.table

  awk 'END{for(i=0;i<1000000;i++){ print int(rand() * 1000000) } }' </dev/null > data.txt
   
  time datamash sstdev 1 < data.txt
  288619.72189328
  0.72s user 0.01s system 99% cpu 0.736 total

  time R --vanilla --slave -e 'sd(scan("data.txt"))'
  Read 1000000 items
  [1] 288619.7
  1.09s user 0.04s system 99% cpu 1.134 total

R read.table read performance is fairly slow by default because it has to infer the types of columns and check for inline comments, quotes ect.

This seems like a better replacement for awk and bash one-liners to me than tasks I would use R for.

For instance counting unique elements.

  #naive approach
  time (sort data.txt | uniq | wc -l)
  632209
  13.09s user 0.04s system 101% cpu 12.984 total

  #using hashing
  time (awk '!a[$0]++' data.txt | wc -l)
  632209
  1.34s user 0.03s system 100% cpu 1.360 total

  #R
  time R --vanilla --slave -e 'length(unique(scan("data.txt")))'
  Read 1000000 items
  [1] 632209
  1.20s user 0.04s system 99% cpu 1.244 total

  #datamash
  time datamash countunique 1 <data.txt
  632209
  0.83s user 0.01s system 99% cpu 0.840 total

Quite good performance in that case, although R surprised me here as well.

In fact, the documentation mentions that their operators are tested to match those of R https://www.gnu.org/software/datamash/manual/datamash.html#S... which looks like a pretty neat idea.

That is not fair; you count R's start-up time and all the guesswork which read.table does and datamash doesn't have to do.

Well, it is fair if all you want to do is mash some data together. Why shouldn't startup times be taken into consideration?

Because with R you can and usually do multiple things with multiple data sources within one session, which effectively dissolves the start-up and load time. Even if reading one file and calculating mean or something with a single script is the only thing you do, you can use Rscript which runs R without loading heavy stuff like the methods package.

Reply on news.ycombinator.com