Faster Command Line Tools in D

Sometimes I wonder if software should error first, then when you bounded the failure space, you iterate on the success space as you see fit.

https://github.com/eBay/tsv-utils-dlang

They basically explored new languages to rewrite some perl script in and liked D enough to shift over. They have other tooling in other languages, my guess is they'll unify a good amount of it in D. Disclaimer: this is based on my own assumption that they like D so much they want to just use it all over. It wouldn't surprise me to see this confirmed by an eBay employee. Although seeing as he wrote TSV Utilities, it really wouldn't surprise me if he wants to rewrite all in-house tooling he uses in D as the repository states.

PyPy probably manages to optimize the hash map/method call lookups for these small programs, which explains the speedups. Removing memory allocations is still hard.

The D language provides finer mechanisms to control memory and data structures. This makes the language larger, but enables you to optimize if it becomes necessary.

Still, I agree and I would like to see a Python expert to optimize it.

Edit: do you mean the tsv utilities? Because they're working fine here on the Creator's Update.

People use Python all the time to manipulate data sets >= 100G in size despite its speed failings at that size. Why? Because Pandas is just so damn convenient. It would take me a grand total of 30 seconds to write Pandas code which read a TSV and gave me the sum of two multiplied together columns grouped by the day of a timestamp column. Doing that in C would take several orders of magnitude more time.

It's an optimization of people's time problem. You could probably spend several hours (or days) writing a C program for a specific problem. But if you can spend only 40% of the time writing the program and have it only 20% slower, then that's a definite win (these numbers are just an example).

  $ cat sum.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      if (a[i] > max) {
        maxk = i
        max=a[i]
      }
    }
    print "max_key:", maxk, "sum:", max
  }

  $ time gawk -f sum.awk -O <ngrams.tsv
  max_key: 2006 sum: 22569013

  real    0m7.041s
  user    0m3.797s
  sys     0m3.156s

According to the gawk profiler, and strace -c (count) -- the awk program mainly spends its time reading the file (without the loop at the end, looking for the max value, the runtime is essentially the same).

In fact, on the surface, pypy and python3 are quite similar on the syscall/strace front - with roughly 23k "read" calls -- awk did 375k. And adding cat in front sped it up by about two seconds:

  $ time (cat ngrams.tsv |awk -f sum.awk )
  max_key: 2006 sum: 22569013

  real    0m3.969s
  user    0m3.719s
  sys     0m0.516s

  $ time awk -f sum.awk ngrams.tsv
  max_key: 2006 sum: 22569013

  real    0m6.465s
  user    0m3.609s
  sys     0m2.859s

    1#desc sum each group (!/) (" II";"\t") 0: `:tsvfile

EDIT:

    q)\ts d: (!/) (" II";"\t") 0: `:tsvfile
    2489 134218576
    q)\ts 1#desc sum each group d
    486 253055104

Certainly eye-catching but I wouldn't call it conclusive.