jq performance is pretty terrible. Here I'm going to do something super simple like pull out a single field out of a large log file:
$ wc -l big.log
979400 big.log
$ du -hs big.log
570M big.log
`count` is a small program that counts lines on stdin. like `sort|uniq -c |sort -n`
jq takes 12 seconds:
$ time cat big.log |jq -cr .method |~/bin/count
848000 GET
94800 POST
34000 HEAD
2400 OPTIONS
200 null
real 0m12.381s
user 0m12.427s
sys 0m0.333s
my tool takes .5 seconds
$ time cat big.log |~/bin/jj method |~/bin/count
848000 GET
94800 POST
34000 HEAD
2400 OPTIONS
200
real 0m0.466s
user 0m0.512s
sys 0m0.198s
`jj` is a little tool I wrote that uses
https://github.com/buger/jsonparser