Looks pretty nice! The readme says it's not useful for repos with >2000 commits, but I wonder if we could hack together some sort of cache for it to facilitate delta updates?
Crunching diffs at scale is actually an extremely I/O intensive operation and I can say this from experience, since my product is doing just that. You can see an example of what I mean at
https://public.gitsense.com/insight/github?r=Microsoft/vscod...
If you scroll down, you can see how the indexed information is used. The red, green, and blue bars, denotes how many lines were deleted, added and changed. The red, green and blue bars in the second row, shows how many lines were deleted, added and changed, which were non blank/comment lines.
If you want to crunch diffs at scale, you really have to throw CPU cores, RAM and multiple SSDs at the problem. The more CPU cores, translates to more diffs that can be processed in parallel. The more RAM, translates to better Kernel page caching. And multiple SSDs in RAID 0 configuration, translates to better read and write speeds.
This is purely out of interest, our storage performs at around 2-4M random 4K read IOP/s per node and I was wondering (just for fun) how it might look thrown a good I/O crunch task like this.
It's actually pretty straight forward since you know exactly what operations has to be done. With very good I/O, you may find the CPU will become your bottleneck.