Hacker News Reader

It is 6 GB for Tesla GPU. PCI Express transfer speed is 5GB/s, with average compression rate averaging 4/1 the speed becomes 20GB/s so transfer speed is not a bottleneck anymore.

View on HN

I see. So for a large dataset you would still hit a bottleneck, compared to a main mem store which also uses compression. But nonetheless it's a good idea to do certain operations on hot data on the GPU.

Which operations benefit the largest in your experience? I would guess scans with simple predicates and expressions. What about aggregations?

Except that hard drives have a much lower transfer rate?

But we're already working on the premise that data is in the main memory, thus the bottleneck being the GPU<->memory transfer rate.

I had investigated aggregates using gpu for building dwarf petacubes (http://www.cs.umd.edu/~nick/projects/Dwarf.pdf), and yes it looks viable.

I had not heard of that data structure before, thanks!

Sadly, it appears to be patented and thus of no use to me. http://www.cs.umd.edu/~nick/projects/Dwarf.html I think it's really quite crass and mercantile for UMD to patent basic computer science algorithms and data structures like that.

Do you compress per row/column or over a given set?

By column. FOR(frame of reference) compression for integers and decimals, FOR-DELTA for already sorted columns, dictionary for strings.

Reply on news.ycombinator.com