antonmks

Doesn't look like it uses any kind of vectorized execution. So the only advantage one gets is reduced amount of data to process. It also doesn't mention execution on compressed data so it has to decompress the data which doesn't come cheap.

Is it possible to generate the dataset that you used ? I would like to run a benchmark for myself and downloading a 1 TB file from Amazon unfortunately is not an option.

More videos : government building takeover in Vinnitsa, Ukraine : http://www.youtube.com/watch?v=vi_B637FCDY

Disk time matters only when you read your data first time. The following queries won't have to read from the disk. The compressed data may sit in memory for days and the queries won't touch the disk. Now, if your compressed data doesn't fit into memory, then disk speeds of course would matter a lot, on this I agree with you.

The disk time was subtracted because it is in-memory system and when your compressed datasets fit into memory your disk subsystem hopefully will be irrelevant.

Comparing against Impala doesn't change anything, Impala is still well behind even on a cluster with just 600 million records : https://docs.google.com/spreadsheet/ccc?key=0AgQ09vI0R_wIdEV...

Your other considerations are still valid though. Although the point was to show the inefficiency of Hadoop/MapReduce when it comes to relational operations.

Big advantage of Hadoop is that you do not need to pay license fees which in case of relational databases can reach into tens of thousands dollars.

The big disadvantage is that Hadoop is two orders of magnitude slower than relational databases. Also, Hadoop clusters are not what one would call a "green" solution. More like a terrible waste of computing resources.

Let's see the reasons why the author of nginx would not be accepted into ycombinator :

Strong Russian accent : check

Over 40 years old : check

No revenue stream : check

This is exactly why I wear "do not resuscitate" bracelet. I prefer to go peacefully, without having to spend a few extra years brain-dead in a coma.

For GTX 680 the double precision is actually 1/24 of the single precision so it is closer to 4%. Pretty sad when comparing the performance to similar priced Radeon cards.

Here the details : Integers and decimals are both stored as 8 bytes. So one record takes 42 bytes. After compression it takes about 8 bytes ( I use FOR - frame of reference) compression and it works really well with decimal numbers (check the range of decimal numbers in lineitem.tbl file - usually you encode them in just a few bits). So I can stream about 10 million records per second from my disk. Which makes it exactly 180 seconds for 1.8B rows ( CUDA calculations are done in the background process). Records are decompressed in GPU and processed there. Actually the main bottleneck is not the disk access but the latency of CUDA calls - each call takes about 15 ms and when you process the entire file in chunks of ,say, 6 million records, it really adds up.

By column. FOR(frame of reference) compression for integers and decimals, FOR-DELTA for already sorted columns, dictionary for strings.

It is 6 GB for Tesla GPU. PCI Express transfer speed is 5GB/s, with average compression rate averaging 4/1 the speed becomes 20GB/s so transfer speed is not a bottleneck anymore.

Thank you ! The data are stored on a disk and only the columns necessary for transformations are brought to GPU. In case the data does not fit into a limited GPU memory, it is processed piece by piece. I tested it on terabyte sized files so the data does not have to fit into GPU memory .