>converting data such that it is represented in ones and zeros
er, what? Isn't it all?
>This dramatically reduces the size
huh? There is no symbolic encoding less efficient for length than binary.
er, what? Isn't it all?
>This dramatically reduces the size
huh? There is no symbolic encoding less efficient for length than binary.
As a concrete example, we took the NYC taxi ride data set which is something like 300GB of CSV files and when it was indexed in Pilosa, the total size of all the bitmap files was closer to 40GB.
What's not obvious is what we're associating with what in the NYC taxi ride data set.
A bitmap can also represent a set: the bit positions denote enumerated element symbols, and the value indicates whether that element is present.
So we rearrange the NYC taxi ride data into a data structure based on graphs and sets, and make large bitmaps?
I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.
> The central component of Pilosa’s data model is a boolean matrix. Each cell in the matrix is a single bit; if the bit is set, it indicates that a relationship exists between that particular row and column.
But why do I want my integers sliced apart into these associations?
Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.
E.g., turning something like a JSON string of the same data into bits and sticking it in Pilosa.
But really, you use one bitmap for each binary bit of an integer, and it turns out you can generate arbitrary range queries on your dataset by doing various combinations of boolean operations on those bitmaps.
> Pilosa is built around the concept of representing everything as bitmaps.
What this literally means is that not a single datum is stored that is not in the bitmap.
But then the diagrammed examples look like this:
manatee loris sea_horse
Vertebrate 1 1 1
Invertebrate 0 0 0
Breathes Air 1 1 0
If those strings are removed, the bitmap means jack squat.
Are they understood to be in another bitmap?
Sometimes this is a function which might map (for example) row 3 to the letter 'd', 4 to 'e', and so on. Sometimes it has to be a lookup table which can be kept within Pilosa, or externally. Sometimes the IDs map directly to what they represent (day-of-month, year, passenger count, etc.)
So strictly speaking, not everything is a bitmap, but the bulk of the heavy lifting in terms of serving queries is computation on bitmaps.