> To do this, you generally want your data in column-major format
I'd argue that the basic element of linear algebra is matrix vector multiplication, which I figured was best done row-major. Column major is great in other data use cases, but 'linear-algebra-like, therefore column major' doesn't feel right.
* Dictionary encoding: US,US,US,US,FR -> US:0,FR:1;0,0,0,0,1
* Run-length encoding: 0,0,0,0,1 -> 4x0,1x1
* Delta encoding: 0,1,2,3,4 -> 5x'+1'
* Storing the min and max for a chunk
Basically: exploit the data type to compress it.
Which enables very fast filtering and projections. (And now that the IO bottleneck has been managed you can do your gigantic logistic regression)
But it's also possible to think of it as "Grab one element of the vector, use it to scale the corresponding col of the matrix, and repeat, summing results." Both are efficient means of finding the result, and both have block-level versions that play nicely with the machine cache.
Meanwhile, linear algebra also often involves finding vector norms, and scaling vectors, and so on, and the way we usually set up tables means that the vectors of interest are generally columns of the data tables.
As others have mentioned, for some operations it can also save you from loading whole columns that aren't relevant for your transformation. The compression point in the sibling comment is definitely also relevant, especially for serialization. A whole lot of reasons to use column vectors.
Using "column-major" here might've been terminology abuse; sorry for the confusion.
if you're doing a linear algebra like transformation, you want to do it on all the prices or all the quantities, and a linear algebra library expects a big array of numbers, which is why you have to transform your records into an array of prices and an array of quantities.
"column" here refers to properties of objects, and not rows vs columns with in an array of number