Hacker News Reader

Comment by jazza68 | Hacker News Reader

q is the programming language underpinning kdb+ by Kx Systems - often claimed (by Kx) to be "the fastest database in the world". Possibly also the most expensive? L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative.

View on HN

Absolutely and given Kx is now private equity owned and are in the customer-squeezing phase of the acquisition, this is very welcome indeed.

and fusion (e.g. f g h x has no intermediates e.g. mutates in place) is new as is compute on compressed vectors (very helpful performance unlock) https://lv1.sh/blog/compute-on-compressed/

yeah, that is super neat! Very innovative to bring it to q.

[1] seems to have the actual details on compressed vectors. looks like for each page you're storing min&max, and the values come as offsets from the min (smaller->less bits per value), did I get that right?

  primitive speedup  
  sum price 7.7x  
  avg price 33x

how does this even happen though?

[1]https://lv1.sh/blog/compression-memory-wall/

simple example at https://lv1.sh/blog/compute-on-compressed/ But in general compression is reducing the bit width of the input through an encoder (FOR or Frame of Reference is an old and good example). So we store the base in an offset location then the large payload is a much smaller size. E.g. i64 can goto i16. Then simd gets more #'s per cycle on the smaller, and the base is added to the scratch in stack (for sum). avg is similar (since it is just sum / count)

there's even a secret -17!`name that will show the details. e.g.

  //100k random 32b ints ... 
  l>v:100000?255
  l>v
  196 124 18 216 63 169 151 173 126 99 90 133 92 158 217 169 201 191 138 105 13..
  // but actually they are 1/4 the size e.g. int8
  l>-17!`v
  1b        // is compressed?
  100032j   // compressed bytes
  400000j   // original bytes

Reply on news.ycombinator.com