Hacker News Reader

Comment by gravypod | Hacker News Reader

Kdb is a very niche product that has open source alternatives that are made use of by the rest of the software industry outside of FinTech. It's not bad, it's just not what the market wants to use because it's a closed box.

Wikipedia has an article on Time series databases and they contain a short list of popular TSDBs [0]. On that list 11 are libre-software to some degree and 4 are commercial. Relational TSDBs are new to the open source space but they're still being built despite there being the existing kdb+. This shows that people will spend considerably more funds to engineer a replacement (and give that replacement away for free) to avoid using a closed source product for this use case.

[0] - https://en.wikipedia.org/wiki/Time_series_database#List_of_t...

View on HN

Unfortunately, there are no open source alternatives to kdb+. Effortlessly handling hundreds of terabytes of data on a single machine, while consisting of only a 300 kB (yes, kilobytes)... Nobody even came close.

Market adoption could be better, but the license costs around $100,000/year (probably the most expensive software per kilobyte). Fintech can afford it, other industries can’t.

You'll also end up spending at least that much on consultants. While KDB is remarkable in many ways -- the first time I saw it I thought the demo was fake, the performance so great -- it is very difficult to develop in.

They have a an SQLish interface, that allows non-specialists to get work done, but anything serious needs to move beyond that.

I haven't used it in a few years, but queries had to be carefully optimized -- swapping the order in a where clause could cause order of magnitude differences in performance. Also it is append-optimized. If you need to update or insert data, it is a nontrivial exercise.

Regarding being append-only — these days, where reactive programming, immutability and FP are all in vogue, this is a feature. As for the rest, yes — K/Q are idiosyncratic, but they are simpler than it looks at first sight (and second.. and fifth). :)

kdb is frequently described as an in-memory database, or with great emphasis on how efficiently it utilizes a ton of RAM. I've no idea how much does its performance actually drop when the amount of data goes past, say, 2x the amount of available memory, but I'm pretty sure hardly anyone can afford multiple terabytes of RAM in addition to the software license these days…

Now there is shakti, https://anaconda.org/shaktidb

This seems pretty much next generation k.

We can see how things goes.

> Effortlessly handling hundreds of terabytes of data on a single machine

Single machine, because it can't handle parallel queries, at all, and there are no options to scale it.

Every financial institution I've worked at is busily unwinding their investment in q/kdb. It's legacy tech.

What are they moving to?

Other TS products like OneTick, RDBMSs for TAQ (esp. if you need non-TS indexes), streambase for UI backends

I even met a programmer who hacked something together in Java using the same columnar kdb layout but, you know, multithreaded, so different basket optimization jobs could run simultaneously on the same store.

Reply on news.ycombinator.com