Hacker News Reader

Comment by gd1 | Hacker News Reader

It is very sad that they won't open up, and very short-sighted as well. I'm sure Whitney and KX are perfectly happy with their millions, but they are sitting on one of the most impressive software achievements around and refusing to share it. It's not even like it would cost them. First Derivatives could grow a consulting empire. I mean, look at open-source MongoDB. It is a piece of shit, and it is valued at $1.4B. KDB is finest thing I have ever used, and First Derivatives purchased a controlling stake in KX for £36.0M. It makes no sense.

Maybe kOS will be open-source. Maybe it will be the big one, Arthur's chance to change the course of software development history.

View on HN

It is really sad indeed. But I suggest you take a look at the J language. It's free for commercial use, unless you want to integrate the JDB, the equivalent of KDB+ column store. It also provides a rich feature set:

http://code.jsoftware.com/wiki/JDB/Announcement

http://www.jsoftware.com/jdhelp/overview.html

https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-... comes very close to capturing the ingenuity of J language.

After going through the J primer, KDB+/Q feels like a distillation of J - it basically took the most accessible ideas of J and packaged them for mainstream production use - some of their features such as these make it easy to setup a computing harness for electronic trading in no time:

http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles

http://code.kx.com/wiki/Startingkdbplus/tick

Also, the footprint of the kdb runtime and installation is unparalleled. I've run systems 24 x 7 x 365

I am aware of J, but it seems to me that J is comparable to K, not Q/KDB+. If you want a python comparison, then K is python, and Q/KDB+ is more like pandas. Except much, much quicker, and more powerful, but without as many tools and packages to support it. But that could be fixed - those tools and packages don't exist because the community isn't there, because it is (sadly) closed source.

J versus K is like C versus C++, similar syntax but different programming style. J is array-based and K is list-based. J is mostly just APL converted to ASCII representation. Q is mostly just some light syntactic sugar over K.

"they are sitting on one of the most impressive software achievements around and refusing to share it"

they tried. About a year ago they released 32 bit version for free with shockingly liberal license - anything goes, including commercial use, unlimited. About couple of months ago they pulled it back and replaced with free non-commercial use only. well, they are the owners, it is their call, but still k will remain as "one of the most impressive software achievements around", orthogonal to most of the [fucked up] IT trends of the past 15 years. It is a shame if it fades into a quantitative elitist oblivion.

If you got a free 32-bit version while while the terms were "shockingly liberal" then maybe those are the terms that govern the use of that binary? I don't know for sure.

I do know in addition to changing the license they have made changes to the software since that time. The size of the binary has increased.

I worry more about Kx being acquired by some large company, maybe a competing database vendor, that cares little about software quality.

yes, if you got the binary during that "liberal" period, you are free to use it under "liberal" terms.

KDB with a 2GB addressable RAM limit is almost pointless. It was good that they were at least trying to encourage new developers, but you could never build anything serious with that limitation.

Ask the commenter one down from me, 'wsfull, he knows what I'm talking about!

This is the classic "innovators dilemma". My 2-cents: Kx has a good business selling kdb+ licenses to the big banks and trading firms. They've been exploring how to expand beyond this niche, but many big data/analytics startups go straight for the FOSS solutions. In addition to the cost/licensing advantages of FOSS, a large user-base and ecosystem has developed around things like mongoDB, giving them compelling business advantages over kdb+ (even with kdb+ performance advantages).

Although mongoDB is a demonstration of how to build a successful business around open-source, it's very different to start out that way rather than move that way after you're already established as closed/licensed product. Kx would have to essentially kill-off their existing profit stream with the goal that they could build a larger business around FOSS kdb+. That's a risky proposition, fraught with many pitfalls.

It is a shame they backed-off of the fully-open 32-bit version. At least that had some potential to spur some user-base/ecosystem growth (at a minimum, this could have encouraged the development of novel clients/editors/REPLs/debuggers/charting/etc...), without threatening their core profit stream.

It's not obvious to me that an open core business can sustain the necessary margins to be interesting as an engineering company rather than a glorified services business. There are been very few examples (friend of mine argues it's just RedHat).

You are correct though that their model over the years has been to extract a large amount, up front, from a small number of users. They (mostly FD) have failed to make the leap to users outside finance due to what I'd call cultural reasons. They also lack strong technical leadership imnsho (they're, at heart, not an engineering company).

I'd certainly take a swing at doing an open source version for them but not clear to me that they'd know how to play it.

q/kdb+ is _enormously_ fast (or it was when I used it), but that was a result of 1) having a specific subset of problems to which the binary was highly tuned, 2) by Arthur Whitney who, in my mind, is one of those 100x engineers who was the key reason behind why it was so good as evidenced by 3) he's no longer with them and as such you'll notice an appreciable decline in code quality. I'm sure if AW was still working on the code-base, you'd be seeing a lot more AVX512 SIMD usage and clever things like that. (Take IDA to your q binary, it's...acceptable but nothing magical like AW's work. In fact, the lack of attention to detail is so significant the new engineers allowed even a poor rev. eng.[1] like myself to use a very very standard method to get a symbol table fully populated, confirming my suspicion that there's very little platform specific code).

RE: 36MM - that makes sense. q/kdb is the definition of "low volume, high margins". You only have so many IB's in Midtown, wealth management funds in Stamford, and a PIMCO here and there in Newport to shop your product to. (As opposed to say, Oracle, where there's broad appeal, residual government contracts as a result of legacy PL/SQL code from 20 year old code that's kept in production.)

[1] E-mails in the profile if you want to hear how I got a symbol table in, but I'm sure you already guessed by now.

Have there been any new reports of kOS? It was supposed to be "impending" in 2014...

http://archive.vector.org.uk/art10501320

A lot of the decisions that some might quibble with are not actually Arthur's end of things. The former CEO drove a lot of it.

Reply on news.ycombinator.com