Hacker News Reader

Comment by RodgerTheGreat | Hacker News Reader

One of the compelling features of kdb+/Q that isn't explicitly called out here is vertical integration: it's a single piece of technology that can handle the use-cases of a whole stack of other off-the-shelf technologies you'd otherwise need to select and glue together. The Q language, data serialization primitives, and IPC capabilities allow a skilled programmer to tailor-build exactly the system you need in one language, often in a codebase that would fit on a few sheets of paper instead of a few hundred or thousand.

If your organization has already committed to serving some of these roles with other pieces of software, protocols, or formats, the benefits of vertical integration- both in development workflow and overall performance- are diminished. When kdb+ itself is both proprietary and expensive it is understandably difficult to justify a total commitment to it for new projects. It's a real shame, because the tech itself is a jewel.

View on HN

I agree that the vertical integration capability of kdb+/Q is amazing, and it is beyond comprehension why Kx themselves don’t effectively leverage it. Kx Platform appears to be mostly written in Java, and the API’s callable from Q are not very well documented. My team and I find the dashboards product is difficult to use, and there are some nasty bugs that cause frequent editor crashes for dashboards of moderate complexity. Q is so feature rich that it would be a blast to write web applications in, but instead we’re forced to use this drag and drop editor if we want to make something available to our users.

I think Shakti could become a viable competitor to Kx if they included libraries that handle some common enterprise usecases, such as load balancing, user permissions and SSO. I have no doubt that an experienced K programmer could whip this up in a week or two, but in my experience a sufficiently large enterprise will specify that all these capabilities need to be implemented before they let the product in the door.

I'm a little too close to be throwing stones, but without going into specifics I believe that key leaders at Kx do not properly appreciate the unique characteristics and benefits of their own technology, and are trapped in a mindset of trying to make their products more similar to their competition in order to make sales and marketing easier. In the process, they discard their competitive advantage. Tale as old as time.

I think it is very difficult to judge how much of an advantage your competitive advantage actually is. It’s very easy to look at the things which directly cost you sales and conclude that those are the things you need to fix rather than doubling down on your strengths. The most common way to avoid that is to go too far in the other direction and become convinced that your niche technology is vastly superior to the mainstream choice and anyone who rejects you for your shortcomings is just shortsighted and wrong.

From the outside it’s always seemed that kdb fans tend to land in the second camp, and I think it would be understandable for Kx to have overcorrected into undervaluing their work instead.

I agree that being able to write one piece of code that solves your use case is a big benefit over having to cobble together a message queue, stream processor, database, query engine, etc.

We've been playing around with the idea of a building such an integration layer in SQL on top of open-source technologies like Kafka, Flink, Postgres, and Iceberg with some syntactic sugar to make timeseries processing nicer in SQL: https://github.com/DataSQRL/sqrl/

The idea is to give you the power of kdb+ with open-source technologies and SQL in an integrated package by transpiling SQL, building the computational DAG, and then running an cost-based optimizer to "cut" the DAG to the underlying data technologies.

Reply on news.ycombinator.com