Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android | Hacker News Reader

Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

github.com

62 pointsluu30 comments

Do I understand correctly that this is proprietary (closed-source) software, even though it's hosted on GitHub? This repository seems to contain two binary blobs without source code.

I don't get all the negativity RE: kerf not being open source in this thread. It's obvious the implementation is not free software (check the README in the actual submission) because the author wants to sell commercial licenses. Nothing inherently bad about that -- some of us need to make a living.

The query language and concepts which are explained in the documentation are an exciting/novel (at least to me) contribution and "out in the open" for everybody to see/discuss/adapt/improve upon. Let's not dismiss that because we can't read the source.

Strangely, there was no negatively in the comment you replied to, they were simply asking if is is open-source or not, as it didn't align with their expectations of github-hosted projects (I would argue this would be the case for most people).

Contrary to what you state, it is _not_ clear from the README that this is a commercial venture, just that the author provides an email address for any queries in relation to licensing, and actually includes _no_ copyright or restrictions on running the code in the provided documentation. One might even assume that since we've been provided a copy of the binary without any notices, we might be free to run said code without restrictions... but really, who knows? Hence the OP's questions.

https://en.wikipedia.org/wiki/Talk:GitHub#Licenses_of_Projec...

Not a very helpful link.

Yes, just because it is hosted on github does not mean it is required to be open-source, but the documentation doesn't include any information about the licensing terms, so it is a valid question (that is as yet unanswered in public).

Nothing bad ever happens to proprietary database software... nope.

Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use something other than glibc to compile these binaries?

Also did you roll the SQL engine from scratch or does it built on some existing parser/runtime? Would love to learn more.

Linux linking: I wasn't doing anything special with the linking. The Linux binary was compiled on a VirtualBox instance of Mint and stripped and compressed, so it's possible your tools are misreporting, or I did something I wasn't expecting. At any rate, I haven't made that as a conscious design choice, and it's easy to change (fix?) compilation styles in a future build.

SQL engine: all from scratch

Awesome stuff, I am actually hacking on something similar, so I would love to hear your thoughts RE: supporting "windowed" aggregations. (E.g. a moving average over a timeseries).

Are you planning on adding that and which syntax are you planning to use? Have you looked at postgres "OVER PARTITION"? While it seems powerful it's also fairly unintuitive IMHO. I was experimenting with adding a GROUP BY clause that allows each input row to appear in more than one group in the result set. Something like:

    SELECT time, mean(value) FROM mymetric GROUP OVER TIMEWINDOW(time, 60);

You can do that like this: GROUP BY date_trunc('minute', time)

The snippet you posted will compute an aggregation based on a fixed time interval. I.e. it will put every row into a "bucket" of per-minute granularity and then compute an aggregate function for each of those buckets bucket, taking into account only the rows that ended up in that specific bucket (i.e. only rows from that specific minute). To put it another way this is asking the question "Please give me the aggregate of some value per minute".

To make my question more precise; I was trying to ask specifically about a "moving window aggregation" (e.g. a moving average over a timeseries). This is more like asking the question "Please give me every minute an aggregate based on all values in the last N minutes". To do that you need each input row to end up in more than one bucket (or have a special type of aggregation function like postgres does).

For example, if you were doing a moving aggregation with a 1-minute interval ("bucket size") and a 5 minute window ("lookback"), you would need to place each row into 5 buckets: The bucket into which it belongs based on it's timestamp and the 4 previous buckets. And a vanilla SQL GROUP BY can't do that.

Hope that makes sense.

The executable appears to be packed using vanilla UPX. Decompressing it and grepping the symbol table reveals the presence /lib/libc.so.6, /lib/libm.so.6 and /lib/libpthread.so.0.

By method of exclusion, it cannot be musl because musl unifies everything into a single /lib/libc.so.

It most likely is not uClibc because uClibc is typically symlinked to /lib/libc.so.0 from /lib/libuClibc.

Considering that the soname of /lib/libc.so.6 has been reserved by glibc since the 2.x days (to differentiate it from the Linux libc days of libc4 and libc5), it is most likely indeed glibc in use.

It's obvious this program isn't making use of NSS or anything like that, so it's doable to use glibc for static linking, if frustrating.

Ah turns out this binary actually isn't statically linked at all. It's just the UPX packer that you mentioned that adds a trampoline/stub (the decompression routine I suppose) around the actual binary [and that trampoline code is what's completely static]:

    ldd kerf-unpacked
    	    linux-vdso.so.1 =>  (0x00007fff5e7f1000)
	    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff1833aa000)
	    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff1830a4000)
	    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000)
	    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff182a99000)
	    /lib64/ld-linux-x86-64.so.2 (0x00007ff1835d1000)
	    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff182870000)

So using NSS shouldn't be an issue; and the binary actually does use it

      open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3

	    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000)

libreadline is GPL'ed, not LGPLed, Kerf is therefore in violation.

https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html

Edit: The GPL doesn't allow any linking into non-GPL compatible software, dynamic or static. Readline has this license for exactly this reason, and they have forced software into the public with it before (ncftp comes to mind)

http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynam...

Edit2: https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_G...

It is not statically linked but loaded dynamically at runtime (see my comment above). And I think the .so is not bundled by UPX/shipped with the binary either. I believe this usecase is explicitly allowed by the GPL, but IANAL, so happy to be corrected.

EDIT: Looks like I stand corrected and this is NOT the case even if you are just dynamically linking a GPLed .so; check parent's links/ask a proper lawyer/please don't sue me ;)

Even though its dynamically linked the GPL still applies. The LGPL is designed to allow you to link to the software without requiring you to use the GPL license. Also since you have made your binary available you are required to make the source code you have used to make this release publicly available.

I don't think anyone cares enough to sue you, however you are clearly in violation.

I am not the author of kerf/the linked software ;)

The OSX version links against libedit. The Linux version is supposed to as well so if it isn't that's an oversight. Let me take a look.

I've committed a Linux version that for sure uses libedit. It is decidedly not my intention to use libreadline.

Right, the reference to ld-linux(8) should have been a giveaway.

I am really interested in this as it seems like a more accessible version of KDB - especially in its support for SQL.

I'd like to look into it further, but I can't find any information about licensing. Given that there is no source code in the repo, it appears that this isn't an open source project.

I would generally consider it a bug to not have any form of license in the repo, but from the top of the Readme (last modified 10 days ago):

"Contact Kevin (e.g., licensing, feature/documentation requests): k.concerns@gmail.com"

So if you're interested, I suggest you send Kevin an email.

Personally I think it would've been interesting if it was Free Software -- as this doesn't come with any license, it's essentially less useful than the 32bit kdb[1] -- and I can't imagine it's as feature complete, or has a similar level of documentation, support or real-world testing.

If the intention is to make the binaries deployable/usable, a license file/note in the Readme would be the absolute minimum requirement IMNHO.

Also find it strange to host just binary artefacts on github -- I can't see how that's useful for the author or the users. I suppose one could submit pull-requests against the python api, but then it would make more sense to split the repo in two -- one for the source-available (unknown license) python code -- and host the binary artefacts somewhere.

[1] http://kx.com/software-download.php

What is tick?

I had no idea and googled this page https://www.mongodb.com/post/45116404296/how-banks-use-mongo...

"Tick databases are real-time database servers for capturing, managing, and processing market data."

They go into the requirements after that.

For the curious, in financial markets ticks are datapoints that represent every order (price, size, conditions etc.) placed (or modified) in the market for a particular security by a buyer or seller, as well as trades. In liquid equity markets across a large number of securities, this data can have a very high frequency and typical databases are not well suited for it's storage and retrieval.

More generally, tick databases are databases specialized for storing and querying high frequency time series data.

A somewhat similar project by Man Investments: https://github.com/manahl/arctic

As a data nerd, I got very excited once I read all the features, but I'm just assuming this will wither and die since it's closed-source =(

I see Linux and OS X, where are the iOS, Android and *BSD binaries?

No source code? I'm not even going to give this thing a glance. There is nothing so compelling in the database world that I would even be tempted to give up source access. What an anachronistic throwback this kind of thing is.

And it's packed using UPX too, presumably to "protect" the program somehow. Just say no to proprietary software from small vendors.

Reply on news.ycombinator.com