Hacker News Reader

Comment by paulasmuth | Hacker News Reader

Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use something other than glibc to compile these binaries?

Also did you roll the SQL engine from scratch or does it built on some existing parser/runtime? Would love to learn more.

View on HN

Linux linking: I wasn't doing anything special with the linking. The Linux binary was compiled on a VirtualBox instance of Mint and stripped and compressed, so it's possible your tools are misreporting, or I did something I wasn't expecting. At any rate, I haven't made that as a conscious design choice, and it's easy to change (fix?) compilation styles in a future build.

SQL engine: all from scratch

Awesome stuff, I am actually hacking on something similar, so I would love to hear your thoughts RE: supporting "windowed" aggregations. (E.g. a moving average over a timeseries).

Are you planning on adding that and which syntax are you planning to use? Have you looked at postgres "OVER PARTITION"? While it seems powerful it's also fairly unintuitive IMHO. I was experimenting with adding a GROUP BY clause that allows each input row to appear in more than one group in the result set. Something like:

    SELECT time, mean(value) FROM mymetric GROUP OVER TIMEWINDOW(time, 60);

You can do that like this: GROUP BY date_trunc('minute', time)

The snippet you posted will compute an aggregation based on a fixed time interval. I.e. it will put every row into a "bucket" of per-minute granularity and then compute an aggregate function for each of those buckets bucket, taking into account only the rows that ended up in that specific bucket (i.e. only rows from that specific minute). To put it another way this is asking the question "Please give me the aggregate of some value per minute".

To make my question more precise; I was trying to ask specifically about a "moving window aggregation" (e.g. a moving average over a timeseries). This is more like asking the question "Please give me every minute an aggregate based on all values in the last N minutes". To do that you need each input row to end up in more than one bucket (or have a special type of aggregation function like postgres does).

For example, if you were doing a moving aggregation with a 1-minute interval ("bucket size") and a 5 minute window ("lookback"), you would need to place each row into 5 buckets: The bucket into which it belongs based on it's timestamp and the 4 previous buckets. And a vanilla SQL GROUP BY can't do that.

Hope that makes sense.

The executable appears to be packed using vanilla UPX. Decompressing it and grepping the symbol table reveals the presence /lib/libc.so.6, /lib/libm.so.6 and /lib/libpthread.so.0.

By method of exclusion, it cannot be musl because musl unifies everything into a single /lib/libc.so.

It most likely is not uClibc because uClibc is typically symlinked to /lib/libc.so.0 from /lib/libuClibc.

Considering that the soname of /lib/libc.so.6 has been reserved by glibc since the 2.x days (to differentiate it from the Linux libc days of libc4 and libc5), it is most likely indeed glibc in use.

It's obvious this program isn't making use of NSS or anything like that, so it's doable to use glibc for static linking, if frustrating.

Ah turns out this binary actually isn't statically linked at all. It's just the UPX packer that you mentioned that adds a trampoline/stub (the decompression routine I suppose) around the actual binary [and that trampoline code is what's completely static]:

    ldd kerf-unpacked
    	    linux-vdso.so.1 =>  (0x00007fff5e7f1000)
	    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff1833aa000)
	    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff1830a4000)
	    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000)
	    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff182a99000)
	    /lib64/ld-linux-x86-64.so.2 (0x00007ff1835d1000)
	    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff182870000)

So using NSS shouldn't be an issue; and the binary actually does use it

      open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3

	    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000)

libreadline is GPL'ed, not LGPLed, Kerf is therefore in violation.

https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html

Edit: The GPL doesn't allow any linking into non-GPL compatible software, dynamic or static. Readline has this license for exactly this reason, and they have forced software into the public with it before (ncftp comes to mind)

http://www.gnu.org/licenses/gpl-faq.en.html#GPLStaticVsDynam...

Edit2: https://en.wikipedia.org/wiki/GNU_Readline#Implications_of_G...

It is not statically linked but loaded dynamically at runtime (see my comment above). And I think the .so is not bundled by UPX/shipped with the binary either. I believe this usecase is explicitly allowed by the GPL, but IANAL, so happy to be corrected.

EDIT: Looks like I stand corrected and this is NOT the case even if you are just dynamically linking a GPLed .so; check parent's links/ask a proper lawyer/please don't sue me ;)

Even though its dynamically linked the GPL still applies. The LGPL is designed to allow you to link to the software without requiring you to use the GPL license. Also since you have made your binary available you are required to make the source code you have used to make this release publicly available.

I don't think anyone cares enough to sue you, however you are clearly in violation.

I am not the author of kerf/the linked software ;)

The OSX version links against libedit. The Linux version is supposed to as well so if it isn't that's an oversight. Let me take a look.

I've committed a Linux version that for sure uses libedit. It is decidedly not my intention to use libreadline.

Right, the reference to ld-linux(8) should have been a giveaway.

Reply on news.ycombinator.com