Hacker News Reader

paulasmuthon Tetris as a C++ Template Metaprogram

I am not sure if I should be amazed or scared by this. Guess it won't be long before somebody comes up with the only thing left to do, going full meta and writing a "c++ to c++ templates compiler" (or something along those lines).

Now if only we had a way to use all this goodness to reflect on structs/enumerate members...

On the actual topic: I wonder how much memory it takes to compile a single round of tetris?

paulasmuthon Dynamic software updating in C

Thanks for the clarification! I understand that you got rid of the "trampoline"/dispatch code for calling methods by rewriting the respective bytecode at runtime. But you still need to register file descriptors and such with some rubah runtime code, and that piece of rubah runtime code doesn't get replaced on process upgrade, right?

I guess this really doesn't matter from the user's point of view, just for the sake of curiosity, want to work out that/if there seems to be a fundamental difference (FWIW) between what rubah is doing which I understand as "hot patch the code within the running process and have some kind of 'trampoline'/dispatch code within that process to switch between new and old version, regardless if it is actual explicit code or just implicit in the rewritten bytecode, but never actually leave that original process" vs. what you'd do in native code on linux which is more like "tell the kernel to start a proper new process and then pass all open fd's and programm state to the new process" (which obviously requires that process to be written in a way that allows it to "continue where it left off") --- and the latter is not possible to do when running on the stock jvm if I understand correctly, because the jvm doesn't allow you to implement the part where you pass the filedescriptors, right?

paulasmuthon Show HN: Brandsketcher – Free startup name brainstorming service

Yeah, esp. love the "Would you possibly need a logo design later?" part of your questionnaire. That seems incredibly relevant to choosing a good name for a company. (Not intending to troll you though, hats off to a clever lead generation strategy!)

paulasmuthon Show HN: Brandsketcher – Free startup name brainstorming service

I am not convinced this isn't some kind of innovative lead generation method or an ingenious scheme to collect lots of juicy juicy business ideas. Could at least put your name on there so people know who they're dealing with ;)

paulasmuthon Dynamic software updating in C

Thanks for the very interesting article. One thing I didn't find very clear from the paper was the rationale of using a framework/kitsune/all that wrapper code vs just implementing the feature natively. In other words, what is the upside of using kitsune to make e.g. redis live-upgradable vs just implementing it in redis proper [I believe that e.g. redis already uses a fork model for writing the data to disk/getting a snapshot of the data]? Is it much simpler when I am using kitsune? Do I really not have to reason about what happens when the binary upgrades? Or is this mostly useful for retro-fitting the feature to software that wasn't designed with the usecase in mind?

paulasmuthon Dynamic software updating in C

To answer my own question here, but without being a java expert, so talking out of ignorance probably. This paper [1] argues there are only two methods to do this on the stock JVM. One is putting in some trampoline code, the other one is patching the JVM. So it looks like the stock JVM doesn't actually support restarting the program while keeping the old heap and FDs etc around.

[1] http://www.cs.umd.edu/~mwh/papers/rubah.pdf

paulasmuthon Dynamic software updating in C

Honest question; How can you even do this (replace the running binary/java code without closing file descriptors, etc) in plain java/on the jvm? [ Without resorting to tricks where there is some trampoline code that actually does the connection handling that never gets updated. I realize that's kind-of what the authors of the linked paper are doing too, but it's certainly possible to do this "the old fashioned way" without such hacks in native code on linux]

paulasmuthon PixQL: SQL for image processing

I love this! Now all we need is some higher level DRAW operations (draw a line from here to there, draw a piece of rendered text here) and we can glue PixQL and ChartSQL together to implement SQL query result visualization to bitmaps completely in SQL (without going through SVG).

PS: Under which license is the PixSQL code available?

paulasmuthon Why Don't Software Developers Use Static Analysis Tools to Find Bugs?

I think that was parent's point. Coming from other languages, it's kind of unexpected that a var defined within a curly brace block (if/for/etc) is actually scoped on the whole function in which it appears and not just that block. Hence it's "useless" to define it within the block/loop -- might as well hoist it outside, it's visible there anyway

paulasmuthon ThingieQuery: a SQL plugin for Excel

Any chance you could expand on some of those pains? Is the problem with simply sharing the data itself or with being able to slice/dice/query/do smth with the shared data? I'm sure there are a bunch of people around here who'd love to hear some user pains/stories to improve their apps.

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

I am not the author of kerf/the linked software ;)

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

The snippet you posted will compute an aggregation based on a fixed time interval. I.e. it will put every row into a "bucket" of per-minute granularity and then compute an aggregate function for each of those buckets bucket, taking into account only the rows that ended up in that specific bucket (i.e. only rows from that specific minute). To put it another way this is asking the question "Please give me the aggregate of some value per minute".

To make my question more precise; I was trying to ask specifically about a "moving window aggregation" (e.g. a moving average over a timeseries). This is more like asking the question "Please give me every minute an aggregate based on all values in the last N minutes". To do that you need each input row to end up in more than one bucket (or have a special type of aggregation function like postgres does).

For example, if you were doing a moving aggregation with a 1-minute interval ("bucket size") and a 5 minute window ("lookback"), you would need to place each row into 5 buckets: The bucket into which it belongs based on it's timestamp and the 4 previous buckets. And a vanilla SQL GROUP BY can't do that.

Hope that makes sense.

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

It is not statically linked but loaded dynamically at runtime (see my comment above). And I think the .so is not bundled by UPX/shipped with the binary either. I believe this usecase is explicitly allowed by the GPL, but IANAL, so happy to be corrected.

EDIT: Looks like I stand corrected and this is NOT the case even if you are just dynamically linking a GPLed .so; check parent's links/ask a proper lawyer/please don't sue me ;)

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

I don't get all the negativity RE: kerf not being open source in this thread. It's obvious the implementation is not free software (check the README in the actual submission) because the author wants to sell commercial licenses. Nothing inherently bad about that -- some of us need to make a living.

The query language and concepts which are explained in the documentation are an exciting/novel (at least to me) contribution and "out in the open" for everybody to see/discuss/adapt/improve upon. Let's not dismiss that because we can't read the source.

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

Awesome stuff, I am actually hacking on something similar, so I would love to hear your thoughts RE: supporting "windowed" aggregations. (E.g. a moving average over a timeseries).

Are you planning on adding that and which syntax are you planning to use? Have you looked at postgres "OVER PARTITION"? While it seems powerful it's also fairly unintuitive IMHO. I was experimenting with adding a GROUP BY clause that allows each input row to appear in more than one group in the result set. Something like:

    SELECT time, mean(value) FROM mymetric GROUP OVER TIMEWINDOW(time, 60);

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

Ah turns out this binary actually isn't statically linked at all. It's just the UPX packer that you mentioned that adds a trampoline/stub (the decompression routine I suppose) around the actual binary [and that trampoline code is what's completely static]:

    ldd kerf-unpacked
    	    linux-vdso.so.1 =>  (0x00007fff5e7f1000)
	    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff1833aa000)
	    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff1830a4000)
	    libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff182e5d000)
	    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff182a99000)
	    /lib64/ld-linux-x86-64.so.2 (0x00007ff1835d1000)
	    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff182870000)

So using NSS shouldn't be an issue; and the binary actually does use it

      open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3

paulasmuthon Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use something other than glibc to compile these binaries?

Also did you roll the SQL engine from scratch or does it built on some existing parser/runtime? Would love to learn more.

paulasmuthon MapD: Massive Throughput Database Queries with LLVM on GPUs

> But as far as I understood this is about what in classic db would be called a sequential scan

You misunderstood my comment then. The scheme I described is used to do "full table scans" on the data. Also this is not something I came up with; there is a ton of research on this and it's how a number of fairly well known DB/analytics products work.

> But on a single machine the problem is checking what's in your own memory fast enough using CPU. That's where GPUs come in.

You can already scan these data volumes in millisconds on traditional hardware without doing too much fancy stuff or using GPUs. My original point was that I have a hard time coming up with an "interactive analytics" usecase where you'd want to process TB/s on one node that can keep only a few gigs around -- the improvement of doing this on GPUs instead of the old fashioned way on general purpose hardware seems to be that a query over the same dataset returns a few milliseconds faster. I reckon a user running interactive sql queries doesn't really care if their queries return in 10, 50 or 100ms -- I am not even sure I would be able to notice that difference myself. However, if you actually had a usecase for this you could still use the approach I described and make it as fast as you desire by scaling it out [tweaking the shard size] on conventional/commodity machines.

> you can just throw more boxes at it. But on a single machine

The linked article is discussing a setup with at least 8 distinct processing units, too.

> Even apart from the name of your company, the idea seems very interesting. I suppose I'm not the only one curious, so could just say a few more words? Are you working on some new technology

We are focused on delivering actionable insights as well as solving some very specific data problems for our customers right now. IMHO the hard part of doing that is making sure we understand the customer's domain, gather/track the right data from their systems and then work with them to slice and dice and visualize this data to discover "signal" from the "noise" which we can then feed back and use to optimize their website/app. The technical part of being able to handle the relatively large volumes of (non-preaggregated) source data is really just the means to that end. We do not currently offer an "off-the-shelf" version of our product, but again if you or somebody else would like to talk more, please ping me.

Also I hope I not coming across too negatively here. The linked research (compiling SQL to LLVM IR) sure is exciting stuff. I just couldn't help but feel that the hyperbole PR speak was a bit too strong with this one.

paulasmuthon MapD: Massive Throughput Database Queries with LLVM on GPUs

I believe these guys are actually going straight from SQL to "GPU instruction code" (so AIUI the SQL query execution is not some hack that is layered on top of the programmable shader pipeline [which is what's responsible for the "here is a list of vertices now draw me a 3d scene part" you mentioned] or something like that but runs directly on the more or less general purpose processor in the graphics card)

I understand the part where they "map the result set to opengl primitives" as that they write the output/result set of the sql query into some datastructure that allows them [after executing the query] to easily use it from opengl to render pretty stuff. This part shouldn't be much different from other texture/vertex data that you'd bring in e.g. from files on disk or a MySQL database into opengl specific buffers so that you can subsequently use it for drawing in your program.

paulasmuthon MapD: Massive Throughput Database Queries with LLVM on GPUs

I actually recently left my job at google to start my own company and that's what we do. We didn't consider using GPUs yet as the queries we are running for the specific usecase we are currently working on (web/ecommerce analytics) are all IO bound [so the first step if we wanted to speed stuff up would be moving the data onto ssds or into memory as we are currently storing everything on disk. however this hasn't been necessary so far]. Our product allows you to run interactive queries (segment/aggregate/etc) against a billion record dataset (in the terrabytes; basically a giant logfile that stores an entry for every interaction we observe on the website/app with lots of metadata) with O(seconds) latency.

We wrote our own code that does "the heavy lifting" but it's pretty much the same that everbody seems to be doing right now and comes down to minimizing the # of bytes read at query time; we index all data into a columnar format (so we only need to load a subset of the data and can pack it tightly for most queries) and then slice the query up into lots of individual shards that we compute in parallel (again, being bound mainly by the 200megs/sec disk read we get out of each machine).

Without being completely serious now but regarding your question; "a simple WHERE" (scanning each row and applying a predicate fn with no shared state/shuffle/merge phase) is pretty straightforward to run on a huge dataset as you can split the execution up into as many pieces as you want and run them individually. Since this allows you to basically choose the problem size per shard you could scan those 750m rows on a ginormous cluster of tamagochis if you can fit a single row onto one ;)

If you or somebody else would like to talk more, please shoot me an email: paul@deepcortex.io

paulasmuthon MapD: Massive Throughput Database Queries with LLVM on GPUs

Are problems that fit in memory on a single machine (96GB in this article) considered "big data" now? This buzzword is becoming absolutely meaningless. But yeah, "Hyper-interactive visualytics at scale.", go nvidia PR.

Also, while it's absolutely amazing that they are able to scan 240B rows/sec, I wonder what one would use this capability for if they can only keep a few hundred million records around at a time? The difference between taking 10 or 100ms to scan the dataset should hardly matter to a user that is running "interactive analytics queries".

paulasmuthon Introducing the Fan – simpler container networking

I don't seem to get it. How is this different from just using a non-routed IP per container?

paulasmuthon Recommending items to more than a billion people

Have a look at Google's MinHash algorithm. While it's a probabilistic solution, You can run it as a mapreduce and will at no point need to have the full data set in memory/on a single machine. So it does scale pretty well.

http://www2007.org/papers/paper570.pdf

EDIT: I see you changed your comment to include "no full graph recalcuation". Incremental recos are possible to do with minhash but I think you can't solve decay of old data easily.

paulasmuthon Show HN: My first tiny web service, a simple key/value store

How do you determine if I am flooding your service with random data?

My understanding is that one of your main ideas is to encrypt the data so I am honestly wondering where you would even start to check if a user was randomly flooding you or legitimately storing lots of keys.

Not trying to pick nits here... My original point was that -- in my opinion -- it comes across slightly dishonest if you insist that there is no limit even when it's obvious there must be one. (I mean seriously, how many machines is this running on right now? Even if it's running on amazon/google you can only scale my backend up so much with 20$/mo)

paulasmuthon Show HN: My first tiny web service, a simple key/value store

I am not sure I understand the pricing and/or the service. What are your guarantees about dropping or not dropping keys? Obviously you can't sell unlimited memcache space for 20$/mo and you must protect against customers exhausting all your available resources.

So since it's quite obvious that there must be some limit and/or expiration policy I think you should really just spell it out on the page.

paulasmuthon Prometheus: An open-source service monitoring system and time series database

Shameless plug: This looks quite similar to FnordMetric, which also supports labels/multi dimensional time series, is StatsD wire compatible and supports SQL as a query language (so you won't have to learn yet another DSL)

paulasmuthon Introducing Atlas: Netflix's Primary Telemetry Platform

Sounds similar to FnordMetric (http://fnordmetric.io/chartsql) which also supports dimensional timeseries data. Major differences between Atlas and FnordMetric on first sight:

- SQL based query and charting frontend (ChartSQL), so you don't have to learn yet another DSL

- ships with a a wire compatible StatsD API

- supports pluggable backends

- renders charts to SVG

- will probably not scale to petabytes of data out-of-the box

- single c++ binary, deploy it in 5 minutes using docker

- includes an interactive query editor

paulasmuthon Graph everything with Graphite

You might also be interested FnordMetric ChartSQL (http://fnordmetric.io) which is a SQL-based graphite competitor.

paulasmuthon ChartSQL: Create Charts and Dashboards from SQL

You didn't mistunderstand the purpose (However it's not an explicit goal of FnordMetric to substitute chart.io)

Here is an example that connects to a MySQL database on localhost: https://github.com/paulasmuth/fnordmetric/blob/master/fnordm...

Note that the IMPORT statement might be a bit misleading. The IMPORT statement only creates a "virtual table", it doesn't actually copy any data. As much of the query as possible is pushed down into MySQL/the external data source and the charts will be generated from the query result that is returned by the external data source.

paulasmuthon ChartSQL: Create Charts and Dashboards from SQL

It's actually fairly straightforward, you can check out the code here: https://github.com/paulasmuth/fnordmetric/tree/master/fnordm...