Norikra: Schema-less Stream Processing with SQL | Hacker News Reader

Norikra: Schema-less Stream Processing with SQL

norikra.github.io

27 pointsmatsuu6 comments

This looks like an interesting, simpler alternative to stream-processing solutions like Spark or Storm, which require quite a lot of upfront investment in terms of infrastructure and learning.

I'd be interested to know how this works behind the scenes. Does it recalculate answers every time you run a query, or does it update the query answers as the events flow in?

Also, the examples all seem to be using a time window, like this one from the documentation:

     SELECT COUNT(DISTINCT cookie.id) AS uu,
            COUNT(*) AS imps
     FROM impressions.win:time_batch(1 hours)
     WHERE cookie.valid

So does this only answer questions of the type "how many unique users did we have in the last hour"? Or can you also calculate a total count: "how many users have visited this page in total?".

Norikra's queries are once registered, and then, outputs answers automatically (until it is removed).

The query above puts "how many unique users in an hour" per an hour. Of course, we can write queries like: * how many unique users in the last hour per every input event * how many unique users in the last 1000 inputs * and many more way to calculate metrics

Norikra is often combined with Fluentd log collector.

http://fluentd.org/

Fluentd document has the article for Fluentd and Norikra use-case :)

http://docs.fluentd.org/articles/cep-norikra

How does this compare to the IQbservable interface in Rx.NET? (IQbservable = Queryable IObservable)

IQbservable doesn't seem to get much press -- but here's an interesting implementation that works over TCP: https://rxx.codeplex.com/wikipage?title=TCP%20Qbservable%20P...

This uses Esper, right?

Esper is awesome all by itself.

Yes, Norikra uses Esper. That is great software.

Norikra provides:

1. schema-less processing: Esper requires schema for all events and queries, but Norikra query does not requires any schema

2. nested hash/array access by extended SQL(EPL)

3. standalone server software: applications in any languages can use stream processing over RPC

4. lightening deployment: only 3 minutes by jruby+gem install

And UDF over rubygems, connecting query output into other query inputs, and much more.

Reply on news.ycombinator.com