Tigon: High-throughput, real-time stream processing framework | Hacker News Reader

Tigon: High-throughput, real-time stream processing framework

github.com

36 pointsalexatkeplar13 comments

In the Big Data era, I can no longer feel excited about most projects, because my project ideas don't operate at that scale - at least initially. So I can't imagine anything cool to build with it - unlike Redis or Postgres, say.

You can take a look at sample applications built using Tigon : https://github.com/caskdata/tigon-apps

Also, Tigon is the realtime processing framework used in CDAP (cdap.io). Sample applications built using CDAP (Cask Data Application Platform): https://github.com/caskdata/cdap-apps

So this is their own custom solution rather than using Kafka and Storm?

Yes, it is a transactional stream processing solution implemented on top of Hadoop YARN and HBase (using Tephra as the transaction engine).

Why not Kafka+Storm : Tigon is primarily targeted at simplifying stream processing application development -- and in that regard provides exactly-once processing guarantee, dynamic scaling of processing nodes, transactionally access to user datastore etc.

Can you talk more about how you guarantee exactly-once semantics? The documentation is pretty vague on the architectural details.

You can get more technical details from the QCon SF 2014 talk:

Slides: http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 Video: http://www.infoq.com/presentations/acid-stream-processing

Underneath it uses Tephra (https://github.com/caskdata/tephra) as the distributed transaction manager for HBase together with a queue implementation on HBase to provide exactly-once processing.

Thanks for clearing up the use case. I can now understand how it is completely different in use than a Kafka/Storm implementation. The reference implementations seem intriguing with the example of a real time ad network marketplace.

Sure! I would encourage you to also take a look at CDAP (http://cdap.io) (http://github.com/caskdata/cdap), where Tigon is used as the realtime stream processing framework.

CDAP's features allow developers to build even more powerful bigdata applications (https://github.com/caskdata/cdap-apps).

Here are some CDAP guides/tutorials that might be of interest: http://cask.co/guides.

Specifically this one: https://github.com/cdap-guides/cdap-flow-guide

Thanks a lot!! You helped immensely. Coincidentally, you posted this at a time that a large project came up requiring transactional and "Exactly Once" delivery requirements - where Kafka, Rabbit, SQS, etc won't work well. Thanks again!

Great! If you have any questions about CDAP, please don't hesitate to contact us at : cdap-user@googlegroups.com. We are also on IRC Freenode : #cdap

Hate to nitpick, but in FF on windows, the font is almost unreadable (very choppy, and the top few pixels of each character is cut off). The GothamLight font is not doing you any favors!

Point taken and your are right. It sucks! we will fix it.

Reply on news.ycombinator.com