Precog is being open-sourced | Hacker News Reader

Precog is being open-sourced

precog.com

70 pointschrisdinn13 comments

Relevant: This comes on the heels of their acquisition by RichRelevance last month (http://techcrunch.com/2013/08/14/richrelevance-acquires-prec...).

What is Precog? Why this is being upvoted? Anyone care to explain a bit?

I worked on Precog. Here's my explanation:

It's an analytics platform for big data. Basically, you upload lots of JSON, use the custom language (http://quirrel-lang.org/) for analytical queries and the platform will handle efficient storage and querying of your data.

There are also has extras like a library for charting and reporting query responses using JavaScript and an IDE for writing Quirrel and uploading data.

Thanks for the explanation. What do you actually upload the JSON into - is it a proprietary NoSQL technology, something built on top of Akka, a wrapper around HDFS or something else?

How does this compare to rethinkdb (sharding, map-reduce, indexes, queries and joins on JSON data)?

columnar storage, compression etc

Interesting, so precog breaks JSON objects into columns dot-named by their path. I wonder how rethink does it, internally?

How is this different/better than using something like Elastic Search? Does it handle real time upload/searching or do you have to bulk upload, process the data and then run your searches?

No processing necessary. Just upload JSON.

The Quirrel language is much more high level than the API for ElasticSearch. It comes with things like machine learning (e.g. clustering and linear regression):

http://precog.com/developers/how-tos/machine-learning/

Even without libraries like machine learning, you can write somewhat readable queries which do some impressive things:

http://precog.com/developers/how-tos/common-analytics/

Great, now I'm gonna be up for another three hours! Thanks for your feedback.

At this point I would not have been surprised if "PRECOG" was a codename for another NSA program.

Fortunately this looks much more benign and pretty cool.

It would be really easy to write a bot to make a snarky political reference regarding the NSA on every new article regardless of its content.

But I cannot find his previous open-sourced tool anymore, UNA.

Reply on news.ycombinator.com