SnappyData: OLTP and OLAP Database Built on Apache Spark

1> Most of it is straight up Apache 2.0 license. The only thing that is not Apache licensed is the approximate query processing piece which is closed source right now. We intend to build out a community and that predicates that much of what we do has to be open source

2> Since we have spliced a database into Spark such that Spark and Snappy share the same memory space. Everything that Spark supports will be supported. The data can be accessed as data frames and accessed using the Spark API (and we will be a 100% compatible)

3> Absolutely. In upcoming releases. Both from a data standpoint and from a management and monitoring, and security.

4> I think becoming a full fledged tool is not on the cards. Think of this as a enterprise class real time operational analytics platform. But we intend to integrate and work closely with the leading tool vendors (especially the AQP piece is great for aggregate class visualizations)

I'm an employee for SnappyData, just letting you know I'm going to make sure these questions get answered.

Metabase looks very interesting. We will go through it in more detail. Can you point to any doc on what it takes to create a connector? Note that Snappydata has two straightforward entry points - SQL (e.g. JDBC connector) or using the Spark APIs. Looks like the SQL connector route is the simplest?

Hi hsshah, we talk about some benchmarking we did in our technical paper (page 10, "Experiments") http://www.snappydata.io/snappy-industrial. This is not a real-life scale deployment, but may be useful.

Fundamentally, any enterprise today has to deal with OLTP data, OLAP data (transactional plus other sources), Streaming and finally machine learning. Our premise is that you can choose to use 4 different platforms for each one or move to a unified platform. Spark offers streaming, it offers Spark SQL and there are a bunch of machine learning libraries available on Spark. Also, everyone who has anything to do with data has a connector to Spark so it becomes a good data integration platform. The API is uniform across these. So it forms a good substrate for what we are trying to do. As for bugs, it is a platform that is growing rapidly , going through some growing pains, and over a period of time, it will mature. We believe that the core capabilities will become powerful over time (SparkSQL, Streaming etc.) It is a somewhat opinionated choice but one that we think will pan out over time.

LDAP and OLAP are different things.

In what way is LDAP related to this project?

(We run Spark in production and don't have the issues you are seeing at all)

@superkk Ironically, we debated this extensively and concluded the reverse (primarily because we believe the Spark API is succinct and integration with the eco-system will be easy) - much more like spark than Gemfire. In fact, the API is pretty much Spark. For instance, you don't care about updating data, you just use the Spark DataFrame API. ( https://github.com/SnappyDataInc/snappydata#quick-start-with... )

The beauty of open source is the we do not have to guess. Take a look at the source. GemFire offers some amazing capabilities and it would be foolish to not use them, but the integration of Spark into the platform is fairly deep and we intend to contribute things back to Spark over time. But marketing is not complaining :)

No we were not referring to snappy compression when we came up with the name.

Yes. Hopefully soon.