Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

Nicely done and thoughtfully executed.

As a quibble on the suggested use cases, this platform will not work for the Internet of Things generally for two reasons. First, it lacks support for the spatial data types, including polygons, and operations, including spatial joins, that are typical of those types of data models. Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day, which implies continuous insert rates far beyond what this architecture can support and still have real-time queries. It is why Internet of Things is a completely different class of Big Data problem that does not fit on platforms like Hadoop or Spark that were not specifically designed for it.

You can design databases for Internet of Things workloads but this isn't an example of one. However, Crate should work great for more traditional Big Data workloads.

View on HN

> Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day

Interesting. Do you have a source on that?

These are the kinds of applications and workloads I have been working on for almost a decade now, spanning many industries, so it is my area of domain expertise. The above numbers are a pretty typical range for the myriad companies and organizations I work with. A small system or pilot project might be 10 TB per day. For some government systems, 1PB per day is a small fraction.

People are just beginning to take advantage of these data sources but table stakes is being able to continuously ingest and index many millions of complex spatial relationships every second, which by itself is something no popular Big Data platform supports.

The (fun!) computer science challenge of it is that the systems have to be unbelievably scalable, but you can't use hash partitioning or range partitioning, and many things you were taught about database engine design turn out to be completely wrong in this context. It is an area ripe for innovation and growth.

I was hoping you could give a source not because I was questioning the veracity of your statement, but because I wanted to learn more about it, specifically what kind of data that would be and what you do with it. I bet there are a lot of interesting challenges in that space.

Assume Nest has 10 million users, which is ballpark-ish. 100TB is 10megs per user per day. That sounds high unless my Nest is sending audio clips to Google.

I'm sure there are some IoT apps sending 1PB/day, but there are plenty that don't.

so what kind of tool would you recommend instead to deal with 100TB-1PB per day ? I'm genuinely interested.

There are commercial database solutions focused on IoT that have no problems with these workloads (e.g. my company, SpaceCurve, or Pixia) but nothing open source. A single rack of servers arranged as a parallel system with a 10GbE switch fabric can support it if you design the software correctly.

If you look at every company that is working in this space, one of the first things you will notice is that they all use custom storage engines that do a full operating system bypass i.e. they manage all the system resources in userspace. If you do not do this, you cannot reliably get the necessary throughput out of the system for IoT. As far as I know, no scalable storage+execution engine in open source is designed like this yet. It requires much more computer science sophistication and lines of code to implement compared to traditional storage engines, so not the kind of thing you hack together over the weekend.

Cassandra is open source and used in a large number of IoT applications. Most famous one being Nest.

Also as far as throughput Netflix is doing 1.5 trillion (yes trillion) transactions per day in production on Cassandra.

Do you have a source on the Netflix number? Not doubting, just curious...

Also, to nitpick, these are not transactions; these are "operations" or some other word that doesn't imply what the word "transactions" implies.

This is a great example of not understanding the scaling problem of IoT. Not only is the above (quasi-)transaction rate modest by IoT system standards today, but Cassandra is not doing real-time analysis or ad hoc querying of complex relationships across those transactions at the same time, which is usually a requirement.

I know of a production IoT system in the private sector that does 1.5 trillion (quasi-)transactions every 10 minutes, so almost three orders of magnitude higher throughput. Cassandra is an okay choice for storing IoT data but it isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system.

"Cassandra [...] isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system."

That depends on how you are using Cassandra. Typically, you are expected to know your query patterns up front, and so you will lay your data out accordingly when ingesting. When done properly, this allows for ~1ms queries that return completely up-to-date results.

spatial data types are high on our backlog. you'll be able to store geo points, query them by location, bounding box, polygon, sort them by geo distance, and also aggregate them based on geo-distance (e.g. 0-10km, 10-50km,..). we haven't thought about additional topics on top of lucene - but that's our list for a start. feedback, github issues welcome!

Reply on news.ycombinator.com