Comment by jandrewrogers | Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

Parent Full thread

There are commercial database solutions focused on IoT that have no problems with these workloads (e.g. my company, SpaceCurve, or Pixia) but nothing open source. A single rack of servers arranged as a parallel system with a 10GbE switch fabric can support it if you design the software correctly.

If you look at every company that is working in this space, one of the first things you will notice is that they all use custom storage engines that do a full operating system bypass i.e. they manage all the system resources in userspace. If you do not do this, you cannot reliably get the necessary throughput out of the system for IoT. As far as I know, no scalable storage+execution engine in open source is designed like this yet. It requires much more computer science sophistication and lines of code to implement compared to traditional storage engines, so not the kind of thing you hack together over the weekend.

Cassandra is open source and used in a large number of IoT applications. Most famous one being Nest.

Also as far as throughput Netflix is doing 1.5 trillion (yes trillion) transactions per day in production on Cassandra.

This is a great example of not understanding the scaling problem of IoT. Not only is the above (quasi-)transaction rate modest by IoT system standards today, but Cassandra is not doing real-time analysis or ad hoc querying of complex relationships across those transactions at the same time, which is usually a requirement.

I know of a production IoT system in the private sector that does 1.5 trillion (quasi-)transactions every 10 minutes, so almost three orders of magnitude higher throughput. Cassandra is an okay choice for storing IoT data but it isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system.

"Cassandra [...] isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system."

That depends on how you are using Cassandra. Typically, you are expected to know your query patterns up front, and so you will lay your data out accordingly when ingesting. When done properly, this allows for ~1ms queries that return completely up-to-date results.

Do you have a source on the Netflix number? Not doubting, just curious...

Also, to nitpick, these are not transactions; these are "operations" or some other word that doesn't imply what the word "transactions" implies.

Reply on news.ycombinator.com