Timeseries data storage in MongoDB
slideshare.net
slideshare.net
What you really want for time-series data is a column db such as cassandra (or vertica etc.), perhaps HBase, perhaps a RDBMS, or perhaps a plain old log-file.
What you most definitely don't want is Microsoft Access or MongoDB. Thinking about it, MS Access might still work to a degree.
Vertica was not free, HBase was really heavyweight, & I knew i didn't want RDBMS. The sensor data itself is plain csvs ;)
Cassandra still does look interesting, although it looked like it would take me much longer to get it going; perhaps it will get rewritten. (The mongo solution only took a week to get something usable.)
This was around the end of 2009, and i don't think there were many clients available (i see pycassa dates to 2010-04).
again, ease of development is much more important to us than performance; it'll probably be a long, long time before our db engine is the choke point, at which point we'll have resources to use a heavier tool.
To expand on the parent, a column store like HBase or Cassandra is perfect as each row can represent a timeslice and then each column can represent a single event or record within that timeslice. As the row gets evicted from it's initial storage in memory, it will be written to disk contiguously, in sorted order, and batch reads of this data is sequential.
It is possible to use MongoDB to store a timeslice as a document, but it is not designed to scale out to store very large numbers of columns within a single document.
What do you mean by "very large numbers of columns" ? I've seen some mongo users with very rich, i.e. large, document models.
Even without tens of thousands of concurrent users, you'll eventually run into a deeply critical performance wall when MongoDB starts reading from disk. It's really best to think of it as an in-memory database.
Temporal Collection { _id: "X1", data_temporal : [ { time_start: SomeDate, time_stop: SomeDate, _id: "ID2" }, { time_start: SomeDate2, time_stop: SomeDate2, _id: "ID2" }]
Data Collection { _id: "ID1", parent: X1, data: { field1: "some info", field2: 34 }, _id: "ID2", parent: X1, data: { field1: "Some info new", field2: 34 } }
What is cool about this is that if you have access to the data like ID1, you can easily find out when it was added and how it changed.
If you have access to the temporal ID, X1, then at any time you can see what the data looked like.
If you need to relate data, the "foreign key" used is the data_temporal ID. In this way, it is possible to ask what your key value store data looked like at any time.
But, this could be off from the article.
This also works quite well in a relational database.
I would love to have a look at their Arduino based solution.
This might be poorly titled: it's not just about the storage as it is about the aggregation of disparate sensor data into coherent, continuous data streams.
however mongo allows you to store complex structures, think nested dictionaries/lists, and query on those nested structures, even allowing you to reach inside of nested structures to do the querying.
I guess you could do nested structured arrays in numpy, I've never done that before.