CMU Time Series Database Lectures | Hacker News Reader

CMU Time Series Database Lectures

db.cs.cmu.edu

211 pointsassface17 comments

Andy Pavlo never ceases to amaze me! I'm adding this to my must watch list.

He is a very colorful guy, I was at Brown back in the days and had 2 years overlapped with him.

I took 2 databases courses organized by Stan and Andy, like his adviser, Andy was very good at explaining complicated things in layman terms.

Few people manage to combine "humble", "nutcase", and "kind" as well as Andy Pavlo.

I'm interested in peoples experience with Timeseries DBs. I deal with a lot (~1 TB and growing) of sensor data and store it all in sql server. Seems to work pretty well.

Conceptually, there's no real benefit to using a TSDB if all you need is a list of values (or pairs), and storage/performance is not a concern. Are you using the time-seriesness of the data, or is it just happens that your data come in a time-series shape and you don't do anything special about it?

Many TSDBs offer specialized advantages:

* need fast append-only changes? influxdb covers that

* need performance for short-term time-series data? go for Prometheus

* most TSDBs offer compression to the data stored. In TSDBs, especially the timestamps, can be highly efficiently compressed. Some TSDBs (Chronix[0], for example) even offers an option to completely drop the timestamps in exchange for space savings. E.g., in Chronix's home page, they listed: The dataset contains about 3.7 billion pairs and takes 108 GB serialized as CSV. Chronix needs only 8.7 GB to store the dataset.

* high-level analytics that only makes sense for time-series data. Chronix[0], again for example, offers a list of high-level functions[1], such as derivatives, integrals, frequency, scaling, or even SAX. Note that you could implement in SQL Server, but would be a huge pain. It's really great that you could push these to the database layer.

(I'm not affiliated with Chrnoix in any way, I just happen to be a very happy user in my recent project. You can see a review of Chronix by Adrian Colyer in [3])

[0]: https://github.com/ChronixDB/chronix.server

[1]: https://github.com/ChronixDB/chronix.server#function-query

[3]: https://blog.acolyer.org/2017/03/10/chronix-long-term-storag...

There's a bunch of things we do, but when we do things like transformations, then they generally aren't temporary, so we generate new transformed series of data. There's some aggregations where we use SQL to do some basic stats, but a lot of transformations are done via application level transformations which can essentially do anything with any set of data to produce new data.

The most interesting thing about Chronix, to me, seems to be data size ( would be interesting to see what the performance is like too ). I think I'll do an experiment and spit a bunch of data into it and see how it does.

I can vouch for KDB as I've been using it extensively for financial time-series for 7+ years now. The free version is 32-bit and the 64-bit version will cost you. Other than that, the q language is a super-terse, functional language that lends itself to writing code quickly - that runs even faster...

The speakers in this video series give great overviews of their products and their uses. This lecture series reminds me of an in depth meetup, which is nice.

Caught this one made me wonder...

"Due to problems in previous seminars, we will not be allowing therapy snakes into the room anymore. Thank you for your cooperation."

Is that an inside joke or is there a snake story somewhere?

Pretty sure it's just a joke

the x-wife thing is a joke too AFAIK - but why? i don't get it and it reads very unprofessional. explain.

Humor, and human language in general, is complicated and its appropriate use is subjective.

If you don't understand it, maybe it just misreads unprofessionally.

It's weird and a little tone deaf but a common older brand of humor. The speakers (in general, I only skimmed a few of the videos) don't seem to share it.

This is a college, the audience is teenagers and the lecturers are academics. They don't experience any need for professionalism and I don't think they should.

> explain

It's a thing called humor. Many humans, even older ones with an actual profession seem to relish it. This even includes some great academic minds like Einstein and Feynman, whose accomplishments could dwarf those of most other humans who dislike this humor thing.

It has been observed by prominent humanologists however that in certain grey corporate circles (especially in protestant-derived cultures) it is frowned upon -- much alike in our own professional culture here on Vulcan.

Inside joke.

Reply on news.ycombinator.com