Hacker News Reader

Comment by alexk | Hacker News Reader

Would love to know more about the datastore that you've built. Any chance you would elaborate on this?

Sure! It's really just a Go wrapper around LevelDB. LevelDB is an in process database (like SQLite) written in C++. It's a key value store where the keyspace is ordered (so you can do range scans).

However, we've written enough code around it to turn it into a scalable distributed time series database with a pubsub layer built in. It does rollups on the fly or can preemtively downsample data coming in. We're actually working on a new version of it that we intend to open source at the end of the year.

That sounds pretty useful, something I would love to see open sourced as I've thought about making a distributed LevelDB based DB as well.

Any word on how it handles the distribution? Are we talking a quorum based system like Cassandra/Dynamo or are we talking more like a master / key range type system like HBase.

The version we're working on now uses the go Raft implementation to handle cluster configuration. Then each server periodically (< 30s) performs read repairs to ensure that it has the latest writes for its spots in the hash ring. Writes can either be done via either write once or quorum, with write once being the most often used thing. Since most of the time this data is in aggregate, we don't care about losing a few writes.

This sounds very cool, any plans on open-sourcing it?

That'd be awesome if you open sourced this!

Reply on news.ycombinator.com