Why observability requires a distributed column store | Hacker News Reader

Why observability requires a distributed column store

honeycomb.io

92 pointsboyter13 comments

The article is not actually about observability and why it requires a columnt store but a marketing article about Honeycomb's database, IMO. The topic is better explained in another article they link to - https://www.honeycomb.io/blog/so-you-want-to-build-an-observ... .

Also while PostgreSQL as well as MySQL were initially row-based DBs, they both nowadays support columnt store. PostgreSQL supports it via Citus and MySql (MariaDB) supports it natively.

Both articles try to redefine "Observability" into a set of requirements that (they claim) only their product meets. It's fine to argue that your product is the best, but it's a bit much to come up with an entirely new definition for a widely and loosely used word, which is not that similar to the common meaning.

I agree they’re a bit strong on the sales pitch, but they’re not trying to come up with a new definition so much as bring the words usage back to its technical meaning https://en.m.wikipedia.org/wiki/Observability

TIL! Still, the common usage within technology is not that meaning, and many of the claims are put perhaps more strongly than is justified.

As I understand it, honeycomb popularised observability before it got redefined by other vendors to match their offering. So I think you're the victim of other vendors' marketing here.

Column store is just the first step in OLAP optimization. You also need an engine that's optimized for columnar calculation(minimize data transfer/copy, vectorized calculation, etc..). It's the reason why purpose-built columnar DB like Clickhouse still run circle around MySQL/PgSQL in analytic jobs, even when the later use their column store.

This totally makes sense. At Spacelift[0] we're using Datadog for most of our observability. It's great overall, but the way traces and retention work there, debugging issues you didn't predict beforehand is nigh impossible.

Retained spans don't include the whole trace with them and you can't make them do that. You can't filter traces by fields in multiple spans. You have to set facets beforehand on fields you want to filter on, otherwise you can't.

So we created a small fork of the dd tracer repo which sends all spans to firehose on the side and stores them as compressed JSON files on S3. There we query them using Athena.

It's cheap, fast, and has 100% retention for an extended period of time. If you get the partitioning right and create a temporary table with just the time period you're interested in before starting a debugging session, then Athena will also be incredibly cheap.

Datadog APM is still nice for exploration and initial analysis, but Athena is great for any required deep dives.

PS: We'll probably write a blog post about the specifics of that setup at some point.

[0]: https://spacelift.io

Their column store is really fast, but I am disappointed they did not mention what you lose: complex queries. You cannot do joins or combine data from separate events. For example if want to measure errors per active worker, as fraction of total load, all those fields have to be sent in same event to Honeycomb from the same service. This kind of question is answerable with promql, and other systems I have used.

It really like using Honeycomb vs Grafana + Prometheus for figuring out what is broken. I cannot wait until their dash boarding and alert/slo management mature enough that I don’t have to use separate products for it.

Early in Prometheus' life (2011-12), I had really pined for an indexed sparse columnar data store similar to the semantics of what Bigtable provided Google internally for time series storage for many years.

Pre-public-Git repository attempts at Prometheus' data store had used Cassandra in prototyping, but the Thrift protocol, which Cassandra then used, was unfortunately riddled with bugs in the wireformat (how were enums indexed over the wire? turns out it varied by what kind of client you were using even in the same distribution release, which was to me incredulously bad), which resulted in a lot of byzantine bugs. I eventually gave up on Cassandra and settled on a different data model using LevelDB (Levigo bindings). It worked well and was stable and allowed me to not need to worry about:

https://danluu.com/file-consistency/

and

https://danluu.com/deconstruct-files/

I had a feeling in the back of my mind once Prometheus matured a bit it could have fallen back onto Cassandra for distributed series storage (again, ca. 2012 thinking), since it largely could have used an append-only storage model for series archival. Even the metrics metadata (metric families, fields, and indexing) could have been stored in such a way.

It would be fun to revisit these design problems with today's technologies and knowledge.

Why X requires you to buy our product

I've found honeycomb ux quite dumbfounding for a new person. In general the ideas behind it are good. I guess one would have to spend a few hours reading documentation. This is similar to many other tools. Until then I can just look at ready made views posted by others. A modern SaaS company uses tens of different internal tools all with their own query languages so intermediate users can't reasonably be expected to know them all. Yet these users would like to answer their own questions by doing somewhat customized queries.

This article does a very good job of explaining the main difference between column vs row databases at the storage level. Knowing this low level structure is extremely helpful to make considered trade offs when choosing the right technology.

This is worth a read.

Reply on news.ycombinator.com