Hacker News Reader

Great question! At the moment it is 30 days. We've had a few requests to make that configurable, but the reality is that the most costly part is to lay it out in an optimized way in the first place which happens very early on in the ingestion path, and from there on it's only object storage cost, so if there is customer demand we can increase this.

View on HN

Any plans on the roadmap for aggregation (compression) and long term retention or would you instead ship data back out to a separate tool?

I’ve enjoyed having 3-5 years of prom metrics retention to look at seasonal traffic trends but we spent a fair bit of CPU aggregating raw metrics down to the right granularity for that kind of long term retention. My feeling is that the observability world is moving towards small, localized installations with real-time data and separate systems with cheaper, slower, but long term retention. Curious how you see it as you’re building a product in the space.

Yeah that’s basically how this system is built as well. We buffer a certain amount of data in the ingestion nodes and then offload it to object storage. In object storage it’s actually laid out in the parquet format so we could allow users to export/download this. Or like I said, the object storage cost is really the smallest piece in the system we might even just decide not to have any retention at all.

Reply on news.ycombinator.com