Hacker News Reader

Comment by hodgesrm | Hacker News Reader

> Sure, you can stitch together specialized stores to make your own accidental distributed database for your app, but the simpler, more straight-forward and lower-latency alternative available.

Organizations like Facebook, Amazon, Netflix, Cloudflare, and many, many others have been combining specialized data stores for many years. Stonebraker, Madden, and colleagues summarized the argument for this approach in 2007. [0] At this point I can't think of any major SaaS application I've seen recently that did not have multiple data store types. That includes event streams, which are data stores as well.

[0] http://nms.csail.mit.edu/~stavros/pubs/hstore.pdf

Disclaimer: I work for Altinity.

View on HN

So, you think that the 99.99% of businesses around the world who are not in the cloud hyperscaler business should all build their systems as if they had the engineering resources of cloud hyperscalers? I've seen first-hand how enterprises with Web 2.0 envy made costly, multi-year mistakes in technology choices and architecture with that approach. Be inspired by the engineering prowess of FB, AWS, Netflix, etc., for sure, but don't make the mistake of taking the approach of a $470B (AWS) or $86B (FB) company directly when your company's situation is nothing like that. Stated another way, you can't coach exactly like Pep Guardiola if you don't have his resources and his players. Existing enterprises have a totally different starting point. Enterprises simply can't follow the hyperscaler hegemony for their digital transformation strategies and expect to win. Also, think about who is served by building inefficient, accidental distributed databases underneath your SaaS applications when that can be avoided by using a proven technology alternative already exists. It's the CSPs you're paying in the monthly cloud bill for every byte stored, every byte computed and every byte moved.

MySQL, ClickHouse, and Kafka are all proven technologies. You can run them yourself or choose any of a number of excellent cloud services to run them for you. It's not that hard to combine them and let each part serve the purpose for which it is designed. I've seen companies from small startups to organizations the size of Facebook take this approach. It's a reasonable architecture with many thousands of successful examples.

My own company is not especially large, but we run MySQL, ClickHouse, and Prometheus to manage our own SaaS platform for ClickHouse. We are happy with the result. (Though it would be nice to use ClickHouse as a Prometheus backend.)

> it would be nice to use ClickHouse as a Prometheus backend

Well... that's already possible and it works great! As you might know https://qryn.dev turns ClickHouse into a powerful Prometheus *remote_write* backend and the GO/cloud version supports full PromQL queries off ClickHouse transparently (the JS/Node version transpiles to LogQL instead) and from a performance point of view its well on par with Prometheus, Mimir and Victoriametrics in our internal benchmarks (including Clickhouse as part of the resource set) with millions of inserts/s and broad client compatibility. Same for Logs (LogQL) and Traces (Tempo)

Disclaimer: I work on qryn

Reply on news.ycombinator.com