The Sisyphean struggle and the new era of data infrastructure
jack-vanlightly.com
jack-vanlightly.com
It is true that having a unique idea for a startup is no longer (if it ever was) sufficient since it will be copied almost instantaneously by another company, that may actually do it better than the original startup. In the hardware area, this is absolutely true. Any decent-selling product on Amazon is guaranteed to have Chinese copies within a matter of days/weeks.
Also, I think databases were always a commodity. About 40 years ago in a developing country, I was using dBase & FoxPro (obtained free) as a database for my father's business. Today's databases are a lot more sophisticated, but commoditization is not new.
My assumption is that the author is painting out a longer-term vision both for Confluent and its audience.
It probably makes more sense in the context of other posts, e.g.
> This trend towards object storage is not just happening at Confluent but across the data ecosystem. Many different types of cloud data systems are integrating object storage into their architecture, and I have covered a few of these such as Neon and ClickHouse Cloud. There is also a flurry of start-ups doing logs (such as the Kafka API) over object storage directly (six and counting). There is money being plowed into data systems that use object storage as the primary, and sometimes, only storage layer.
https://jack-vanlightly.com/blog/2024/5/2/hybrid-transaction...
In other words: managed Kafka is becoming a commodity and Confluent should "start with why" to figure out where to go. I figured the advice and perspective is of general enough interest.
Sigh. They built ClickHouse Cloud in a year, not ClickHouse. ClickHouse started in 2008 as an in-house project. [0] It had run production workloads for years before it was open sourced in 2016, let alone before there were public cloud offerings.
Nobody builds a database in a year. It's typically 10 years before the decent ones are really stable.