Hacker News Reader

Comment by alexmorley | Hacker News Reader

Most of these issues will be ring true to lots of folk using Iceberg at the moment. But this does not:

> Yet, competing table formats like Delta Lake and Hudi mirror this fragmentation. [ ... ] > Just as Spark emerged as the dominant engine in the Hadoop ecosystem, a dominant table format and catalog may appear in the Iceberg era.

I think extremely few people are making bets on any other open source table format now - that consolidation has already happened in 2023-2024 (see e.g. Databricks who have their own competing format leaning heavily into iceberg; or adoption from all of the major data warehouse providers).

View on HN

Microsoft is right now making a huge bet on Delta by way of their “Microsoft Fabric” initiative (as always with Microsoft: Is it a product? Is it a branding scheme? Yes.)

They seem to be the only vendor crazy enough to try to fast-follow Databricks, who is clearly driving the increasingly elaborate and sophisticated Delta ecosystem (check the GitHub traffic…)

But Microsoft + Databricks is a lot of momentum for Delta.

On the merits of open & simple, I agree, better for everyone if Iceberg wins out—as Iceberg and not as some Frankenstandard mashed together with Delta by the force of 1,000 Databricks engineers.

The only reason Microsoft is using Delta is to emphasize to CTOs and investors that fabric is as good as databricks, even when that is obviously false to anyone who has smelled the evaporative scent of vaporware before.

Microsoft's gonna Microsoft.

Very different business, of course, but Databricks v. Fabric reminds me a lot of Slack v. Teams.

Regardless of the relative merits now, I think everyone agrees that a few years ago Slack was clearly superior. Microsoft could have certainly bought Slack instead of pumping probably billions into development, marketing, discounts to destroy them.

I think Microsoft could and would consider buying Databricks—$80–100B is a lot, but not record-shattering.

If I were them, though, I’d spend a few billion competing as an experiment, first.

Anti-trust is the reason a lot of the kinds of deals you’re talking about don’t happen.

I agree. If the anti-trust regime had been different Microsoft would have bought Databricks years ago. Satya Nadella has surely been tapping his foot watching their valuation grow and grow.

The Trump folks have given mixed messages on the Biden-era FTC; I'd put the odds that with the right tap dancing (sigh) Microsoft could make a blockbuster like this in the B2B space work.

Reply on news.ycombinator.com