Comment by richardw | Hacker News Reader

Comment by richardw | Hacker News Reader

Parent Full thread

Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it. It’s SO expensive for an enterprise to rejig for a new platform that you can’t rely on (insert open source project here).

I managed to add one startup and so far it’s done very well, but it was an exceptional case and the global CEO wanted the functionality. But it used MongoDB and ops team didn’t have any skills, so rather than learn one tiny thing for an irrelevant data store they added cash to use Atlas with all the support and RBAC etc etc. They couldn’t use the default Azure firewall because they only know one firewall, so added one of those too. Also loaded with contracts. Kept hiring load down, one number to call, job done. Startups cost is $5-10k per year. Support BS about $40k. (I forget the exact numbers but it dwarfed the startup costs.)

Startups are from Venus, enterprise are from Jupiter.

Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.

I'm curious - as someone who hasn't been paying super close attention to the space, what does it mean to be a full data platform? Is it just having all the different flavors of DB you might need all from one vendor, or is there also tighter integration than if you cobbled it together from multiple vendors?

Essentially, yes. Different DB’s, federated queries (aka delta sharing, zero copy), definition/semantic layer tools, data engineering/pipelines, model training and notebooks, governance, data lineage, row/column/whatever access control.

It’s basically a luxury minivan. It’s may not be the fastest or prettiest or cheapest, but it’s a safe way for a large family of “data and AI people” to traverse a large organisation.

More seriously, I like to call it an “analytics workbench” in a professional setting.

Are there even any "plain data warehouse" vendors left? Even the oldest of the old-school providers seem to have crawled up the modern data stack since the data warehouse glory days, refocused on lakehouse + Open Table Formats either in their core platform or complementary products.

Exactly, look at what GCP are adding to BigQuery.

> My last gig wouldn’t allow open source software or any company that might not be there in a decade

I bet they had VMware all over the place.

Yup. And OpenShift. And Red Hat for Linux. And SAP. And IBM. But you know what? Tiny group of people relative to the impact, revenue and competitors. If we needed skills we clicked “buy”, 100 consultants would arrive who are experts, sort it and we’d move on. Not scratching around looking for people who know what we use and needing to learn 50 different open source tools. Coming from a much looser universe I learned to appreciate the principles for that context.

I've worked at a Swedish streaming company, we bought a filesystem from IBM called "GPFS" or something before it was renamed. Well we had shit performance, reliability and everything else that could be bad. It was running on FC switches recommended by IBM, config from IBM, everything from IBM and it was an absolute fucking joke, metadata timed out regularly and the solution was "deal with it". I think my superiors should've been harder claiming we haven't received what we bought but we're Swedish so we'd rather just fall over and die.

We only used MQ. Against my recommendation, but it was fine if imperfect, and a known entity to all concerned. Did not use that file system.

Battered woman syndrome

> Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant.

Hence IBM talking up Iceberg: https://www.ibm.com/think/topics/apache-iceberg

Reply on news.ycombinator.com