Comment by lukev | Hacker News Reader

Comment by lukev | Hacker News Reader

I have no doubt the problem is worse in academia, but to be quite honest nobody is really doing a good job at curating, governing and unifying data across large organizations (at least without extreme expense.)

Just look at the treadmill of architectures designed to "solve" this problem: integrated databases to "data warehouses" to "linked data" to "data lakes" to "data fabric" to "data mesh."

Organizations that succeed do so because they have the resources to prioritize data management, and do so at significant team size and expense. But any organization that doesn't (or can't) view data curation as first class, top-line budget isn't in a position to capture even a small part of the theoretical value of their data.

Consider the "data mesh" architecture, the latest fad in this space. It basically assumes that not only do you have a dedicated data curation team, each data source also has it's own dedicated team of curators capable of productizing data sources.

That kind of thing is simply out of reach for most organizations.

I lived through an acquisition in the teens that I feel was an illustrative story. An old financial services provider embarked on a massive, whole organization modernization effort. The new CEO went all in and actually made the necessary investment in staff and headcount, which was remarkable in itself.

A big part of the effort involved providing customers with big data solutions and all the implied benefits that would entail. They jumped aboard the hype train with gusto. Each quarter progress was reported and margins improved with a clear upward trend. From the trenches it was clear that even with the investment made, it was still not enough to reach any of the goals.

After a year of “modernization” the company was acquired, the buyer literally salivating over the coming profit margin increase and “synergies” in selling into related markets. After another year post acquisition it became clear that the benefits were simply out of reach, and the profits did not arrive. The acquiring company was left with the low margin business it had always been. Layoffs ensued as the buyer sought to salvage something from the transaction.

Turned out the entire modernization and big data effort had been an elaborate bait and switch committed by the board and CEO of the acquired company, masterfully executed.

Same as it ever was.

>Turned out the entire modernization and big data effort had been an elaborate bait and switch committed by the board and CEO of the acquired company

Are you saying that rhetorically, or do you actually believe (perhaps with evidence) that the modernization and big data effort was not undertaken in earnest?

> masterfully executed.

I feel like this part you omitted clarifies my opinion on the matter.

Observation/mini rant: unfortunately the data industry is both very susceptible to fads and hype, and does not have widespread standardization or a generally-accepted set of best practices. The subset of data "influencers" whose discourse occurs mostly through Twitter and Substack have especially heavy sway in what is the current Data Big Thing. These people introduce new buzzwords and ride them to the mainstream, or redefine what were previously more-or-less anchored down concepts into new interpretations. It feels tepid and arbitrary, almost postmodern. On top of this, much of the "thought leadership" is being driven by individuals who lean heavily towards the soft-skill side. So we have a heavy overindexing on strong opinions around organizational methodologies, team structure and roles/responsibilities, and other Big Ideas without much engineering representation or consideration. What spawns from this are things like the "data mesh" and the bastardization of both data concepts and clear communication/thinking. The data mesh white paper perfectly encapsulates this[1]. I challenge anybody to try to read it and understand what the hell the author is even vaguely trying to say after a few passes.

[1]: https://martinfowler.com/articles/data-mesh-principles.html

I do see what you are saying, though I think a lot of what they say makes somewhat more sense if you are familiar with their consulting "bubbles" and the language used within them.

But the problem exists even on a concrete technical level. Look at how many different "big data" products are out there. The Apache project alone has probably 15 different tools that largely overlap.

Reply on news.ycombinator.com