Data-as-a-Product and Data-Contract: An evolutionary approach to data maturity | Hacker News Reader

Data-as-a-Product and Data-Contract: An evolutionary approach to data maturity

blog.owulveryck.info

88 pointsowulveryck11 comments

I can’t see this data mesh idea working in any workplace I’ve worked at.

Application developers and application administrators don’t care what happens when data leaves their application, they don’t understand what happens to data when it leaves their application, and they don’t care to understand what happens to data when it leaves their application.

And even when you are lucky enough to find a developer or admin who will go above and beyond their management will slap them right back down and tell them to do what they’re told.

The whole mesh idea seems to think that this is a technology or architectural problem, but it’s really a people and cultural problem.

All hard technology problems end up being people problems.

The hard part about modern data which this article doesn't really get at (but also isn't directly addressing) is that you have tight coupling of the destination and the application database API (Database schema), and by conventional standards absolutely massive response payloads (GB instead of kb).

Any web API can quickly get to the same point if you're trying to get very specific data on the entirety of the backing data store.

Hello,

In the data mesh, we mark the difference between an operational data (a representation of a state) and an analytical data which have (at least) one temporal dimension.

I did not insist on this in the article because that was not the point.

What you mention is right, it is very difficult to find incentive to get operational data. It is not the same in the analytical world. At its core, data mesh is a paradigm that applies to the analytical world.

I'm still digesting the full article, great different angle to look at things. My comment hence is purely focussed on the reply of mr_toad. For me, the core of the mesh idea is the fact that everything is a people and cultural problem, that's the core of the paradigm. I do agree however, that too much content on the topic addresses the technological implications of it. It's (only) the means to an end. And this people and cultural problem, needs to be fixed both bottom as up. Yes, you need to find developers who care about the data, but you also need management to not slap them right back. In fact, it should be managers to tell developers that it is their job to offer the data back to the organization in a reusable format. That's when you have reached a true data literate and data driven company. What the mesh idea tries to resolve, is that a central data team can not understand all domains and business goals and data linked to it. Rather than pursuing this dream, it aims to explain to the domains, and in your example developers and admins, the value of data in general, which just might have a bigger chance of success.

Have you checked out the latest European Dataspace movement and the Gaia-X developments? They're generally infrastructures trying to support exactly this level of data maturity!

Links plz?

The rabbithole begins from the initial protocol - https://docs.internationaldataspaces.org/ids-knowledgebase/v... and the GitHub repository for the IDSA - https://github.com/International-Data-Spaces-Association/ids...

Thanks you, I will have a look

Gaia-X is still active?

From what I know/remember it started as an attempt to create an EU cloud (to become independent from US providers)... then funds started to dry up, you can just search Gaia-X here on HN to get an idea of the way things developed from 2019 to https://www.theregister.com/2024/01/08/gaiax_future/

very good article

Thank you

Reply on news.ycombinator.com