Hacker News Reader

Comment by politelemon | Hacker News Reader

Is there an underlying assumption here that all of the datasets' domains are perfectly in sync with each other in the context of domain metadata?

As an example, a Team1 might define the manufacturer of a Sprocket as the company that assembled it, whereas a Team2 might define the manufacturer as the company that built the Sprocket's engine. Since the purpose of a datamesh is to enable other teams to perform cross-domain data analytics, there needs to be reconciliation regarding these definitions, or it'll become a datamess. Where does that get resolved?

View on HN

Data mesh is not a complete framework, more sociopolitical rather than technical at the moment. When tested in practice, I think you already allured to a key technical component that will need to be more central, I.e. reconciliation. What that means in terms of domain ownership of reconciliation that is an open question.

The chief data officer in close collaboration with the chief data engineering officer must elaborate automated normalization guidelines backed with implementations used across all data streams to insure any skew in the data model is limited to non production environments and all data entities are materialized consistently across the whole data model.

what type of company you are working for? Usually there is not even CIO, I haven't even heard about company with both CDO and CDEO (or even CDEO itself).

I thought big portion of need that data mesh fills is the organizations who are missing resources in their core BI team.

There's no magic. you need a core team that pivots from writing code at O(n) cost enterprise wide to more or less amortized O(1) where n is the amount of work required to process a new data stream - ie having to write code once per stream vs once for a standardized stream that gets reused. With only datamesh I don't think it's going to work but with standardized tools that allow your teams to write transformations and code as data then every team effectively gets access to a self-service data warehouse with only access to pre-approved happy paths that can be automatically monitored for the most part. That's where you gain in efficiency and can let your BI teams focus on BI and not boilerplate code, infrastructure, conformity, etc.

Yes, its similar path that I am taking (while leading BI in my org.) Having first sights of self-service from analysis perspective is super easy thanks to tools like metabase.

For bringing data in, thats completely different story, especially in non-tech organizations. The gap between how power user from specific department and somebody from my team brings and transforms data is still too big and somehow hard to enforce (following naming conventions, keeping same data formats for same columns, lowercasing certain columns, so joins are done correctly...). They usually have their "playground schemas" they use, but its very far from saying that they "own" data quality there.

A data mesh approach probably wouldn't work in the sort of organization you describe.

IMO - To make it work you need a consistent taxonomy or way of translating from a particular domain to some sort of interchange format.

If you have that then a set of centralized tools can pull from the separate domains using a core set of protocols to produce reports etc.

Reply on news.ycombinator.com