Data Fabric vs. Data Mesh: What's the Difference?
blog.starburst.io
blog.starburst.io
Sounds like startups might adopt Data Mesh, while it's easy to reorient org-level behavior, but big enterprises are doomed to carry forward their current messes until AI magically delivers us Data Fabric as a viable option.
Anyone having more success with these approaches than my pessimistic take implies? Is it easier to adopt Data Mesh in a large org than I realize? Is Data Fabric a more viable option than the author considers?
if you view the data pipeline, you start with Data Mesh and individual teams create datasets that maintain the data for their team's domain. following that, you then get the data fabric which blends those domain specific datasets together automatically based off of the combination of a defined data model and the declared needs of consumers. a centralized team then owns those tables but not the business logic.
an example of this is Airbnb's data stack. You can read about it here: https://link.medium.com/qzqciW7Milb
Another thing to point out: besides relying on the future promises of ML, there are already many signals that can be used by a centralized catalog for data discovery. For example: data sketches (MinHash, Hyperloglog) for joinable datasets, social signals (likes, comments, stars, etc. see Alation and Select Star SQL), lineages through data movements (e.g., Azure Data Factory and Azure Purview). If the centralized catalog uses those signals, then the data producers are incentivized to provide them for better visibility.
Perhaps apps and services will end up having to go through data-coverage and data-quality verification steps before being released. Analytics (and caching, joins, etc) as an after thought is unacceptable in this day and age.
> The highest levels of technology are not necessarily full autonomy, but situated autonomy
> All autonomous systems are joint human-machine cognitive systems
Fundamental questions: Where are the people? Which people are they? What are they doing? When are they doing it?
Decentralized adding of data. Sure, but at the expense of giving up Enterprise wide access governance and performance. It makes me uncomfortable.