The Next Big Challenge for Data Is Organizational | Hacker News Reader

The Next Big Challenge for Data Is Organizational

locallyoptimistic.com

38 pointskiyanwang13 comments

I wonder how much money is wasted trying to engineer ourself out of the mess left from when the secretaries were fired. There is no way to organize data without ongoing work in catalogizing it and purge irrelevant items.

You've got it. My company just got a fancy data catalog and while it is helpful, what it's really revealed is the godawful mess of 1000x duplicated, completely undocumented tables and schemas we have, thanks to decades of neglect and shortcuts.

I've been trying to populate the catalog with a bit more explanation of what things are for + data dictionaries, but it feels like boiling the ocean.

Organizational and as a part of that access control. One of the current critical issues I see in my own work is that we have a significant amount of downstream consumers of our data and each one needs access at a different layer.

This means someone may need access to raw data, data after it's been cleaned and put in the data lake, data in an intermediate state at some point in the transformation chain, pristine fully joined master records for some data in the warehouse. Some teams shouldn't have access to certain types of data, or data earlier and later in the stack.

There's not a central way to do access control at a fine grained (row in some cases, files in others etc) level across layers and multiple services/dbs in the data stack. At least no plug and play way from what I can see. I'll be in the midst of building that layer for an entire enterprise organization soon and I expect it to be enlightening.

I'm at the same time putting in some leg work to explore how common this problem is across other companies right now for obvious reasons.

The two big challenges here are building a culture that encourages people to actually document their data, and secondly is providing the tools for data documentation.

There are lots of good tools for data profiling, but a good data catalog is something that is structured, easy to use and actually separate to the data. By keeping your catalog separate to the data you get more people involved which means more people can find and know who to talk to, to take advantage of data.

That’s why we’ve built our startup - aristotlemetadata.com - to give organisations a tool for users to collaboratively document why their data is important and what it means conceptually, along with raw column information (which is an easier, solved problem).

Building data culture is a bit more challenging, but again tools play a role, as if they can see others getting involved in data governance, can see examples and see others in their organisation getting more value from data more efficiently they are more likely to want to be involved too.

> secondly is providing the tools for data documentation.

notepad.exe is enought. The main "unsolved" problem is keeping track of the documentation, which needs secretaries on non-management level which for some reason is a no-no.

Culture eats strategy (and tools) for breakfast. Until there’s a cultural mindset shift towards how companies value data and metadata, nothing will ever truly change.

Organizing data is a problem everyone wants to solve until they figure out the cost. It's much easier for organizations to throw data over the fence and let data engineers fend for themselves than it is to get buy-in from the top to organize data closer to the source.

There seems to be a lot of hype and hope to organize domains of data much like you would products of a business. I just don't see it happening today, largely because people in engineering/product, upsteam of DE teams, are unwilling to put in the effort.

Check out data mesh. It's pretty much supposed to be where the next shift is.

Can you share a good resource on that topic? I found the following from a quick search but it all seems a bit too abstract to understand for a casual reader. Is there something more practical and approachable? https://martinfowler.com/articles/data-mesh-principles.html

I do have personal experience so Happy to have a chat through. I haven’t come across any decent material and would love to connect if you stumble upon something.

I've had the same issue with finding more concrete details about it. I've read an entire book on data mesh, and while I get all the concepts, I sense that it's more like agile (a loose methodology) in the sense that it's not going to be something that can be taken off the shelf and applied to all organizations the same way. Basically, here's all these ways to make data handling successful, but how you do it will be specific to your company. Guidelines to follow and pitfalls to avoid as opposed to a specific technology.

That said, check out https://datahubproject.io/ . It's the closest OSS I've found that is following the spirit of Data Mesh.

> Error establishing a database connection

That's an older challenge.

I moved a whole region from paper and fax to the digital ages and the re-organization and re-mapping of data was a critical workflow.

Reply on news.ycombinator.com