The Modern Metadata Platform: What, Why, and How? | Hacker News Reader

The Modern Metadata Platform: What, Why, and How?

metaphor.io

35 pointsmarslan200018 comments

What is the product? It's really hard to figure from the home page, so I looked up on the web and found this[1]

> Today, we are excited to announce that Mars, Seyi, and I have teamed up to start Metaphor Data, a company dedicated to building out the DataHub ecosystem. Our mission at Metaphor Data is to help all organizations better understand and manage their data through the power of the metadata knowledge graph.

So, it's DataHub (hosted I guess) and ... something else? But I'm still not sure what this "ecosystem" is.

[1] https://pardhugunnam.medium.com/announcing-metaphor-data-c86...

Yeah, the post didn't do a good job of explaining what their product actually is / does.

Their About page is similarly vague:

> We are on a mission to empower individuals and companies to change the world through data. This mission started when we created DataHub at LinkedIn. We’re leveraging the learnings from that experience to build a modern, scalable, and cloud-first data catalog and discovery service.

It sure sounds like vaporware from that description. ¯\_(ツ)_/¯

All I could gather from their website is that they catalog schemas across various databases / data warehouses.

It looks like the team is all engineers — they could really use a marketing & comms person…

Thank you for your interest. We launched our product about 2 months ago. You can find out more about the problems Metaphor is solving here: https://metaphor.io/blog/metaphor-product-launch

I still don't fully get what the product is. In the launch, you state that "Metaphor has its root in DataHub [...] However, Metaphor is not a managed version of DataHub". My observation:

- perhaps intentionally, but you really go to great lengths to obfuscate what the product actually does. How is it NOT DataHub? Is it a new implementation of DataHub? Is it extending DataHub somehow? (I would expect the latter, but it's unclear exactly how it extends it, and you never mention exactly how it's different).

- it's very weird to place that important description in "who we are" (I almost skipped it), instead of e.g. "How does Metaphor work".

- lastly, I kinda' feel that this needs to be in the homepage somewhere, not in a blog post somewhere. Not sure who you're targeting with your marketing copy - but if you're targeting engineers who'd deploy a data catalog, this can't be all fluff. Give me something concrete that I can work with: I'm interested in the domain, I want to implement this in my company, but I need to understand what you're offering. Saying that you have a "platform" tells me nothing. Tell me why I should use Metaphor instead of e.g. deploying DataHub myself.

Appreciate the feedback. We'll work on improving the clarity in our messaging.

I'm losing count of all the metadata management systems being open sourced. First there was Wherehows out of LinkedIn [1]. Then there was Amundsen out of Lyft, which is now stemma.io [2]. Then there was Marquez out of WeWork [3]. Now we have DataHub out of LI. It's a little crazy how underserved this problem was several years ago and how over served it's becoming.

I use Amundsen and am pretty happy with it.

[1] https://engineering.linkedin.com/blog/2016/03/open-sourcing-...

[2] https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-...

[3] https://marquezproject.github.io/marquez/

Thanks for pointing it out. You may find this post interesting and many of these OSS projects (including DataHub) failed to deliver the promises: https://metaphor.io/blog/why-cant-i-find-the-right-data

I think it’s https://www.stemma.ai. Thank you for this broad list!

One day we will be able to automagically add an alert to a dashboard saying that the database four systems upstream is having a temporary quality issue.

Any tool that gets us in that direction is good in my book.

Thank you for the suggestion. That's exactly what we have with our recent integration with Soda: https://metaphor.io/blog/metaphor-and-soda-partner-to-unify-...

Is this just the new way of saying "big data"?

No, it’s saying how important metadata is, going over the components and key features for a platform to manage metadata, and includes some of the history of that category of software at LI.

If you’re at a company with more than 10-20 people doing data science type work, this is an important service.

Perhaps, but it's also a service you are unlikely to want to outsource if your data is proprietary.

You're absolutely right. That's why we treat our customer's data with utmost care, even if it's just metadata. Here's our security white paper if you're interested in learning more: https://www.notion.so/metaphordata/Security-Whitepaper-371ac...

Why not? I don't think that's an issue if the actual data is already sitting on a third-party cloud in some middle-of-nowhere undisclosed location or being transited through various other cloud services, right?

Depends on the data, but everyone should be treating metadata at the same classification level as the underlying data. Perhaps even moreso, as metadata leaks information about the company, versus leaking data about the user. (Rightly or wrongly, things that harm the company are more likely to be prioritized.)

Big metadata is one of the big upcoming trends in the data space.

Data tooling has gotten a lot better over the last 10 years with Spark + Airflow + Cloud data warehouses. The next step is getting everything to talk to each other with better details.

+100

Reply on news.ycombinator.com