Well, one of the differences is that we only require PostgreSQL as a external dependency in contrast with DataHub (MySQL, Kafka, ElasticSearch). Please correct me if I wrong about this list of DataHub's external dependencies.
66 karma · joined December 29, 2021
We are continue working on a documentation, thank you for bringing this up! We'll take a look how this can be improved.
> For example: all the steps under 3. are not part of ODD, or are they? Only step 1 is performed in ODD, yes?
Yes, that's correct. In this scenario ODD acts as a source of knowledge about the problem.
Actually everything is working on a push basis in ODD now. ODD Platform implements ODD Specification (https://github.com/opendatadiscovery/opendatadiscovery-speci...) and all agents, custom scripts and integrations, Airflow/Spark listeners, etc are pushing metadata to specific ODD Platform's endpoint (https://github.com/opendatadiscovery/opendatadiscovery-speci...). ODD Collectors (agents) are pushing metadata on a configurable schedule.
ODD Specification is a standard for collecting and gathering such metadata, ETL included. We gather metadata for lineage on an entity level now, but we plan to expand this to the column-level lineage at the end 2022 — start 2023. Specification allows us to make the system open and it's really easy to write your own integration by taking a look in what format metadata needs to be injected in the Platform.
ODD Platform has its own OpenAPI specification (https://github.com/opendatadiscovery/odd-platform/tree/main/...) so that the already indexed and layered metadata could be extracted via platform's API.
Also, thank you for sharing links with us! I'm thrilled to take a look how BMW solved a problem of lineage gathering from Spark, that's something we are improving in our product right now.
May I ask you what do you mean by saying "all steps under 3"? Are you referring to https://docs.opendatadiscovery.org/use_cases/dq_visibility?
As for the
> How is the lineage generated or manually maintained
All lineage in the platform is generated and not manually handled by user in the UI. We are leveraging ODD Specification (https://github.com/opendatadiscovery/opendatadiscovery-speci...) and all ODD Collectors (agents that scrape metadata from your data sources) send payload to the ODD Platform in this specification's format. ODD Specification introduces something called ODDRN — OpenDataDiscovery Resource Names. These are basically strings, identifiers of specific data entities. All ODD Collectors generates same identifiers for same entities, allowing us automatically build a lineage graph in ODD Platform.
Not letting a user to manually change lineage in the UI is kinda our solution to one of the lineage problems. This way users can be sure that the lineage is correct, up to date and no one messed with it at least in the UI.
Of course if there's an described API endpoint, there's a way to change the lineage by sending a request on your own (e.g. via curl or custom script), but I wouldn't call it manual. This approach allows companies and users to write their own integrations, making the system open.
We have a lot of repositories on GitHub, please feel free to pick any issue from the list. Do not hesitate to ask us anything in GitHub issues' threads or in our Slack community. I'll provide links for your convinience
1. ODD Platform GitHub: https://github.com/opendatadiscovery/odd-platform
2. Slack Community: https://go.opendatadiscovery.org/slack
3. Documentation with information on how to contribute: https://docs.opendatadiscovery.org/developer-guides/how-to-c...
Let me cover some of your reactions from my perspective as a Data Engineer. Please feel free to add your opinion on those
> Shorten data discovery phase. In my experience, analysts and data scientists are always very familiar with what relevant data exists, or else they can find the right people to acquire what data they need. Often, kick-off meetings for new projects cover with stakeholders which data is useful.
You're right, but from my experience it's not always the case. Sometimes finding the key person/team responsible for a dataset might be challenging. You mentioned the kick-off meeting, about which I agree, but it's not always the silver bullet. Data goes outdated/deprecated all the time and we are trying to solve a problem of telling about this to all people which may be affected by this as soon an as easy as possible.
> Know the sources of your dashboards and ad hoc reports. All dashboards I am aware of surface this sort of information
Again, you are right. All dashboard services and BI tools can show you from what data source what data are they getting. But from my experience sometimes it's useful to take a look at the origin of data some dashboard uses. This is where end-to-end lineage comes in hand. Also, I consider useful to have metadata of all of my dashboards from all of my company's BI tools in one place.
> Deprecate outdated objects responsibly by assessing and mitigating the risks. This is a good idea, however, it is challenging
Couldn't agree more. We are working not only to improve our way to solve this problem, but the solution itself, if it makes sense. We are basically trying to find a right approach to this and offer it to everyone else. I know it's ambitious and really is a loud statement, but I hope we are getting there.
In overall, thank you for your input!
@germanosin, would you like to add something I may have missed?
Thank you for the input, we are going to work on this.
It explains how to set up a platform in a way that certain features were enabled/disabled. Maybe this is something you will find useful in a way.
It'd be great if you could provide an example, if I got you wrong
Perhaps you would be interested in a call with us where we can answer all your questions including integration with your infrastructure, provide help configuring the platform if needed, etc?