Ask HN: Any tool to create diagram for data lineage and ETL rules? | Hacker News Reader

Ask HN: Any tool to create diagram for data lineage and ETL rules?

12 pointsemaghsoudi6 comments

Our BAs typically use Excel sheets to show how a field from a source system has to translate to destination. Is there a tool out there that would allow me to create a diagram to show all the rules associated with fields (ex. take the sum of field A and B and divide by C from source and that'll be field D in destination) AND be able to show data lineage at a high level so that I can see all the systems involved in my ETL? It'd be great if it's fluid sort of like Prezi presentations where you can look at the high level and drill down and see all the rules without clicking through pages and separate sheets and losing context. Thank you.

Traditionally data modeling is done using ERD diagrams using tools like `Erwin` (Proprietary) and pipeline design is captured in Data Flow diagram using tool like `MS Visio`

Opensource alternative are available for both tools. In my experience BA's generally prefer to documents requirements using Excel.

I am one of the makers of https://www.gleek.io What diagram do you need to create, ERD? If yes, we have those.

However, the app cannot do drill down yet (and don't know what app can do it).

If you define your transformations in DBT (data build tool) you can view high level lineage documentation in the form of interactive diagrams. But if you haven’t started this way it is likely a labour intensive and poor way to get documentation.

You might be looking for a data catalog.

Many players out there

Data.world is one

Thank you, this was very helpful. Data.world seems to provide a lot of features (governance, collaboration, data lineage, etc) I found Manta while looking at Data.world which is specific to data lineage and seems to be what I was looking for. I just don't know if it can handle manual rules yet (like rules defined in a Python process using Pandas code) or if it can only show data lineage in databases

yEd will let you do diagrams, doesn't have the zoom feature you want.

Reply on news.ycombinator.com