Data Catalogs Are Dead; Long Live Data Discovery

Some points are good but I challenge the assumption that our data is a mess because of “unstructured data”. In fact the one example the author makes is of highly structured data - salesforce. The issue I see on my end is that we keep buying solutions focusing mostly on functionality and not evaluating how such solutions store and manage the data they produce. Lack of integrated solutions results in complicated data pipelines due to having to jump through a lot of hoops to blend thing together.

I look after a lot of data. We have gone thru many different ideas and solutions. Data catalog is the only working solution we found after we POC ton of vendor solutions.

I'm not AI scientist, but features generated by models are not structured?

Anyone have a good data lineage story while still using mainstream data processing and ETL tools like Spark/Flink/etc.? I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque.

To me, that's the core part (As the article also mentions) that allows data discovery. Otherwise you're just guessing with heuristics [0]. It also allows someone to go from map [Input Table(s) => This Table's Fields] which I think is half the battle in understanding a table.

[0] Not to say guessing is the worst, just that the information is already in the ETL code so why are we guessing?

I gather that data catalogs, a rectangle on the diagram, is dead and data discovery, an ellipse on the diagram takes over. Is there anything more to this article than geometric shape of buzzword containers?