Show HN: Harmonized Data Platform | Hacker News Reader

Show HN: Harmonized Data Platform

12 pointsBBx365 comments

Hi HN, My background is on datascience and I thought during the last few weeks on how to turn the data into commodity problem: Getting any meaningful out of data can be erroneous, time-consuming and includes repeatable processing work (and it is done over and over again). Im trying to mitigate this by harmonizing data, so it is ready for being consumed via an API or Spreadsheet.

After a few iterations I ended up with this prototype that I wanted to share with you. Please notice, that this is an early prototype and not a finalized product yet. Im also delighted to know your opinions or thoughts or advices. You can get the first impression at https://databarnum.com/

View on HN

Hi, my team does this - there is a whole industry growing around productizing and democratizing datasets.

Some stuff to check out:

- AWS registry of open data: https://registry.opendata.aws/

- Datahub (metadata platform for presenting datasets with documentation): https://datahubproject.io/

- For a data exploration frontend, consider Datasette from HN regular SimonW: https://datasette.io/

- Nasdaq Data Link (formerly Quandl): https://data.nasdaq.com/search

- FinServ has had harmonized data for years, in the form of horizontally integrated data solutions from Bloomberg (B-PIPE), GS (Marquee), S&P, and LSEG.

- Data.gov: https://catalog.data.gov/dataset

- A number of data marketplaces have popped up to facilitate easy access to datasets (free and paid) through standardized data sharing methods, including Snowflake Marketplace, AWS Data Exchange, Google Cloud Marketplace, Databricks Marketplace, and Datarade.

You'll want to explore the space and get an idea of what the current State of the Art is, so that you can understand how you can best contribute.

> how to turn data into a commodity

This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:

* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)

* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?

* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?

I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/

I'm happy to answer any questions :)

This project is interesting, we at https://www.databoutique.com are building something similar, a curated dataset marketplace for web scraping data. We believe that using standardization, quality controls and high density verticals, we can cut prices and time to value for web scraping data.

When looking at these kind of datasets, there's couple of very high level things I would like to know: 1)How granular the data is in terms of geographical location or time 2)What geographical areas and/or time scopes it covers 3)One-of dataset or constantly updated 4)License

Having these as filtering conditions or as summary would help in browsing the datasets.

I'm not really sure what I'm looking at. Is this something like Google Dataset search? It sounds like you're adding something else on top of this with the "harmonizing" you mentioned, but it doesn't appear to be implemented yet unless I'm missing something.

Reply on news.ycombinator.com