Hacker News Reader

Comment by snewe | Hacker News Reader

One reason i-banks and academics pay top dollar for data is reliability and consistency. How does Data Marketplace address errors in user-collected data? Do they guarantee its quality? If I spend $10,000 on a quarterly update on CRSP and find a systematic mistake, they will fix it without charge. Such mistakes are hard to find because CRSP has a reputation to maintain. I wonder if users hired to collect data have the same constraints.

View on HN

We want to be as transparent as possible about who is selling the data. Ultimately, it will be the reputation of the seller that determines how 'trustworthy' the data is (and how much you should be willing to pay for it).

Will there be bids for exclusive rights to a certain data source?

If users want it-- definitely. We'll need to see how many people contact us with these types of requests.

In that case, what is stopping the data consumer in directly contacting the data provider?

The name of the company is Data Marketplace i.e. a marketplace for multiple vendors. I think their goal is going to be to aggregate the various providers of data and through competition, quality should be relatively stable and hopefully improve. I'd compare it to buying a ticket through orbitz. There might be a little bit more customer service overhead if you have a problem with your ticket (or data), but the tradeoff is you don't have to shop at 20 different places to get what you want, plus the price might be more competitive.

Having worked in a data management group at a large financial institution I can say there seemed to be little relation between price and reliability and consistency. There were whole teams devoted to standardizing and checking data for errors and maintaining history, not to mention regenerating datasets whenever historical corrections were made. Pricing was mainly determined by who owned the rights to the data and how much in demand it was.

But demand -- in the long run -- is directly related to reliability/consistency. I think that this model can work, but it will take time before institutions integrate the data into mission-critical applications. I see this idea as a means of small, niche data-providers to start their own firms and perhaps start to provide the data independently of Data Marketplace.

Abundance of data sources and collectors can usually increase data quality. For instance, when we post something on mechanical turk we can ask multiple users to repeat the same task thereby scrubbing the output better.

(I also wish TC crowd-sourced grammar checking of their articles prior to pushing them out:-) )

Ditto to TC doing any sort of grammar checking.

Just mentioning CRSP makes me feel slightly woozy.

Reply on news.ycombinator.com