YC-Funded Data Marketplace is an Amazon for Structured Information | Hacker News Reader

YC-Funded Data Marketplace is an Amazon for Structured Information

techcrunch.com

133 pointsjmorin00759 comments

One reason i-banks and academics pay top dollar for data is reliability and consistency. How does Data Marketplace address errors in user-collected data? Do they guarantee its quality? If I spend $10,000 on a quarterly update on CRSP and find a systematic mistake, they will fix it without charge. Such mistakes are hard to find because CRSP has a reputation to maintain. I wonder if users hired to collect data have the same constraints.

The name of the company is Data Marketplace i.e. a marketplace for multiple vendors. I think their goal is going to be to aggregate the various providers of data and through competition, quality should be relatively stable and hopefully improve. I'd compare it to buying a ticket through orbitz. There might be a little bit more customer service overhead if you have a problem with your ticket (or data), but the tradeoff is you don't have to shop at 20 different places to get what you want, plus the price might be more competitive.

Abundance of data sources and collectors can usually increase data quality. For instance, when we post something on mechanical turk we can ask multiple users to repeat the same task thereby scrubbing the output better.

(I also wish TC crowd-sourced grammar checking of their articles prior to pushing them out:-) )

Ditto to TC doing any sort of grammar checking.

We want to be as transparent as possible about who is selling the data. Ultimately, it will be the reputation of the seller that determines how 'trustworthy' the data is (and how much you should be willing to pay for it).

Will there be bids for exclusive rights to a certain data source?

If users want it-- definitely. We'll need to see how many people contact us with these types of requests.

In that case, what is stopping the data consumer in directly contacting the data provider?

Just mentioning CRSP makes me feel slightly woozy.

Having worked in a data management group at a large financial institution I can say there seemed to be little relation between price and reliability and consistency. There were whole teams devoted to standardizing and checking data for errors and maintaining history, not to mention regenerating datasets whenever historical corrections were made. Pricing was mainly determined by who owned the rights to the data and how much in demand it was.

But demand -- in the long run -- is directly related to reliability/consistency. I think that this model can work, but it will take time before institutions integrate the data into mission-critical applications. I see this idea as a means of small, niche data-providers to start their own firms and perhaps start to provide the data independently of Data Marketplace.

Hey, it sounds like these guys would be a perfect customer for ITA's Needle. Needle provides a GUI-based way for nonprogrammers to scrape data, clean it up, and get it into a structured database where they can create views on it and all that databasey stuff. http://www.needlebase.com/

I hope this is at least a tenth as awesome as it sounds. Have you used it?

I watched an internal demo :) Where they scraped a table off an MLB site, live, using their GUI and prettified the data a bit. If you check the blog there's some demonstrations, and the Village Voice gave it a test drive: http://blogs.villagevoice.com/music/archives/2010/01/pazz_jo...

I'm in Mountain View. Can I get a demo?

up to the team, if you email me I'll forward it to them. Or just use the contact email on the needlebase site, that goes right to them.

I'm really interested in what would make it really awesome for you. In particular, is the cleanup/database part important for you? If you or anyone on HN wants to talk about it (and especially if you've tried http://siteparse.com), my email is in my profile.

What we are doing is not really ready to launch, but we match the first part of this description perfectly ("a GUI-based way for nonprogrammers to scrape data"). The GUI is web based, you can watch a demo and try us out immediately (you get your data in CSV):

http://siteparse.com

How funny! Right as I start talking about my startup to buy and sell datasets, it turns out YC had something like this in secret the whole time. At least I know I'm doing something worth while.

Let's talk! I like where you are going with Webscaled-- maybe we could work together. Shoot me an email me when you get a chance: matt @ datamarketplace.com

Sounds a little like Microsoft's "Dallas":

http://www.microsoft.com/WindowsAzure/dallas/

Also sounds alot like Infochimps:

http://infochimps.org/

Reminded me most of Factual: http://techcrunch.com/2010/02/03/factual-1-million-seed/

As a pay-for-service approach, it seems reasonable: you say you need some data, the appropriate people come up with it in the format you want, and you pay and get the data. But the Amazon-like angle seems odder to me: How will each of the bits of structured information manage to be sold more than once? Since most isn't copyrightable, anyone can buy a copy and then put it up online for free. I could go buy that Wal-Mart Store Locations data set for $30 right now, and publish it on my blog. Are you hoping EULAs will be enough to prevent this, or just that in practice it won't happen enough to matter, or people won't notice that the data is free online and continue buying it anyway?

This is an interesting question that we've been giving a lot of thought to lately. We hope that it won't be a big problem, but if it is we will need to implement public EULAs and a dispute resolution process.

In terms of multiple purchases, we suspect there will be some datasets that are more popular than others. A list of Wal-Mart stores for example could be useful to a number of different people, such as investors, real estate developers, business planners, marketers, etc.

I think you guys are doing the right thing by not worrying about EULAs and other protection mechanisms at this stage in the game. Stay focused on satisfying your customers and improving the product as much as possible until you start making major headway.

Heck, putting it on your blog? What's to prevent you from buying the list of Wal-Marts, and selling it on right back on Data Marketplace?

Is it just me, or are more YC companies launching early this cycle? I can count 5--Etacts, Crocodoc, Cardpool, 140bets, and Data Marketplace--that have already launched.

Is this just a product of them having more companies this cycle, something they're stressing this year, or just a coincidence?

I think the competition to be a YC company is high enough now that most companies show up the first day with an actual product.

Yeah, it's true. We definitely felt like we were playing catch-up from day 1. Not that its a competition, but I think companies benefit more from YC having a working product earlier on.

I was referring to the competition to get in.

It makes sense that if YC is getting more applications than their applicants will be better and more prepared. Several companies in this round had already produced several products, were ramen profitable, and had been operating for sometimes years previous to becoming a YC company.

They will still get lots of help. Correct me if I am wrong. I think the biggest advantage of being a YC company right now is the kind of press, reputation, and trust of investors founders might get. PG's advice is certainly valuable but their connections are the biggest deal.

I don't think you can put the value of YC in one sentence.

lots of groups switch ideas mid-session and have to start from scratch on day >1. but yes, launching early (more specifically before demo day) is quite an advantage.

The irony is that while YC is extremely useful for founders, those who get in probably need that help the least.

Actually 5/27 is probably lower than the proportion who've usually launched by this point. I must be losing my launch-nagging edge.

Are there many companies that have launched before YC? Do you encourage or discourage this?

Yes, there are quite a lot. E.g. Virtualmin, which IIRC had been around for about a year before YC. Or Airbnb, which had been for almost that long.

We don't encourage or discourage it. We care much more about how good companies are than how long they've been working.

Nice - I already made a sale!

I didn't see it posted anywhere, but on a $4 sale, I received $3.68, so that's an 8% fee if it's a flat rate. (Or it could be something like $0.28 + 1%)

@matthodan, you might want to post that somewhere / make it more obvious.

[update]

A second sale of a $2 item brought in $1.84, so it appears to be an 8% fee collected off each sale.

I'm getting an error when trying to upload a CSV data set. :( Not very encouraging to be prompted with an error at my first attempt to use the product. Maybe TC jumped on the story a little soon?

Sorry to hear that. Would you mind emailing me the CSV file at steve@datamarketplace.com? I'll try and recreate the problem and let you know when it's fixed.

Seems like an awfully low signal to noise ratio in the requests at the moment. The vast majority of them are very unrealistic, in not outright spam.

"Request For Data List of all Assisted Living care centers in the U.S., including name, address, phone number, website and any photos on their website. There should be about 20,000 of them. Budget: $20.00 Deadline: March 25, 2010 10:40 PM"

Yeah... I'll get right on that...

Some data providers already have a master DB, so all they need to do is write a few queries and pull the info and sell it n times.

This looks really interesting, though if one is looking for publicly available data, I think it's an open question of how much of the value is in providing a marketplace vs. actually doing the aggregation, organization and some analysis.

Two other interesting startups in this space that are focussed more on the data aggregation end are http://www.AnythingResearch.com and http://www.AggData.com (which has dozens of retail datasets just like the Walmart store example, presumably gathered by screen-scraping).

Data Marketplace can help people who are looking for data connect with people who are able to collect it. One of the problems with all of the data online is that it is in too raw a format to be useful. Someone must scrape/combine/clean the data before it can be analyzed. By connecting business people who maybe don't know how to scrape data with programmers/aggregators who specialize in it, we hope to make it a lot easier to find useful data.

Such a simple idea, but very compelling and nicely executed.

How is the pricing determined? Simply based on what the data seller/aggregator thinks he can get?

The "requesting for data" reminds me bit of a more structured mechanical turk.

Pricing is set by the provider, but we don't mind haggling. We're trying to build a community around buying and selling data tables, so if the price isn't right, that's good feedback for the seller.

I'm just about finished with a dissertation in Finance (teaching job starts in August). Obviously I'm very excited about this launch, I have a couple painfully acquired datasets I wouldn't mind seeing a dollar return from and it'd be great to save some effort going forward, especially after I have a salary to spend on that text box.

We'd love to help you get started! If you have any trouble adding your datasets, please send me an email at matt @ datamarketplace.com and I'll make sure they get added asap.

I would love the ability to browse available datasets without having to register. And I'd really love to be able to request a dataset without having to register.

Structured datasets are hard, and the hardest part is finding one that updates as time passes and new data is available - how are y'all handling that?

You should be able to browse the data that is available without registering by clicking on "Buy Data" in the header. We make people register when they request a dataset because we want to be able to let them know once we've found their data.

We're working on an API that will allow sellers to programmatically refresh their dataset to keep it up-to-date with the source. We're also working on a search and filtering system that ranks datasets based on when they were last updated.

Ah, OK, I thought that was just a representative sample since there were only five datasets.

I'd recommend you find, package, and sell other datasets that exist out there (Freebase, etc.).

I disagree with the registration requirement - if I want to be updated, make that a opt-in box - but I acknowledge that it's a justifiable product decision.

This is every grad student researcher's dream... data finding and cleaning is such a major hassle. If only I had this a couple years ago...

This is going to be great especially when the new Census comes out. Lots of data to be sifted through and organized in unique ways that the government never does for us.

Thanks for the feedback! We'll have to devote some resources towards reaching out to the academic population.

No problem. I think you can get A LOT of data from grad students. They hold the keys to a lot of it, and aren't held to some contract to never release it.

I'm sure there's an opportunity here for someone to hook this up with Amazon's mturk and provide an "automated" data provider ...

We're working on it :)

I would redesign the site to be more enterprise like than web-application-like. This is more b2b. I think I am going to buy or sell some data on this site.

Reply on news.ycombinator.com