Introducing the Priceonomics Business Model: Data Crawling Services
priceonomics.com
priceonomics.com
This announcement ties it all together. Philosophically, it's less important what actual products/services they are outputting, but that they're sticking to their original mission -- discovering the best price for a commodity and sharing it with the world.
While we're on the subject of philosophy and prices, I really liked this ribbonfarm article about bartering that seems relevant, I think you will enjoy it:
http://www.ribbonfarm.com/2008/03/16/bargaining-with-your-ri...
I've done a bit of crawling and have always been curious how selling this sort of service would handle those potential use restrictions / legal issues.
But I spoke to a lawyer and she strongly advised against it (she said if the Terms of Service of a site say it isn't allowed, you have to follow that).
Dunno. Confuses me how something is supposedly illegal yet many companies do all the time without a concern.
I've written "Is scraping legal?" here, with details, and my own view morally. https://blog.scraperwiki.com/2012/04/is-scraping-legal/
In your original business model you wanted to understand the price of everything. In what ways did the problem of a lack of information on the demand side come up? That is, it is easy to scrape the price in many markets (supply side), but what kinds of conversations came up within your team about the lack of information on how many units were actually sold at a posted price?
By the way, glad to see you guys were able to make a business out of crawling. I've landed a handful of freelance gigs since leaving grad school based on scraping data for clients, but never tried to expand it to anything beyond consulting projects.
Without having access to the actual monetary transaction data, how does one know what was sold and for how much? Without this (or a mechanism by which the lister closes or updates the listing), how do you know anything was actually sold?
For example with domain names sales prices only a small amount of transactions are public. For example I've been doing it for 16 or 17 years and have never made any of my data public nor have people that I've consulted for.
Another example might be commercial rents. You can track asking rents but you can't really get a handle on actual rent paid since there are many deal factors (renovations, free rent, triple net etc.) that would change the numbers significantly.
In terms of a few real-life examples, on the one hand you have eBay which provides you with sold data (API through Terapeak). On the other hand you have Craigslist, which is kinda opaque, hates scraping, but you can monitor listings and their half-life. (Listings that disappear quickly presumably get sold quick; listings that stick around for weeks relisted over and over have lower liquidity presumably and/or are priced high.)
As you mentioned, monitoring half-life is another imperfect approach, but it is of course plagued by false positives (a listing goes away but no sale was made). There was a Google Tech Talk many years ago where some economists took this approach[2], except they were looking at pricing power instead of measuring quantity sold.
[1] http://www.hss.caltech.edu/~mshum/ec106/chevaliergoolsbee.pd...
Most websites don't relay this info to the end user but it could be used on those that do.
I guess with Import.io if something breaks, you will have to redo the scraping logic. Maybe PE will manage that on the users behalf and that's their value add? Looks like PE may be doing some intelligent scraping as well (not everyone is going to list Dell LCD Monitors $400 in their title but they will do the normalization for you -- just a guess).
I have used import.io and it works very well for most sites and it is fairly easy to create a scraper.
I think vertical specific ecommerce pricing data like what these guys are doing is a hard problem ( I have clients in this space as well) and it's great to specialize in something like this. I think the real value comes from an end to end service like what these guys are offering.
Current projects for clients include custom sentiment analysis engines, real time event streams, ad intel type projects, location data, and even menus.
Much of this is done with deep learning and more generalizable models that are likely to fit your domain.
I have a dashboard I'm working on with some customers now that will allow you to avoid having to contact a service provider.
Email is in profile if there's any requests. I hope to put up a good portfolio site soon. Only started this a few months after building out my own text analytics engine.
Also again: best of luck to priceonomics here. It's a lucrative market if done right.
"We’re building the price guide for everything, from bicycles to boats, Aeron chairs to iPads. We hope as you look to buy or sell things you can use our data to get a good deal (or at least prevent yourself from getting ripped off). For example Priceonomics can show you how much a used iPad 2 3G should cost, as well as a whole range of prices"
[0] http://blog.priceonomics.com/post/14567999429/how-to-use-pri...
Well, we're trying to build a company that one day has hundreds of writers and engineers working together. So, we've been writing all along and want to do more of it for sure.