Ask HN: Web crawler/data product ideas? | Hacker News Reader

Ask HN: Web crawler/data product ideas?

Is there a data set you would love to have but you just don't have time to compile?

Is there a data set that you currently compile manually that you can't live without?

Would it make your life easier if you could avoid opening all those browser tabs every day/week/month and instead purchase a neat file (csv/google spreadsheet/something else?) with everything you need? Which browser tabs would this eliminate?

View on HN

Probably not what you are looking for, but I would love to see more real-time data curation - I would certainly be willing to hand over cash for some decently priced real time data feeds - stock tickers, sports events, raw traffic data, headlines, currency, flights - many of these are available in pull formats, but it would be great to get a push feed.

Any good examples of a push feed you currently like?

Twitters streaming API used to work fairly well when I was playing with it. https://dev.twitter.com/docs/streaming-apis that was a few years ago though.

one big market for you is researchers. if only i had a penny for each time some economist came to me asking for a list of xxx in yyy, where x could be anything from flight prices, to businesses, to startups and yyy could be city / state/ coutnry /world. The internet is really good for current snapshots, but what reserachers most often need is repeated snapshots of the same thing. that helps us analyze how things change over itme. if you're a data company, please, please dont overwrite your tables to "update". save the old data. its really really useful. for research. and more importantly, for you.

Thanks. Will definitely look deeper into this. Know of any good sites where researchers make such requests?

Have you looked at http://enigma.io/? They won TC Disrupt last year

Priconomic just refocus to this business model

Priceonomics looks like more of a professional service at the moment. I'd like to sell actual data sets in more of an ecommerce/subscription model. Kind of like http://www.cbinsights.com/datastore/

Reply on news.ycombinator.com