Hacker News Reader

I'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to take off?

I could have really used a service like this just yesterday actually, I ended up fiddling around with iMacros and got about 80% of what I was trying to achieve.

View on HN

It's a great question. What we're really trying to do is make data accessible programmatically and at scale. We want to connect data providers and data consumers with APIs in a way that's mutually beneficial vs. being a tool for data theft. Our hope is to (once we scale) actually work with data providers directly on the on the distribution of their data so the IP ban becomes a non-issue.

But isn't the point kinda to let the users come up with data providers themselves? If you say "Only these 500 data providers are available for scraping", you don't have a business. If you don't' have such a limitation, you'll not be able to work directly with all data providers. You'll have IP problems.

Reply on news.ycombinator.com