Show HN: Distributed Scraper

hah, the C/C++ programmer in me cringed the first time I saw it. It's a decent platform though.

Related story: I own stdlib.net, and have been using it for some e-mail and stuff for more than ten years. The original reason for getting it was that we were playing around with using the net.stdlib Java namespace for Apache commons. But too many people had the same reaction as you. So everything is org.apache...

Founder here; as much as it pains many of us to hear, the vast majority of software developers never interface with <stdlib.h> directly. You can think of our choice of name more as an homage for a new generation of web-based developers.

The question we asked ourselves was --- if I'm a developer today, and I want to release services for people to easily use and compose, take advantage of new "serverless" technology, where do I put it and how do I distribute it? How do I keep my services organized? Can I build a business around an API? What does that look like?

The name "Standard Library" was a natural fit --- I consider it a huge amount of serendipity that the domain was available. We've also been lucky to have a huge amount of support from our developer community --- we got a few questions like this to begin with, which naturally caused some unease, but far more positive reactions in total. I'm very happy we've been able to build something people like. The name choice has not negatively impacted our SEO or anything like that, thankfully! :)

If you happen to play with it at all, let me know what you think!

(Humorous aside; it is more frequent for newer developers to ask us about "sexually transmitted diseases" than it is for them to get the C reference - which is totally fine, don't mind sharing knowledge! But when I told my non-CS mother about the business we were building, definitely had to really quickly explain the background of the name.)

This name drives me a little nuts, too. The modules are named `lib`. Good luck grepping that!

I have tried https://dexi.io/ (free account) about a year ago and it looks very good. One downside is that it was issuing requests from Europe by default, so you had to bring your own proxy for the US, for example.

What output are you looking for? IIRC, Kimono was about creating REST apis — are those sufficiently non technical?

I tend to tell this to people quite often, but depending on the use-case – it might actually make sense to use something like Mechanical Turk to get complicated scraping done.

But if the content is pretty neatly and easily scraped then the other recommendations make sense.

Though a bit of Javascript can now take you a long way ;)

Not sure what the UI of kimono looked like but with this one, you can more or less create one for it if you wanted to. (or ask someone to do it)

* shameless plug * Feedity - https://feedity.com

I guess it's not truly distributed in that sense. StdLib uses AWS Lambda, which have widely different IPs and I believe they're multi-region.

I haven't had issues hitting a wall with getting caught doing any scraping. But then again, I haven't done it at a 10k/pages/sec rate or anything like that.