chrisohara

Good for some quick business ideas..

If you're a node.js user, install the command line JSON pretty printer `sudo npm install -g json` and then you can do this:

`curl -s random-inc5000.herokuapp.com | json`

I get simple scalability, server-client code reuse, easy realtime, and, despite what you or others may think, I like Javascript as a language. Node.JS was also a perfect fit for the scraping framework I wrote: https://github.com/chriso/node.io

At one stage I had the same train of thought that you have (http://chris6f.com/synchronous-nodejs / https://github.com/chriso/synchronous/blob/master/lib/protot...) - it would be nice to have the option of fibers, but it's not going to happen.

IMO async code isn't as difficult or ugly as you make it out to be. Is async code as easy to write and follow as sync code? No. Is it worth the benefits I've mentioned? For me, yes.

Only the minority gives a shit how many Hello World requests your stack can pump out. Some of us use node.js because it's a) got some decent libraries, b) makes realtime easy, c) uses the same code server+client-side reducing context-switches, and d) gets quite enjoyable once you know what you're doing.

You obviously haven't written a lot of async code so not sure why you're so against the idea.

the thinking that async code == callback spaghetti shit has to stop.

There are plenty of async idioms that make callbacks a breeze.

gplus.to

..although it's not run by Google.

Maybe they'll end up getting gpl.us

Serious question - anybody want to buy unfilter.com.au? I own it

how is this anything like express?

See klein (https://github.com/chriso/klein.php) for a real PHP express port..

Yeah definitely - I'm waiting patiently for the EVAL stuff to make it into the main branch. Very excited to see what's possible now that atomicity can be retained with higher-level structures

It's a higher-level abstraction. With most traditional clients you call commands like `hget(key)`, `zinterstore(key1, key2, key3)` or `mset(key1, value1, key2, value2)`

Redback gives you the same power/flexibility but hides the complexity. You call `hash = redback.createHash(hash_key)` and then call something like `hash.set({key1:value1, key2: value2});`

It's more than just convenience though. I had a few goals with the library:

- Make Redis more accessible

- Create some useful derivative/hybrid structures (SocialGraph, etc.)

- Expose useful atomic constructs such as auto-incrementing keys

- Make it as easy as possible for users to create their own structures (e.g. see the Queue example in the README)

It's not the default mode, but you can select JSDOM as an alternative parser and interact with / scrape pages as a headless browser

Yes, it's something I'm experimenting with at the moment. You can select JSDOM (https://github.com/tmpvar/jsdom) - which has the ability to handle JS - as an alternative parser. Set the following two options:

    jsdom: 1
    external_resources: 1

Watch this space: https://github.com/chriso/node.io/blob/master/lib/node.io/do...

Node.JS seemed like a perfect fit for a few reason:

1. JS selectors make scraping _very_ easy.

2. Asynchronous is fast as it is, but the page is actually parsed as it's received - contrast this with other scraping solutions where you need to download a page and parse it once it's complete.

3. With asynchronous scraping it's trivial to handle failures, timeouts, retries, nested requests, recursing similar URLs, concurrent requests, etc. - just add one of the many options (https://github.com/chriso/node.io/wiki/API---Job-Options)

Hey Ben, thanks for the write up on my framework. Firstly, the lack of documentation for more advanced scraping is something I plan on getting around to. You can incorporate proxies, scraping pages behind logins, etc. If anyone needs help in the mean time, send me a message on github.

The main thing I'd like to point out is that I built it primarily as a command line tool. "By implementing an input method there’s no way to specify a search term from the command line" - so leave/comment the input method out! The default input method is to read lines from STDIN, just like the default output method is to write to STDOUT

Try commenting out the input line on that google example and running it with a list of words in a file:

    node.io google_keywords < input.txt

Or you could feed the results in to another node.io job:

    cat input.txt | node.io google_keywords | node.io someotherjob

or mine: https://github.com/chriso/klein.php

Exactly what I was going for when I released klein.php (https://github.com/chriso/klein.php)

Sure, you could do the same thing with other frameworks, but most of the other frameworks are full of bloat and not particularly fast. I think it's better to have a few components that do one thing really well rather than a monolithic framework that tries to handle every aspect.

nice seeing my lib (validator) in use. Also, damn CoffeeScript is sexy

The one liner is cool, but I guarantee that node.js's non-blocking IO will outperform perl any day of the week. Try scraping thousands of pages at once using perl..

Hey David, it wasn't a stab at your blog - your post was great - anything that builds some more interest in node.js is positive :) I just hate reading about people having trouble with JSDom and putting it down to the node platform. JSDom is an excellent parser, it just fails miserably when you feed it malformed HTML, and as we know, a majority of the internet falls in to this category. I needed a framework that could scrape anything on the web so I built it myself

Hi Simon, great libs - thanks! there's been many improvements added to node-soupselect and node.io though - the API is here: https://github.com/chriso/node.io/wiki/API---CSS-Selectors-a...

Interesting, thanks. I've been looking for a decent C lib to bind with node and play around with

This is in response to all the node/jsdom/jquery scraping posts that are popular lately. JSDom is hopeless for scraping - try parsing some slightly malformed HTML..

I suggest taking a look at node.io (https://github.com/chriso/node.io) - a scraping framework written for NodeJS. It uses htmlParser rather than jsdom and scales nicely. It also has support for handling timeouts, retries, etc.

care to enlighten us with what you deduce to be advanced JavaScript?

Willy Wang