Comment by shavenwarthog2 | Hacker News Reader

Comment by shavenwarthog2 | Hacker News Reader

Parent Full thread

It's hard to tell -- I tweaked it quite a bit, and rewrote it from scratch 2-3x. I'd say 4-10 hours.

Was it worth it? As a programmer, no. I'm very familiar with scraping (raw) web/RSS feeds for data, then processing it. I was hoping Pipes would have enough intelligence, so that I could subscribe to (cooked) data sources, then split and refine the results.

In practice, Pipes worked, but the data always required further post-processing, which was awkward to do in Pipes. You have to be a dev to understand what your system is doing, but you don't have easy access to all the standard dev things.

I look forward to seeing Pipes take off, or another technology (Huginn? Ifttt?) replace it. It was a lot of fun to wire things up graphically then for example get a text when someone's RSS feed changed.

  You have to be a dev to understand what your system is doing, but you don't have
  easy access to all the standard dev things.

Interesting, this mismatch may be a good description of the problem of visual languages.

Curious: what do you think is the minimal subset of unix tools to do this? i.e. instead of pretending the problem is simpler than it is, accept the complexity, but minimize it.

I'm thinking of a tool like "jq" (sed for json) for json data sources... but I don't think its raw-text manipulation is up to the task (and of course you need tools to monitor the feeds etc).

Trying to manipulate structured data as text makes about as much sense as parsing XML with a regex.

http://stackoverflow.com/questions/1732348/regex-match-open-...

Python :-) there are libraries specifically for parsing malformed html. I'm happy using Unix tools for scraping and parsing, but you run into a brick wall rather quickly. Python is more reliable, flexible, and easier to integrate.

Reply on news.ycombinator.com