Hacker News Reader

Comment by minimaxir | Hacker News Reader

As someone who used Kimono a few times before the closure...this announcement does not surprise me at all.

The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.

Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.

View on HN

"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?

Laziness is a secret form of technical debt.

For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.

While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).

See x1024's comment above. (best case scenario is that it's not significantly faster)

It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity:

An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.

What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable.

The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.

> "non-programmers can do it. No code required, etc. etc."

translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...

We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems."

You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.

I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.

Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...

DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.

I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.

Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:

http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...

Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?

Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/

If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com

Hi Max,

All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...

If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/

if you run HN front page through import.io's magic, it works pretty well: https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...

Another one for http://ourincrediblejourney.tumblr.com/

Reply on news.ycombinator.com