Hacker News Reader

Comment by DanBlake | Hacker News Reader

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

View on HN

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data?

of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?

The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly.

The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension.

The route of proxying the website does in fact do away with the need to install any external 3rd libraries.

This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well.

https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...

Hah, I've been working on this recently with Facebook, on a TV set-top-box. It was painful and I ended up giving up. xd_arbiter.php is the key, I think.

I'm wondering how you will be able to with the numerous ways of CSRF protection implementations.

I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.

We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free

Glad to hear it. I was just saying that I think auth should be included as a basic feature in all paid for plans.

Creator of Automately here, our service could definitely be something you might be interested in. While we aren't directly in the business of web scraping, we do have a powerful automation service that can accomplish those needs using simple javascript and our powerful scalable automation API.

We are accepting early access requests right now. Check us out! http://automate.ly/

Reply on news.ycombinator.com