Abusing Ahrefs backlink checker | Hacker News Reader

Abusing Ahrefs backlink checker

trickster.dev

57 pointsrl198720 comments

The main point of this article is that you can pay humans to solve CAPTCHAs, via a company which acts as middle-man.

The company in question (Kolotibablo) has a website at https://kolostories.com/ which argues that paid CAPTCHA-solving is a good economic option for the people who do it. Maybe it is? I can't decide.

You can make the text a bit more legible with this userstyle:

    body {
        --fg: white;
        font-family: sans-serif;
    }

Do Ahrefs et al operate their own crawling infrastructure or is there another third party vendor for this?

Ahrefs operates their own crawler, and gets really valuable data from it.

Source: My server logs.

They're really bad at purging outdated search results, I often see them trying to GET pages that haven't existed in 5+ years.

It isn't just a search engine. It's also a link monitor. Someone somewhere is linking those pages. It might not be a current user, but they still crawl it in case the person with those bad links signs up.

They do. And they have their own search engine, too - Yep.com

They have some info on their Data[0] page.

I always saw them as - if Google Search had a public back-end UI, then Ahrefs would be it.

[0]: https://ahrefs.com/big-data

They do: https://ocaml.org/success-stories/peta-byte-scale-web-crawle...

I wonder if there's some architectural reason for using `["Status", {}]` shapes instead of a more traditional response shape

    [
        "Ok",
        {
            "data": {

It seems to place a lot of the error detection burden upon the caller, since the response can be empty `[]`, it can have a status without a body (which, I guess may be fine for their api?) `["OkHarHar"]`, and then the usual promises-made-promises-kept of the expected keys being in the `response[1]` location. One can see this heartache sort of play out in the subsequent python code in the blog post although they just return None in the simple `len() != 2` case

With pattern matching it'd look like:

    match (response) {
      ["Ok", { data }] => { ... },
      ["Error", { error }] => { ...handle known error... }
      _ => { ...handle unknown error... },
    }

It could also just be auto-generated serialization/deserialization code like in Rust.

I guess that explains why Turnstile has been as useful as an ashtray on a motorbike for stopping automated signups for me.

Ahrefs was using recaptcha until recently, I wonder if that was abused in the same way or whatever the reason for them switching up was.

Recaptcha could also be bypassed in the same way FWIW.

Is there any captcha that can’t? I presume LinkedIn captcha

That's Arkose Labs and 2Captcha, Anti-captcha and Nopecha have bypasses for them.

I wonder if it's cheaper to just pay for a subscription and get access to their API - given that solving captcha cost $1-2.

$1 - $2 per thousand queries.

oh, good observation

Looks like API access is only available at $1k/mo.: https://ahrefs.com/pricing

Hmm is this still cheaper than purchasing the service?

Reply on news.ycombinator.com