Comment by sph | Hacker News Reader

Comment by sph | Hacker News Reader

Parent Full thread

Care to link any of them, please? I'm not sure which you refer to.

I plan on writing a long post on my experience writing a web crawler in Elixir, and I believe it is a harder problem than many—including me one year ago—think it is. Web servers out there are just broken, pages have invalid HTML that somehow browsers are able to interpret, etc.

Here's some things Bernard does that many other don't:

* I respect robots.txt for each link I visit, and keep a single connection per {host,port} not to hammer popular domains. Maybe not ideal speed-wise, but I try to be a good Internet citizen.

* Modern MITM-like services such as Cloudflare, Akamai, etc. need to be accounted for, as they might return 403s even if the link is working perfectly. I am planning on registering as a "Cloudflare good bot" very soon, right now I simply ignore and log 403s for Cloudflare-server domains. The goal ultimately is to have as many false negatives as possible.

* In the future, Bernard will be able to parse CSS and JS files, to find and test links to @font-face URLs, imported JS scripts, etc.

* If there is enough demand, as it is quite involved infrastructure-wise, I might consider adding support for client-side only websites, through headless Chrome (on a separate, more expensive tier).

* Not shipped in this MVP, the original goal of Bernard is to remember all the links its seen, so it is able to notify you if one 404s because you changed its URL, moved it around, and forgot to set up a redirect. This happens far too often in my experience, breaking bookmarks and causing SEO issues.

And last, but not least, none of the alternative I have tried, freeware or FOSS, felt good, correct or well-designed enough to use, from my point of view. I do believe there is always a space to do better than the status quo.

> If there is enough demand, as it is quite involved infrastructure-wise, I might consider adding support for client-side only websites, through headless Chrome (on a separate, more expensive tier).

I respect your experience on this project, but a suite (not just Chrome) of headless browser tests checking for broken links, amongst many other things, seems like not only the most sane way to approach this, but something that has been done to death on most serious web dev projects at $corporate_workplace. Obviously these are not open source projects so everyone else has to reinvent the wheel.

Parsing HTTP responses in your favorite language for tasks like this has never been reliable at any time in the entire history of web development. The myth that HTTP alone is all you need to understand the behavior of a website has always been wrong because it's far outside the scope of the spec. All of the link checkers in these comments seem like they'd be unreliable piles of edge case handling because the authors insist they should be able do it all from a Go or Python script or whatever.

I'm having a difficult time understanding why someone would pay for this or even want to run this anywhere but locally. Other than some individual's blog or affiliate marketing website I think most projects with a ton of links to check are already well supported by teams of devs who write tests like this all the time, and small time individuals with blogs wouldn't want to pay for yet another relatively expensive 3rd-party service.

Doesn't match my experience.

Yes, in theory you would expect everyone to check their links. In practice even my non-tech mother knows what a 404 error is.

My initial target audience are web agencies, that either operate in a ship-ASAP-and-fix-problems-later way, or otherwise allow their non-technical people to create pages and upload images. Mistakes happen often, links break, and might stay broken for months before someone notices.

I haven't worked with Fortune 500 companies, so I don't know how they operate at that level, but pretty much no small/med company, including web agencies, has an automated stack of headless browsers checking for dead links. For most people, dead link testing is done once in a blue moon through the abysmal interface of Google Analytics.

EDIT: you added the paragraph about parsing HTTP responses after I have answered, so I wonder if we're talking past each other. This is not a simple bash script exactly BECAUSE the web is full of inconsistencies and broken servers, I am well aware of it. My bet is that someone wants to pay me to deal with all that nonsense, instead of using... I'm not even sure what is the state-of-the-art FOSS links checker. I wrote Bernard because I found none that was good enough.

I have a large directory uses affiliate links. I’ve been running it for 1 year now and one of the biggest issues is affiliates changing their links. I was planning on building a basic link checker. I already have a script that is scraping and testing demo urls and gathering screenshots so it would make sense to just extend that. Bernard seems like what I need here but I’m not sure it fits neatly into my stack and I don’t think I would pay for it. If I didn’t already have the script and I was using Wordpress or some no code tool to build the directory then I feel I’d be more likely to use this service

Here is the one I wrote for myself a few years back https://github.com/ashishb/outbound-link-checker

Amusingly, the first image link on that README is broken.

I have not updated code in a while and the GitHub Action's last execution expired, so, the status information was not available.

I just did a new run of the test and the status information is now available.

Reply on news.ycombinator.com