Hacker News Reader

I just tested it and i like it : i have 46 broken links, certainly due to the latest changes i made. I know i would have the broken links specifically in the blog part, but was procrastinating to check them. GOD JOB, I LIKE IT

Now, regarding the pricing, i really don't like subscriptions, we already deal with a lot of subscriptions every month, but it's up to you.

I would suggest also to give ability to pay once for checking the links. So, when i come back maybe in 6 months later, i will again pay one, and so on, etc...

I would also like to download the link in an HTML page and consult them or do whatever stuff i like, for example, scrapping my own HTML to find broken link and do some action in development server or robot testing, but downloading shouldn't be free to

But it's nice, keep going and find your perfect market

Re: subscriptions—I hear you. They are certainly simpler for a business to deal with, but I will consider the possibility of being able to top up a plan at your own convenience.

Exporting the list in CSV format is planned in the very near future, along with better filtering tools, search and ignore patterns.

Thanks for taking the time to trying it out :)

you're welcome

I considered building something like this an year or to ago, and was put off by a few things - marketing, the need to deal with Cloudflare and the like (a lot of potential customer's websites re entirely behind services that block bots) and all the things you mention too.

I really like your blog and find it encouraging with regard to other ideas I am thinking about.

Really surprised Cloudflare doesn't have this built in.

Interesting, I helped someone with this recently. I did it with a github action and an open-source link checker, something like this:

  run: |
    yarn export:start & 
    sleep 5
    wget https://github.com/filiph/linkcheck/releases/download/3.0.0/linkcheck-3.0.0-linux-x64.tar.gz && tar xvf linkcheck-3.0.0-linux-x64.tar.gz
    linkcheck/linkcheck -d -e --skip-file .skip :3111 || export RETURN=$?
    killall node
    if [[ "$RETURN" -eq "0" ]]; then
      exit;
    else 
      exit "$RETURN"
    fi

It would be useful to list IP ranges that the scans come from -- a lot of my web stuff has "hosting providers" firewall dropped because so much malicious traffic emanates from the hosting providers (Do you hear me, Digital Ocean and AWS?)

Good idea. For now, I advertise as "User-agent: bernard/1.0", but I might create a .well-known URL listing all the IPs we scan from. I already need to do that for Cloudflare.

This is nice!

Would love to know whether found a need to define any custom logic for 'international' websites?

For example a lot of global companies will have

www.theirdomain.com/example-page

But then also have

www.theirdomain.com/us/example-page www.theirdomain.com/au/example-page www.theirdomain.com/nz/example-page etc...

If you're crawling from one regional or the 'global' entry point you might not always find all of the regional variations of pages. Sometimes the site might redirect you to your previous region if you try to change.

A common tool SEO teams use is screaming frog which offers similar features, crazy that a web service isn't more commonly used for this kind of thing.

That's a good question. Right now you can only enter root domains, but eventually I want to relax it and you'd be able to enter `example.com/en` and `example.com/au` separately.

If the webserver redirects based on IP geolocation, it is planned way down the line to have a few Bernard workers spread around the globe, but I'll be honest, it's not my highest priority right now unless this is a common enough issue.

Interesting, probably a good example of that last 2% of a product that could easily take 80% of your time.

Big enterprises that might want this kind of thing are probably also the types of people that are least likely to use this kind of tool though.

I gave this a shot and was shown

> We have analysed 68 links on your website and haven't found a single issue.

I'd love to see what this actual list of links was, even if they weren't broken.

I'm partially curious because I personally use lychee [1] for this purpose, but when I run lychee (even while ignoring several classes of links), it is checking _significantly_ more links. E.g., the output looks like:

> 1401 Total (in 7s) 1361 OK 4 Errors 35 Excluded

Curious to understand what it is checking that Bernard is not.

[1]: https://github.com/lycheeverse/lychee

What is the URL are you testing with? If you want also share the /demo URL and I'll check what is going on.

Seems like lychee also extracts links from text, Bernard does not. Only links inside HTML tags are considered, text nodes are ignored.

https://bernard.app/demo/33b26fe3-6c67-4338-a6cd-dd4e9204fbf...

I also suspect a different is when I invoke lychee, I'm running it on ALL of the *.html files output by my static site generator. Perhaps Bernard is only testing a single page?

If I only run lychee on the same single page, the numbers are closer:

> 76 Total (in 1s) 73 OK 0 Errors 2 Excluded

No, it follows every link recursively, as long as they are part of the same domain.

So, if you add example.com, the crawl starts at / and all links found therein are added to the queue, and so on. For convenience www.example.com is also considered within crawl scope (and viceversa). Everything else is treated as an external link, so tested but not parsed.

Is it possible that your links are hosted on another subdomain than the one you have entered? I'll do a deeper dive into this issue tomorrow.

Maybe not all pages are reachable from the root.

Cool!

> I'll do a deeper dive into this issue tomorrow.

FWIW, I don't think I'd go as far as calling this an issue, I suspect both Bernard and Lychee are doing different, but reasonable things.

No need to spend much time on my account, and cheers on launching a useful project!

There's why Bernard isn't finding as many links: a sample one from /blog

    <a href="https:&#x2F;&#x2F;luke.hsiao.dev&#x2F;blog&#x2F;2023-workstation-build&#x2F;">My 2023 Workstation Build</a>

I have never seen forward slashes encoded that way. These were simply not recognised. Added to the todo list.

Huh, me neither. Not an intentional encoding on my part. Looks like I have a bug to fix as well!

This is cool! I recently worked on integrating something like this into our blogging platform [0] to help bloggers monitor their links automatically. One main problem is with popular websites which have pretty aggressive bot prevention mechanisms. They often return 5xx codes even in HEAD requests. How do you combat that?

[0] https://blogs.hyvor.com

I had to write a piece of logic that sends a GET when HEAD fails, because of a non-HTTP compliant servers. HEAD should in theory return the same status as a GET, but without any body. In practice, many web servers return 404, sometimes 500. HN itself returns 405 Method Not Allowed, which makes some sense, I guess.

The code in question, simplified:

    defp try_get_request?(%Link{method: :head}, %StatusResponse{} = response) do
      # Per HTTP specification, the HEAD response should be functionally equivalent to a
      # GET, but shall not contain any body.
      # Not all servers respect this, so might have a different status response on HEAD than on GET.
      #
      # We assume that some HTTP status codes are suspicious and worth retrying.
      #
      # HTTP 520 is seen with Cloudflare to mean "Web Server Returned an Unknown Error", possibly
      # in response to a HEAD request.
   
      response.code in [403, 404, 405, 520]
   end

There's a bit of a "bug" I found where Bernard interprets backslashes in file paths literally and URL-encodes them as %5C, whereas modern browsers automatically correct them to /. You could definitely argue that the URL is specified wrong (assuming that / was indeed intended) but it causes inconsistencies between Bernard and an actual user.

Can you share an example, please?

I've had a lot of URL encoding-related issues in the past 24 hours, due to Elixir's URI library being a little too strict, and a lot of websites having broken links that browsers are able to work around and interpret just fine. I reckon I will have to spend my weekend writing a bespoke URL encoder/decoder.

Sure, the images on this page are where I noticed it: https://chainswordcs.com/e-reader-protos.html

No doubt URL encoding is frustrating, good luck taking it on!

Yeah, the paths on that page are encoded with a backslash, which browsers seem to correct to a forward slash, as they should be.

I'll add those in my test suite and fix them soon, thanks. Indeed Bernard should be able to replicate the behaviour of any browser, however broken the links are. If it works in your browser, it should work in Bernard.

BTW, even curl seems to have issues with those links:

    > GET /e-reader-protos\March-7-2001-Press-Conference-photo-1.jpg HTTP/1.1
    > Host: chainswordcs.com
    > User-Agent: curl/8.2.1
    > Accept: */*
    >
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * old SSL session ID is stale, removing
    < HTTP/1.1 404 Not Found

Tested on various open-source related websites, and while it is usually correct. I found that for lichess.org all dead links were false positives. Spurious connection when scrapping or block from the website, I don't know.

I'll look into it, thanks!

Can you call out what makes your product different from other similar commercial offerings?

https://www.deadlinkchecker.com

Even the pricing models seem similar.

I do not like to disparage competitors, but the issue is that deadlinkchecker.com has far too many false positives and issues that I try hard to avoid or workaround in Bernard.

For example, it marks a URL that's a form action destination (https://bernard.app/demo) as a 404. I technically should return 405 on GET, but it is a false positive nonetheless, because never affects any user.

My suggestion is to try both and see what works better for your website.

I hit the 2,500 limit and has found 316 issues. I'm not sure if I'm interesting in fixing them. It is my personal website and I'm not too concerned about it. I would love to try it out and I'm OK if I can check less than 100 links a week for free, so I can fix leisurely over the next few years, or decades.

I know Google webmaster (did they changed their name!) already has the info but it is not easy to use and is irritating to navigate around.

Anyways, signing up and best of luck to your product.

Does anyone know a good open source tool for checking broken links? I've used get/curl with scripts but it's fragile.

https://github.com/spatie/http-status-check

I am using Linkchecker: https://github.com/linkchecker/linkchecker/

Unfortunately, many web servers will block bots, either immediately or after a few requests.

I still haven't found a good modern equivalent to http://home.snafu.de/tilman/xenulink.html

i like it and would love to replace our existing link checker with something better at some point. a killer feature would be the ability to notify specific individuals by email when their content has broken links detected. love the dog favicon, too. our government entity overlords did not bat an eye when budgeting $2000 USD per year for a service like this.

The dog is supposed to be a St. Bernard :) I want to commission a proper icon, but it's a bit further down my todo list.

Feel free to contact me if you might be interested in test driving it directly or through a REST API, which I'm not sure if there is any demand for.

Just signed up, this is awesome! Ran it over my blog and found 8 broken links.

One light bug: links to HN stories are marked as 403 forbidden (I guess the server blocks the crawler)

For the free tier: honestly I think you can charge even for small blog owners. I would pay a small yearly fee to make sure links are healthy

Yes, I know. I have a task in my TODO list to to check whether the 403 comes from HN, and just ignore it. Thankfully, news.ycombinator.com links never seem to break :)

going to keep and eye on this. maybe I can integrate it in my own service https://montaigne.io.

I saw many broken links on users’ sites and wanted to do this myself, but didn’t have time to implement.

There's a recursion issue somewhere.

1: https://bernard.app/demo/8b13ab64-2440-4520-be05-905b5e2188f...

I like the product and I would use it.

https://img.devrant.com/devrant/rant/r_268897_Geemo.jpg

Thanks for the bug report :) to be frank, there is actually a recursive link in your website (that "hooked on computer" link towards the top of https://keeb.dev/2022/05/28/The-beginning/)

Recursive links are not exactly uncommon - there is little reason to unlink the current page if it is a standard item in menus or other navigation on the site.

There is a difference between a having a link to "/page" and a having relative link to "page".

The first is fine, the second needs special logic to tell that when we reach "/page/page/[...]/page" we should probably stop. I bet $5 that their web server log shows plenty of naive bots like mine getting lost in that recursion, burning unnecessary CPU time.

Thankfully, the solution is not that hard to implement.

I'm glad the fix is easy, but the web has always been notorious for garbage HTML. You'll go farther with this if you adopt an attitude that you can handle low-quality content vs. getting defensive about someone finding a new edge case that needs to be handled.

Not sure why you think I was being defensive, nor annoyed at the problem. I just wanted to clarify for sake of argument.

It is on my todo list to fix. Thanks.

Thanks for finding the bug in my site. I appreciate it! I have fixed it

What's the advantage over FOSS broken link checkers? Some even have a pretty good GUI.

Care to link any of them, please? I'm not sure which you refer to.

I plan on writing a long post on my experience writing a web crawler in Elixir, and I believe it is a harder problem than many—including me one year ago—think it is. Web servers out there are just broken, pages have invalid HTML that somehow browsers are able to interpret, etc.

Here's some things Bernard does that many other don't:

* I respect robots.txt for each link I visit, and keep a single connection per {host,port} not to hammer popular domains. Maybe not ideal speed-wise, but I try to be a good Internet citizen.

* Modern MITM-like services such as Cloudflare, Akamai, etc. need to be accounted for, as they might return 403s even if the link is working perfectly. I am planning on registering as a "Cloudflare good bot" very soon, right now I simply ignore and log 403s for Cloudflare-server domains. The goal ultimately is to have as many false negatives as possible.

* In the future, Bernard will be able to parse CSS and JS files, to find and test links to @font-face URLs, imported JS scripts, etc.

* If there is enough demand, as it is quite involved infrastructure-wise, I might consider adding support for client-side only websites, through headless Chrome (on a separate, more expensive tier).

* Not shipped in this MVP, the original goal of Bernard is to remember all the links its seen, so it is able to notify you if one 404s because you changed its URL, moved it around, and forgot to set up a redirect. This happens far too often in my experience, breaking bookmarks and causing SEO issues.

And last, but not least, none of the alternative I have tried, freeware or FOSS, felt good, correct or well-designed enough to use, from my point of view. I do believe there is always a space to do better than the status quo.

Here is the one I wrote for myself a few years back https://github.com/ashishb/outbound-link-checker

Amusingly, the first image link on that README is broken.

I have not updated code in a while and the GitHub Action's last execution expired, so, the status information was not available.

I just did a new run of the test and the status information is now available.

> If there is enough demand, as it is quite involved infrastructure-wise, I might consider adding support for client-side only websites, through headless Chrome (on a separate, more expensive tier).

I respect your experience on this project, but a suite (not just Chrome) of headless browser tests checking for broken links, amongst many other things, seems like not only the most sane way to approach this, but something that has been done to death on most serious web dev projects at $corporate_workplace. Obviously these are not open source projects so everyone else has to reinvent the wheel.

Parsing HTTP responses in your favorite language for tasks like this has never been reliable at any time in the entire history of web development. The myth that HTTP alone is all you need to understand the behavior of a website has always been wrong because it's far outside the scope of the spec. All of the link checkers in these comments seem like they'd be unreliable piles of edge case handling because the authors insist they should be able do it all from a Go or Python script or whatever.

I'm having a difficult time understanding why someone would pay for this or even want to run this anywhere but locally. Other than some individual's blog or affiliate marketing website I think most projects with a ton of links to check are already well supported by teams of devs who write tests like this all the time, and small time individuals with blogs wouldn't want to pay for yet another relatively expensive 3rd-party service.

I have a large directory uses affiliate links. I’ve been running it for 1 year now and one of the biggest issues is affiliates changing their links. I was planning on building a basic link checker. I already have a script that is scraping and testing demo urls and gathering screenshots so it would make sense to just extend that. Bernard seems like what I need here but I’m not sure it fits neatly into my stack and I don’t think I would pay for it. If I didn’t already have the script and I was using Wordpress or some no code tool to build the directory then I feel I’d be more likely to use this service

Doesn't match my experience.

Yes, in theory you would expect everyone to check their links. In practice even my non-tech mother knows what a 404 error is.

My initial target audience are web agencies, that either operate in a ship-ASAP-and-fix-problems-later way, or otherwise allow their non-technical people to create pages and upload images. Mistakes happen often, links break, and might stay broken for months before someone notices.

I haven't worked with Fortune 500 companies, so I don't know how they operate at that level, but pretty much no small/med company, including web agencies, has an automated stack of headless browsers checking for dead links. For most people, dead link testing is done once in a blue moon through the abysmal interface of Google Analytics.

EDIT: you added the paragraph about parsing HTTP responses after I have answered, so I wonder if we're talking past each other. This is not a simple bash script exactly BECAUSE the web is full of inconsistencies and broken servers, I am well aware of it. My bet is that someone wants to pay me to deal with all that nonsense, instead of using... I'm not even sure what is the state-of-the-art FOSS links checker. I wrote Bernard because I found none that was good enough.

Isn't this just like a 50 line python script to request all the anchor links and if any response is not 200 store in a file somewhere. then recursively do the same thing on wherever the working links lead ?

A large number of useful apps/services could be described as “just a 50 line Python script”, but the devil is usually in the details. Spending a reasonable amount of money to not worry about those details is why a lot of us are employed.

This isnt one of what you described though.

I've literally done this before.

This is an extremely simple product that could be written in a morning by someone competent.

> by someone competent

Which is why there will always be a market for products like this. The point is that someone will pay a developer (someone competent) for software that just solves the problem for them.

Plenty of people run websites and have no idea how to even begin something like this.

The curse of knowledge we have in the bubble we're in makes these things appear simpler than they are to most people.

It's a bit rude to assume I am incompetent.

I am sorry, but you vastly underestimate the problem. Just read the other comments and see how many corner cases are there. It took me 3 days to implement a crawler than says OK on 200, and ERROR on 404, and 6 months to deal with exceptions to the rule, and apparently thanks to the feedback Ive received, there are many other exceptions to deal with.

Sorry, this is a bit like that old Dropbox comment. The thing is, if your website is easily covered by a 50-line Python script, you do not need Bernard.

http is just a very simple request response protocol.

maybe you're right though

One of my websites has links to nytimes.com. They work fine if clicked on manually. Bernard reports them as a 403. I wonder if NYT is classifying Bernard as a scraper?

I'm also seeing valid links to reuters.com coming up as 401 unauthorized

The Internet is a wild place, and I reckon 90% of the complexity of a crawler is dealing with workarounds and non-compliant servers (cough www.apple.com cough).

I'll have a look, thanks for the heads up.

are you setting the headers to make the sites think it's a browser??

edit: User-agent: bernard/1.0"

I bet thats going to cause issuses.

Id fake a browser user agent for off domain sites.

“Error, only full domain and subdomain are supported” when trying to run on a specific path of my website. Why is that?

Testing a sub-path is not supported for the time being, but will be in the near future. I had to limit the scope of this MVP to ship in time.

Our company site has a very strict anti-scraping Cloudflare mechanism. Will your crawler be able to work around it?

Which means that requests to your website might randomly return HTTP 403. At the moment, this is such a common issue I just ignore HTTP 403 errors when there's a Server: Cloudflare header (and a handful of others).

That said, Cloudflare has a process to register as a "good bot" [1], which I believe we are (we respect robots.txt, keep a single connection alive per host, etc.), so it's possible that once I set that up, the gods that control the Internet might let us through.

In general though, Bernard follows the HTTP spec but cannot do miracles. If your webserver is trigger-happy or very strict, it is your responsibility to let us through if you want to use the service.

1: https://radar.cloudflare.com/traffic/verified-bots

For fun: run your local government's site through this.

Good list of free and open source link checkers here: https://www.devopsschool.com/blog/list-of-free-open-source-s...

I've been using Xenu's Link Sleuth (https://home.snafu.de/tilman/xenulink.html) forever, but I should probably try some others out and see if there's something better now. Xenu's is also a funny throwback to the old web, sort of like the Space Jam website.

the list doesn't have links though, and w3c link checker is there twice, is it some AI-generated text?

Looks like it is just ai blog spam. Bookstack is listed there too lol

Show HN: Bernard – a link checker for your website