janreges

Hi HN, I'm the author. I originally built SiteOne Crawler in PHP+Swoole back in 2023. Last year I rewrote it entirely in Rust — 25% faster execution, 30% lower memory, and a single native binary with zero runtime dependencies.

The feature I'm most excited about is CI/CD quality gating. The idea is simple: crawl your entire website after deploy and block the pipeline if quality regresses.

Example:

   siteone-crawler --url=https://example.com --ci \
      --ci-min-score=7.5 \
      --ci-max-404=0 \
      --ci-max-redirects=5

Install:

   # Debian/Ubuntu repo setup:
   curl -1sLf 'https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.deb.sh' | sudo -E bash

   brew install janreges/tap/siteone-crawler    # macOS / Linux
   sudo apt-get install siteone-crawler         # Debian/Ubuntu (after adding repo)
   sudo dnf install siteone-crawler             # Fedora/RHEL (after adding repo)
   cargo install siteone-crawler                # from source, any platform

   # Windows: https://github.com/janreges/siteone-crawler/releases

This crawls every page, scores it across 5 categories (Security, Performance, SEO, Accessibility, Best Practices) on a 0–10 scale, and exits with code 10 if any threshold is breached. Drop it into GitHub Actions, GitLab CI, or any pipeline as a single binary — no Docker, no Node, no runtime needed.

Beyond CI/CD, it also does: - Offline website archiving with a built-in HTTP server for self-hosting - Full-site markdown export with deduplicated content (great for feeding to LLMs) - Interactive HTML audit reports you can email via built-in SMTP - Sitemap generation

Sample HTML report: https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-f... GitHub: https://github.com/janreges/siteone-crawler

I'd love to hear your feedback — especially if you're already doing something similar in your CI/CD pipelines. What thresholds would you find useful?

Hi @kiwicopple, thank you for your reply. The project will be implemented in the EU and of course we will be subject to GDPR regulations and laws related to medical documentation. If the project is successful, we would like to provide the source code free of charge to responsible people in other countries, but that is far in the future.

My post was more about whether the actual power-users of Supabase consider the current state and version of Supabase production-ready even for high-load deployments. If e.g. GraphQL performance is not a problem, etc. If we decide to use Supabase, it is definitely only with strict RLS and our own backend outside Supabase for critical API endpoints. It will be a completely self-hosted solution on your own physical servers, no cloud, no edge-functions, etc.

Of course. I am already in contact with relevant people from the state administration who will help with regulatory issues. We are only at the beginning, so we do not yet know for sure which features will be problematic from a regulatory perspective. For some things, parental consent will be enough, for others more will be needed. But I believe that with competent professionals we will achieve a good result.

There's no reason to apologize, man! :) On the contrary, I am very grateful for this post and your proactivity :-*

I really appreciate your feedback, thank you!

So even if you try to install the latest version 1.0.8 on macOS, it tells you that it won't install it? I'm not a macOS user, but for this project I bought a MacBook M1 and built the app and tested it on it. Thanks to the notarization, the application was installed without any problems. But maybe I missed something. But there will probably be a problem if you report it. By the way, did you download the correct version for your Mac? For M1/M2/M3 processors you need to download the arm64 version, for Macs with an Intel processor you need to download the x64 version.

PS: publishing the application to the AppStore didn't seem easy and unfortunately I didn't have time to go through the whole process. I will try to get it done by the end of the year.

Thanks for the feedback! That's my job :-| I'll ask my fellow designers to improve it in the near future - as a backend developer I obviously fail a lot in graphical sensibility, or I'm just a kid with a need for child fonts ;)

Yes, the tool has a lot of features, settings and usage options. I believe that many people will like the tool and help promote it themselves.

SiteOne Crawler offers 2 open-source tools: 1) command-line version (which is the core of the crawler itself): https://github.com/janreges/siteone-crawler 2) a graphical application for Win/macOS/Linux, which is just a wrapper over the command-line version and displays the output from it through Xterm: https://github.com/janreges/siteone-crawler-gui

Hi andrei-akopian. I am the main author of SiteOne Crawler and thank you for this post!

For about 20 years, I have been leading the development and infrastructure of the Czech webdev company SiteOne, so I wouldn't say that I am not a professional ;) I have around 50,000 hours of practice. But I definitely know a lot of better programmers than me and many of them are at SiteOne :)

However, after the extremely premature birth of my son, I had to change my role at SiteOne, but all the more I wanted to help my colleagues with a useful tool that we generally lacked and at the same time I wanted to give myself joy in difficult times.

I want to implement a number of other useful improvements into the crawler (some of them are also described in the documentation and roadmap) and work on its promoting in parallel. The more people will use it, the more it will help to optimize the website, the more it will help developers and testers with various needs, the more joy it will give me :)

By the way, the GUI application https://github.com/janreges/siteone-crawler-gui is just a visual wrapper over the command-line version of SiteOne Crawler and integrates it into itself.

Thank you for this simple but useful tool! It works. I will be happy to use it e.g. with USB sticks, where I sometimes transfer sensitive data and so far I have only used dm-crypt.

Btw, if somebody's Ubuntu/Debian doesn't want to find "oath-toolkit", install "oathtool".

Thank you for your feedback! Our goal is to help improve the quality of websites around the world, so we will continuously implement improvements based on feedback from real users.

I adjusted the font size for a long time, trying to choose one that would be a compromise in readability and not breaking the content on different screens and typical resolutions. In addition, the report is optimized to work well even in a mobile e-mail browser. See Mailer options - https://crawler.siteone.io/configuration/command-line-option....

That is also why, for example, switching the left menu and light/dark mode is implemented purely using CSS, without JS. The mail client in iOS, for example, supports CSS when opening an HTML attachment, but does not execute JS at all.

If you have other ideas for improvement, feel free to submit a GitHub issue with a feature request. Thank you ;)