SiteOne Crawler – big tool with no users | Hacker News Reader

SiteOne Crawler – big tool with no users

crawler.siteone.io

Hi andrei-akopian. I am the main author of SiteOne Crawler and thank you for this post!

For about 20 years, I have been leading the development and infrastructure of the Czech webdev company SiteOne, so I wouldn't say that I am not a professional ;) I have around 50,000 hours of practice. But I definitely know a lot of better programmers than me and many of them are at SiteOne :)

However, after the extremely premature birth of my son, I had to change my role at SiteOne, but all the more I wanted to help my colleagues with a useful tool that we generally lacked and at the same time I wanted to give myself joy in difficult times.

I want to implement a number of other useful improvements into the crawler (some of them are also described in the documentation and roadmap) and work on its promoting in parallel. The more people will use it, the more it will help to optimize the website, the more it will help developers and testers with various needs, the more joy it will give me :)

By the way, the GUI application https://github.com/janreges/siteone-crawler-gui is just a visual wrapper over the command-line version of SiteOne Crawler and integrates it into itself.

This looks really great and I'll definitely be giving it a go. Like others I really don't understand why it hasn't got more traction (if, when I try it, it does what it says on the tin!). I suspect the comments about the homepage aesthetic probably couple with the fact the dev is a dev and not a marketing person!

Just to add, having tried to install...

I'm a partial nerd and probably exactly your audience - I run a small tech agency, familiar enough with PHP to break things but not a proper dev, work on all range of stuff like SEO / content / accessibility / etc... ie - this is exactly the kind of tool I'm looking for to do site audits, content stuff, etc. Normally I find myself reaching for yet another license of Screaming Frog's SEO Spider [0] and find myself groaning at having to pay another £240 p/a - but do it anyway...

Anyway - at first attempt with this I get a "won't install" warning on my Mac. And it's sufficiently scary for someone like me (techy enough to be aware of all the potential things that can go wrong with just installing any old software from the web but not skilled enough to look at the source code and figure out if you're legit or doing something awful to me...) - that I just can't really justify installing it.

Your message about it being notarized and potentially ok to find on the Mac App Store is reassuring at first, but dated 2023.

(I know I have a slight double standard here in that I install Screaming Frog but I've used it for years and know it, whereas you feel fairly unknown...)

So - just some feedback there, I think I'm exactly your audience, and I'd pay for this, given some further signals that it's legit.

[0] https://www.screamingfrog.co.uk/seo-spider/

I really appreciate your feedback, thank you!

So even if you try to install the latest version 1.0.8 on macOS, it tells you that it won't install it? I'm not a macOS user, but for this project I bought a MacBook M1 and built the app and tested it on it. Thanks to the notarization, the application was installed without any problems. But maybe I missed something. But there will probably be a problem if you report it. By the way, did you download the correct version for your Mac? For M1/M2/M3 processors you need to download the arm64 version, for Macs with an Intel processor you need to download the x64 version.

PS: publishing the application to the AppStore didn't seem easy and unfortunately I didn't have time to go through the whole process. I will try to get it done by the end of the year.

Hey - yeh, I get this [https://share.cleanshot.com/NFRk0XDS] - Mac Mini M2 Pro

For some odd reason, the tool didn't get any replies on Reddit, HN, AlternativeTo.net or GitHub, despite looking fairly sophisticated. The dev is also non-professional.

Maybe I am bad at digging, but I couldn't find anything about siteone.io. The entire situation is strange.

Older discussion with one reply: https://news.ycombinator.com/item?id=41292651

yeah and it looks really full-featured and well-done, while many other crawlers/scrapers with way less get way more stars

Yes, the tool has a lot of features, settings and usage options. I believe that many people will like the tool and help promote it themselves.

Very small amount of sourcecode: Under 20k by my count. Are they outsourcing the actual crawl to a different tool?

It seems to contain the crawler: https://github.com/janreges/siteone-crawler/blob/main/src/Cr...

SiteOne Crawler offers 2 open-source tools: 1) command-line version (which is the core of the crawler itself): https://github.com/janreges/siteone-crawler 2) a graphical application for Win/macOS/Linux, which is just a wrapper over the command-line version and displays the output from it through Xterm: https://github.com/janreges/siteone-crawler-gui

The parent, SiteOne, seems to be a well-established IT company in The Czech Republic. Hopefully, they will gain more traction in the near future.

Maybe people ignore it because there are already many tools like this. I was tempted to not look, but checked it out anyway

My cynical 2 cents: First impression is that the above-the-fold webpage looks like a page out of a kindergartner's book. 2 sentences with different bright colors. The red heart (the cynic in me cringes when it sees all those "Made with (heart)" lines). The cartoon character logo...

I scrolled down, and the video with the cartoon font? Oh god, that reminds me of an exchange student at uni who used a Teddy Bear PowerPoint template for a presentation.

Thanks for the feedback! That's my job :-| I'll ask my fellow designers to improve it in the near future - as a backend developer I obviously fail a lot in graphical sensibility, or I'm just a kid with a need for child fonts ;)

Post mortem:

janreges is according to the website (siteone.cz) one of the top 3 people at his company, and the tool and website look well made. By "non-professional" I meant that there is some kind of mismatch between my expectations and the style he replies to people here.

I was trying to establish the oddity of the situation to encourage more digging, and didn't express my point clearly enough.

My apologies to janreges. I know my own expertise and social sense are bad, so please don't take any of this to heart.

There's no reason to apologize, man! :) On the contrary, I am very grateful for this post and your proactivity :-*

Reply on news.ycombinator.com