hathawayp

Hey I'd appreciate if you could ping me your email address to support@sitebulb.com so we can see what's going wrong with your account (or lack thereof).

We should put that on our homepage.

Yeah for really big crawls your probably better off sticking it on a server or AWS, as much as anything so you don't need to leave your computer on for ages.

But Sitebulb is not resource hungry in the same was as other desktop crawlers. It saves to disk instead of using RAM, so you don't experience the same limitations.

I'm not sure what you mean about Google. There is no link between Sitebulb and Google - it doesn't visit Google at all, so there is no risk of banning. Using it on your 100 Mb work line would be ideal.

It should stick the subdomain you specify in the start URL, unless there is a redirect or something. Other subdomains won't be crawled, although it will HTTP status check links to subdomains. So possibly that is what you saw in the URL log on the crawl progress page?

If you want me to take a closer look send the subdomain over to support@sitebulb.com and I'll see what's going on.

I'm afraid not, at least not yet. It's something we'll work on if the demand appears to be there.

Right now we are focused on other features that appear to be a higher priority to our users.

Thanks for all the detail! Here you go:

- Email confirmation is required for the username/password, which is how free and trial licenses are controlled, and ultimately how paid licenses are doled out. So we need it for the licensing.

- No special characters at all! Excepts periods. Sorry!

- Agreed, we need to improve the settings switcher.

- Crawl Maps is not linked - you mean on the website right? I'll fix that.

- Running audits show on the main Dashboard, seemed kinda overkill to put it on Recent Audits as well. No?

- You can switch of 'Check external' in the Advanced Settings. Kinda 'hidden away' to keep the main settings UI cleaner (otherwise where does it end?!)

- "Filtered URL Lists" - they are there because people want them ('a big list of all the URLs') and kept missing them in our usability tests!

- Why no endless scrolling in tables? It's not easy to do because the data is written to disk, rather than stored in RAM (which is the reason it can typically crawl more pages), so it needs to go and fetch/filter/etc... every time.

I know right, SF is just too cheap for its own good! :p

We think it's a case of horses for courses. Sitebulb has the potential to save you a ton of time when auditing and reporting. If you don't do a lot of that then it might not be a good option for you. If you do, that's where a lot of the value lies.

There's a fully featured 2 week trial to give it a proper go, and the monthly billing means you have the option to switch it on/off as you need it.

:) thanks

You might like these as well: https://sitebulb.com/release-notes/

Answered this one below already. For completeness:

The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc... I've written a more comprehensive answer to this here:

https://sitebulb.com/resources/guides/how-is-sitebulb-differ...

The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc...

I've written a more comprehensive answer to this here: https://sitebulb.com/resources/guides/how-is-sitebulb-differ...

That's frustrating, sorry. It's a 'reputation issue', that over-protective anti-virus software doles out to smaller software vendors like ourselves. Basically they don't know if it is good or bad because we haven't had millions of installations.

i.e. it's a false positive

Nice, thanks. Just ping me or Gareth if you want any of our input :)

It has a lot more comprehensive reporting and data visualization than the likes of Scrutiny. I have no idea of the scale limitations of Scrutiny, but I'd be very surprised if it can handle ~500,000 URLs.

Also Sitebulb is for both Windows and Mac.

Absolutely. In development we tried different ways to make the Crawl Map also represent link data, and they were all just unintelligible. Even the Crawl Maps on big sites are hard to get your head around, and that's with Sitebulb sampling quite heavily.

I'd love for us to come up with some sort of solution for it, I just don't know how we'd do it!

SL presentation I assume?

Thanks for the feedback, good to know.

Yeah we use a username/password so there's no issue with losing a license key.

Sorry, maybe I misread, but I kind of read the comment as 'what separates this from other cloud products on the market?'

So I wasn't trying to argue what is and isn't possible with cloud architecture, simply what is and isn't possible with (our) cloud-based competitors.

The process is along the lines of: 'Click Start', get taken to a screen which says 'Initializing' or similar, then maybe 2-3 minutes later you'll see something start to happen. But there is little to no data on which URLs are actually being crawled.

Sitebulb, and desktop crawlers in general, has a much quicker feedback loop.

Interesting comment regarding the App Store. To be honest I'm really surprised no one has ever said this to us before. We have another product - (http://urlprofiler.com/) - for Windows and Mac that we've been selling for 3 years and no one has ever given us this feedback.

We're not wedded to a price structure, although we're rolling with monthly for now. I'm pretty sure through weight of demand that we'll need to add Yearly plans in the next few months.

There's nothing preventing a Linux version (it's built in Electron) other than demand really. We'll do it if enough people want it, but we have a bunch of other features on our roadmap that are currently a higher priority.

Sorry, I meant to say, there's a free 14 day trial available to anyone once you download the software (no credit card required).

Yep exactly. All crawls are stored locally so there is no data issue to worry about.

Price is always a difficult one trying to get the cost/value balance right. We did some pricing sensitivity testing before launch so I'm hopeful we've not got it too wrong.

Regarding Crawl Maps, yeah it does have some limitations on, which I've written about here - https://sitebulb.com/resources/guides/crawl-maps-faqs/

Although from your comment I think you might be thinking it is a link map, rather than a crawl map. So with the Crawl Map it is mapping out how each URL/node was found when the crawler traversed the site. So each node will only ever have one edge/link.

A link map ends up a LOT more messy, although it's on our roadmap to try and build one of these too!

Thank you! And thanks for the beta feedback, it was super helpful.

The other big thing, in comparison to cloud software, is convenience. You can setup a crawl and start it running -and see URLs being crawled - within a minute.

On cloud software that's simply not possible, due to the way that everything is scheduled.

There are a few other small things, such as being able to view Audits offline (what we call 'train mode').

The cost structure can be a big limiting factor though, especially for smaller companies. Sitebulb effectively remove all limitations around number of domains, number of projects, total number of URLs crawled etc...

Hey HN, Sitebulb is a desktop crawler for Windows and Mac, specifically designed for SEO consultants/agencies. Me and my business partner have been building it for the last 2 years or so, and we're finally launching it today.

It's main differentiating factors: 1. Scale – it can comfortably crawl 500,000+ page websites despite being a desktop program. 2. Reporting – it does a lot of data manipulation and processing so you don't have to. 3. Visualization – it has tons of useful graphs, including the Crawl Maps, which help you visualise site structure.

Our aim was to give it the reporting capability of a SaaS crawler, with the convenience of a desktop crawler.

Looking forward to hearing your feedback on our new product. Thanks, HN community!

Haha, well it's hardly an evolution in terms of sound quality!