Applebot, the web crawler for Apple

I remember some thread from the Safari's early ages where they hid the user-agent, to avoid raising attention.

That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid.

For example, rather than build AppleBot, why couldn't they pump a few billion into DuckDuckGo to get use of DuckDuckBot? Or fund archive.org to access their index? Or buy in to commoncrawl.org?

It would be possible for Apple to use its fortune to benefit both their customers and the world. Google and Bing are not the only options.

I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones"

So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one.

That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before assuming that website does not contain restrictions.

The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web.

Incidentally, this hits close to a pain point: I find it extremely annoying when publishers (like Elsevier) hide content behind a paywall, but still expose it to Googlebot for indexing. The result is that you are able to find a scientific article, which is not accessible (but Googlebot has cached snippets). This goes against Google's own guidelines (they used to tell people that Googlebot must not see different content from browsers). And it goes against the whole idea of the Web: if you want to hide stuff behind a paywall, do so — but then it is no longer accessible.

Going back to Applebot, I love the fact that they will now follow Googlebot instructions. Hopefully people will stop distinguishing who accesses content.

I don't know why people don't treat statements like that with enough cynicism.

And while I would love a privacy focused search engine, I just don't think you can build a good one without private data.

"Our business model is very straightforward: We sell great products. We don’t build a profile based on your email content or web browsing habits to sell to advertisers."

This obviously is not saying Apple doesn't sell ads. It's saying their core business model is selling products. The users are not the product.

On the other hand: it is kind of crazy that when I read about "indexing the web" it starts ringing all kind of privacy bells, while apple's incentive might not even be to violate people's privacy.

EDIT: on that point. It would be very neat but probably very difficult to build a search index based on a agreed upon algorithm like bitcoin. There would be a need for some sort of voting system to update the algorithm and moderate people gaming it.

I could be wrong, but I think Apple might have an ideological problem with a lot of content. It's their call if they decide to filter that stuff out, but it's censorship, and I struggle to see how that would result in a "better" search engine.

EDIT: Applebot still has uses beyond a search engine though. I think Apple are being straightforward in it's explanation. It's for Siri and Spotlight.