Applebot
support.apple.com
support.apple.com
- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).
- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.
- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...
Source: I worked on the original version.
Unlike other languages, Go bypasses system's DNS cache, and goes directly to the DNS server, which is a root cause of many problems.
Since netgo is faster, by default Go will try its best to determine if it must use netcgo by parsing /etc/nsswitch.conf, looking at the tld, reading env variables, etc..
If you're building the code you can force it to use netcgo by adding the netcgo build tag.
If you're an administrator the least intrusive method I think would be setting LOCALDOMAIN to something or '' if you can't think of anything which will force it to use NSS.
If you're on a system with cgo available, you can use `GODEBUG=netdns=cgo` to avoid making direct DNS requests.
This is the default on MacOS, so if it was running on four Mac Pro's I wouldn't expect it to be the root cause.
To make DNS resolution really scale, we ended up moving all the DNS caching and resolution directly into Go. Not sure that's how you'd do it today, I'm sure Go has changed a lot. Building your own DNS resolver is actually not so hard with Go, the following were really useful:
https://idea.popcount.org/2013-11-28-how-to-resolve-a-millio...
As I understand it, Go and Java are both trying to avoid FFI and calling out to system libs for name resolution.
I tend to always offer a local caching resolver available over a socket.
Considering the timeline, are those Trash Can Mac Pro? Or was it the old Cheese Grater ?
The scale of web stuff sometimes surprises me. 1B web pages sounds like just about the daily web output of humanity? How can you handle this with 4 (fast) computers?
Insanity.
The HTTP HEAD method requests the headers that would be returned if the HEAD request's URL was instead requested with the HTTP GET method. For example, if a URL might produce a large download, a HEAD request could read its Content-Length header to check the filesize without actually downloading the file.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HE...
When you request a page, the response includes metadata, usually including a Last-Modified timestamp and often including an ETag (entity tag). Then when you make subsequent requests, you can include these in If-Modified-Since and If-None-Match request headers.
If the resource hasn’t changed, then the server responds with 304 Not Modified instead of sending the resource all over again. If the resource has changed, then the server sends it straight away.
Doing it this way means that in the case where the resource has changed, you make one request instead of two, and it also avoids a race condition where the resource changes between the HEAD and the GET requests.
Does it use a traditional relational database or another existing database-like product? Or is built from scratch just sitting on top of a file system.
Also interested at the higher end of the scale about how 'hard'/polite you should be with authoritative nameservers as some of them can rate limit also.
Apple's corporate network also had incredible bandwidth to the Internet at large. Not sure why, but I assumed it was because their earliest data centers actually ran in office buildings in the vicinity of 1 Infinite Loop.
Apple's server stack has been primarily Java for about 20 years.
Great use case for Go though, especially its concurrency features for web crawlers. I reckon Scala could work too, although it's a lot more complicated / clever.
The book is freely available online at https://nlp.stanford.edu/IR-book/information-retrieval-book....
I guess there are not many places where you can easily get 4GB/s sustained throughput from a single office (especially with proxy servers and firewalls in front of it). Is that standard at Apple or did the infrastructure team get involved to provide that kind of bandwidth?
All three uses of "it's" should be "its".
And I would just write "Mac Pros" instead of Mac Pro's".
https://appleinsider.com/articles/20/08/27/apple-may-launch-...
I use DuckDuckGo for search, and do trust them with privacy. But if Apple did a half-decent job at nailing the search experience, I'd switch.
https://duckduckgo.com/?q=San+Francisco&t=h_&ia=web&iaxm=map...
The ads that Apple sells on the app store contribute $2 billion in revenue -- I don't think they'd build a search engine that isn't built around ads.
I can't see how Apple would even do the kind of search result pollution Google does now. In fact, I suspect the only reason Apple is launching advertising is because they see an opportunity to provide a better experience—namely one where you see organic results immediately.
People thinking Apple aren't going to copy Google's revenue model to some degree are delusional.
[0] https://9to5mac.com/2019/11/26/accusing-apple-of-selling-cus...
In addition, google has become very aggressive with disregarding words. So a search like (a fake example, not tried the search, not a real place) Jim’s tasty pizza Albuquerque 90s where I’m searching for where a restaurant I remember as a child was located, and has maybe 5 real results from a local history Wordpress blog 15 years ago and Facebook posts, Google will just drop everything specific and give you results for Albuquerque pizza, and then mainly show ads for national chains up top. So I need quotes in the search pretty much always.
I use it for certain types of searches, and not others. I'm conscious of the fact that what I'm typing and searching, and subsequently clicking through will be piled on to the stack Google thinks is me. So it's usually tech searches, real estate/apartments, street view (property or exploring).
Health matters, personal matters, etc I tend to keep out of their search box.
Though more and more I tend to do a lot of cross-searching if privacy is not a concern.
Google's job as a search engine is to provide relevant and quality content. Ads are spam and noise, so if Google does want to provide quality content they should be penalizing ads just like they would penalize keyword spam and similar unpleasant SEO tactics. Given their business model however it is not in their best interests to do so, so it's very unlikely that Google will do it.
Another company that does not make their money on advertising has a better chance of doing this.
Meanwhile, ads are distinct from search. The #1 result refers to the first result past the ads.
My point is that as a user, ads, regardless of how "good" they are, are not what I am looking for when I click on a search result (I am talking about website ads and not ads on the results page itself) and thus a search engine that downranks pages with ads in favor of those without them would be beneficial for me.
It would be against Google's interest to implement that (as they'd also be fighting against their own ad network) but a different company not involved in the advertising business (and making high enough margins on other products) has a higher chance of delivering this successfully.
Google search has gone down the toilet. Most Google search results are ads or SEO optimized junk. It was funny once when I was searching for a medical term, all articles on first page were clearly SEO optimized and all articles were very similar from word usage, sentence construction, verbiage, like written by same person. I am surprised that Google doesn't filter out the results using some sort of similarity analysis to offer variety in results. I don't want to see all links giving exactly the same info.
Reason is that Apple already has a Search Engine. It is called Siri. It is what you get when you ask Siri Questions. What I think Apple hopes to achieve is that get 80% of what you want, From may be Recipes, Simple Answers like Exchange Rate, Nutrients, Sport Scores... etc away from Google.
Letting you to Google the absurd, hard, questions that has less Data value.
All while collecting roughly $10 per user for Default Search Engine Placement in Safari.
And remember the Privacy stand Apple has giving less Data to Google.
Basically Apple is squeezing Google in every single direction. And pitching them against Azure and AWS for Cloud Services for discount.
Most of your issues seemed like the old, rehashed ones from nearly a decade ago.
Photos is pretty great too, but I’m not sure how it compares to other photo services.
Obviously that's not the only reason, haha, it's gotten surprisingly good a few years ago. Public transit and cycling is better in Google (though nothing beats CityMapper for transit if it supports your city). But for walking and driving Apple Maps is my winner.
Mail is definitely a local app, even if they did screw the pooch pretty badly with that gmail bug they had a few years ago.
No. Strictly speaking, iCloud doesn’t even do backups; it syncs data. Difference is that, with backups, you can get a copy back if you accidentally throw away a file. You typically also have multiple ‘old’ copies.
iWork apps (1) kind-of have backups in the form of versions that get synced to the cloud (https://support.apple.com/en-us/HT205411), but I think that’s implemented independently from iCloud’s syncing (You can use it to have multiple versions of a document locally, too)
(1) Third-party apps can use this, too (https://developer.apple.com/design/human-interface-guideline...), but I wouldn’t know how commonly this is used
However, Apple operates many services at scales few companies ever reach, and doesn't seem to have outages much more often than those other few companies. So they're clearly doing a lot of things incredibly well, in the if-things-work-you-don't-notice sense.
They don't seem to emphasize their cloud services like Google does, and Apple charges for theirs at a much lower level than Google does, but I'm starting to think my vague intuition is wrong.
I don't know why anyone would think that Apple would make a web page search engine like Google. Apple has already launched the Apple version in your phones integrated into the software you use.
I don't use Siri or any assistant, so I don't seem to use what I'm being averaged into using. Majority opinion here is that Siri is seriously lacking behind the other offerings. Apple won't be oblivious to this. If it is solely due to their back-end search engine, then making that a public search engine website won't do Apple any favors if the results are truly that bad.
That they use their own search for this is not a big headlining marketing feature so they aren’t really evaluated on it.
I do think that’s a direct consequence of their experience with Maps, though I guess their hand was more forced on Maps and they had to tell the world about it.
However, using Siri and search suggestion they can, bit by bit, replace search results from other sources with their own without telling anyone about it. It’s a more careful approach.
But the SEO market would certainly change if webmasters suddenly have to appeal to more than just Google to be successful.
How does this benefit Apple? They’re a hardware company.
How does a search engine sell more hardware?
They don’t have Google as the default search because of the money, they have it because they think it is the best. Even despite the privacy problems and the control this grants Google.
How does a maps service sell more hardware? Because using their own maps service Apple can completely control the experience.
Now maps was hard and search is much harder still so we may very well never see this released. But even then they can use this as leverage while negotiating with Google, both for money and for control.
Because your claim about them being a hardware company is incorrect.
They market themselves as a privacy focused company. Privacy is all about software and services, not hardware. Search is something they could add to their portfolio to strengthen their privacy stance.
Also, Apples main revenue isn’t hardware but services IIRC. They’re not a hardware company.
The last 10% quickly gets very expensive tho.
Apple’s view is that you consented to this sort of unaccountable invisible tracking by dozens of third parties (against whom you have zero recourse), when you agreed to the TOS of the App Store.
The data is aggregated, sold, and resold again.
The big question would be what Apple would gain from a dedicated website for search results. Would people really switch to it from Google? Why would it be a better delivery mechanism for search results than Spotlight? Not sure the answers to these questions has changed much, from 5 years ago to today.
They don't though. There are only a couple countries where they're more popular than Android, and only barely there. There's no country where they have 60% market share.
Apple’s hardware sells were also up slightly in every category year over year.
I believe if anyone can build a search engine that can truly deliver the same quality as Google, the whole market could change. Google is dependent on search ad revenue like nothing else, if they lose that monopoly it could really change the company.
But google pays Apple in billions to keep its search as default, what could be the key motivation for Apple to build its own engine? its a highly saturated market and we have Duck Duck Go for privacy based , bing for alternative and heck, even wolfram alpha for computational! what Apple can improve?
In my personal experience Google Maps has better navigation data, better place information and overall much better maps. The user contributed content makes the platform such a pleasure to use even if you are in remote parts of the world.
That said, even here in the Bay Area I tend to do a sanity check of the route on Google Maps first if I’m taking an unfamiliar route. Just a few months ago a section of I-80 was closed just south of SF and Apple Maps had no clue! I believe it was a fairly last-minute schedule change for some planned construction due to vastly reduced traffic during the lockdowns.
There’s the maps layout and where I want to go and my route. That’s it. Google Maps has ads. Switching between Apple Maps and Google Maps really shows how the Google product has friction and information overload. We are talking about a Maps app. Less is better unless I ask for it. Apple has nailed the color scheme, layout, and information density. If I need more information that Apple Maps doesn’t provide, I double check google maps. On a long trip, I will check routes from both apps. Apple Maps and google maps show the same route and same time and same traffic congestions for me. I rely more and more on Apple Maps now.
I’ve found myself swinging in the other direction. I tend to glance at Waze for my speed instead of my own speedometer, but I’ve been driving a mixed bag of cars lately.
The history here is also at least worth a passing mention -- the iOS Maps app originally used Google mapping services. The reason Apple built their own back end is because Google refused to give them access to vector-based maps and turn-by-turn navigation unless Apple shared more user data with Google than they were willing to provide.
I would say Waze is a solid counter-argument.
For those not familiar "state highway" simply means the central government agency pays for upkeep, rather than indicating any particular road quality or status.
* iPhone + Apple Watch combo is amazing to use w/ Apple Maps. Audible & haptic alerts on your wrist change how one interacts with the turn-by-turn directions, and it's an improvement.
* The fact that iPhone nav is available on the lock screen is a huge help. Doesn't require full unlock to get to the map.
* Apple Maps just seems to handle the accelerometer in the iPhone better. I found that Google Maps never knows what direction I'm facing/walking, and it jumps all over the place all the time. Apple Maps is just smoother and better to use.
The only thing that's missing in Apple Maps is bicycle directions, but apparently that's coming in iOS 14, which is great.
I similarly find Apple Pay great for traveling, and easier to use than a credit card.
But, on my no-longer-bleeding-edge iPhone, the turn-by-turn directions suck. It's constantly telling me to turn onto a street I just passed, or rerouting to find a way to get me back to the street that I'm already on. The worst part is that I can't trust it even when it's right. If it says "in 300 feet turn left," I have to desperately look at the street names because it's quite possible that I actually have to turn in 30 feet.
It was such a relief when I realized that it's not my GPS hardware, Apple Maps gives directions perfectly. And the map data is way better than it used to be. Not as good as Google Maps, but good enough.
Probably 75% of my Google Maps usage is to check business hours or grab a phone number; they've really got that down. Apple is catching up, though!
The iPhone I have now is a work phone, so don't use maps a lot, should see if it has improved.
And I am biased towards Apple Maps for privacy and usability reasons
Google maps added this to their desktop app [1] and maybe their iOS app, but apple has had this for years.
Also Apple maps offers saved places in an offline, non-synced way. Google requires access to all my location data to save places (like web bookmarks) and that's just too privacy invasive for me.
[1]: https://arstechnica.com/gadgets/2020/08/google-maps-starts-w...
Both applications would select the same number of businesses to show (obviously not showing ALL names due to density restrictions on the screen and clutter), but the Google ones were just so painfully arbitrary, and Apple's building skeletons seemed a lot more relevant/recognizable.
I'm guessing that it depends a lot on the area. Google Maps is still probably better in most places, but Apple Maps is catching up and may be better in some populated places. It's impressive considering the head-start Google has.
Apple should add an option of OpenStreetMaps overlay or something. OSM has been better than Google Maps in mapping paths in the forest and such here.
I speak one of the languages, not the other. Guess which language Google decides is the correct one?
I've put in business addresses or searched the business themselves, and then had it route me to a residential block behind the business on a completely different street, sometimes multiple blocks away from it. This is even after confirming the exact address is being shown and everything.
I only really use Google Maps because it's what I've been using, and I kind of like that its UI is a bit simpler with less animations and things than Apple Maps.
1 - It actually runs smoothly on my iPhone 11. Google Maps stutters and yanks. Yes, Google cannot build a performing maps app for the fastest phone in the world.
2 - It is a map. Not a yellow pages of coffee places, not an Instagram with amateur photos of locations.
Also, the Google Maps app has gotten much more cluttered and complex than in past versions. They are still the gold standard map, but they regress over time.
EDIT: I looked back in my emails and Twitter DMs and it's been 25 months at this point. So "approaching three years" is not accurate.
Edit: Also want to add that it has been two years since my parents moved in. Went in to Waze Map Editor and added the road myself, so their turnaround time was instant. Apple Maps took 9 months, like I said, but Google Maps still has no info on the street despite having all the houses in clear satellite view.
It did take Google a few months to add my complex. I'm not sure what an appropriate turn around time would be on something like this. But I do think it's less than "years".
It's not like it's some street address thing, they are missing a whole town.
In general, Google still has better POI data than Apple Maps in the UK, but the gap has closed a lot in recent years.
Whenever someone needs to come to my house I need to ask them to use Google Maps. The challenge from that comes from less techy people who don't know the difference. Many people just know their phone has a "maps app". I will admit, experiencing that more closely (how less technical people interact with their phones that is) was interesting fwiw.
Further, it repeatedly picks a hard route. There's one park I go to that's easier to drive to using one few mile stretch of highway. Google insists it's faster to take these back, poorly lit, windy 2 lane roads. Maybe it is. But it turns a safe easy 3 turn trip into a nerve-wracking 10 turn trip. And it will, as noted above, keep trying to reroute me if I just ignore it and go the easy way.
Fast-forward to 2020 - in the meantime I used Google Maps with no issues at all. A friend of mine suggested that I should try the latest version and maybe I would change my mind. So I used it for a day - same urban area - and it's still lagging behind - mind you: at this time I had a new iPhone and a new carrier. For me this makes the App unusable. I don't want to plan ahead 500 - 1000m (Germany) while using navigation.
I'm really curious, does someone have the same problem?
Yesterday for example it told me to do a U turn in a (sort of) one way street in the middle of the city, and not the first time its been dumb in the last few weeks.
Is there public info on the pricing for sponsored POIs, e.g. is it auction-based, for a period of time? Does the listing get dropped from the map after the business stops paying? How does Google get around the FTC requirement to label sponsored content?
For instance Apple Maps doesn't have U-Bahn support in Munich, which makes it unusable for most people.
Meanwhile Bing maps manages to be the most frustrating experience of any current online maps.
[1] https://appleinsider.com/articles/15/05/06/apple-details-new...
[2] https://appleinsider.com/articles/15/05/06/apple-challenges-...
https://web.archive.org/web/20190429220555/https://support.a...
What seems to be new:
- "About search rankings" section listing its purpose
- Expanded bot identification and verification information
the printer stuff is something else. i put some fault on canon here for pushing me to unofficial sites because their official shit requires you to download a fucking exe to deliver a PDF manual (cmon, wtf, how can your docs delivery pipeline be THAT bad), but the unofficial stuff is some ninth circle of hell
the first was some bog-standard infini-redirect ad fraud crap, but the second was uh... well, i was impressed to the degree that the indian tech chop shop "YOU ARE INFECT DIAL 1-800-LEGIT-MICROSOFT TO CLEAN COMPUTER" site was able to shut down a fully up-to-date copy of firefox, somehow eating all my other tabs, grabbing hold of my mouse pointer in strange ways when the window was in focus AND preventing the windows "close window" button from killing it.
task manager was still able to execute it but damn, that was some ARTISTRY put into making the fraud site as clingy as possible--puts 90s-era malware sites to shame, and this shit kicks around on right on the first page of google search results for a printer manual query, not some shady-ass porn site you'd drunkenly click through to via 3 levels of ever-sketchier ads
people ranted and raved about goods of questionable quality gaming amazon reviews, but eh--honestly, most of those i've bought were fine, if not artisanal high-end shit--they just engaged in a lot of dubious "get free shit for a good review" growth hacking tactics that are A-OKAY THATS JUST HOW YA DO BUSINESS when it's some SF startup pulling the same crap in a different form. it's weird how seemingly nobody gives a fuck that a lot of google searches will boost things that are far worse--offering purestrain fraud on a platter is fine, selling mediocre flash drives and kitchen gadgets with inflated reviews is THE HIGHEST OF CRIMES
Somewhere in between, Google stopped caring, and we have what we have today.
Dark mode to save my eyes, setting my own personal preferences for which search engines to use for results, Cached results link and proxied links, a files search engine...
The only difference is that I wasn't already heavily invested in other Google products. e.g. Gmail, Lots of saved Maps POI, etc.
For someone who is heavily invested, I'm not sure if the switch to Searx is worth the lack of convenience.
Yes, that’s possible, but Applebot is used in a lot of ways today that are wholly unrelated to search. (If search consists of crawling, indexing, ranking, retrieval, frontend, etc., Applebot only does the crawling part.)
Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it). From the docs, it sounds like it’s also used for similar previews in Siri.
Is this true? I would have assumed that would be done locally on device. The preview image, for example, is what's designated by the Open Graph[0] og:image tag.
[0]: https://ogp.me
But do you remember when Apple launched their maps? A decade later and it's still not nearly as good as Google maps. But it's "good enough". I imagine that's what they're going for with this new search service
However, Apple started collecting their own data in 2015, and rolling out their own maps in 2018.
>Apple is filling its map with so many [details] that Google now looks empty in comparison and all of these details create the impression that Apple hasn’t just closed the gap with Google—but has, in many ways, exceeded it.
https://www.justinobeirne.com/new-apple-maps
Apple's map data now covers the US and its territories and data for the UK and Ireland are currently in testing.
- It's established applebot is already used for Siri, but there is no general web search
- Much commentary diverts to how poor Apple maps (is?) was.
- Google pay apple a handsome amount of money to be the default search engine on Apple devices
- If Apple were no longer to accept that payment, about 80% of people would still use Google anyway, given a choice [0]
- Googles high earnings per search means they can offer more than any other search engine, like Bing, DDG et al
- People would generally like more competition in the search space and Apple has the means to do it
- Anecdotally, there isn't a drop in replacement for a search default other than Google that would meet the quality requirements for end users. In this case, the only other alternative would be an Apple search engine.
- As per recent articles, there does seem to be more activity from Apple wrt applebot crawling and hiring
[0] https://spreadprivacy.com/search-preference-menu-research/
This is why Apple Maps is so heavily used on iOS. Most people don't know or don't care. In the first year after the launch of Apple Maps it hit over 65% of maps usage on iPhones in the UK, even though back then the UK maps were awful.
There has never been any point in clawing back the large class A assignments. If that were done it wouldn't even give us an extra year of IPv4 address assignments before we exhaust the pool again... not that there is any legal mechanism to take them back anyway.
The solution is IPv6. Despite the slow rollout it is finally happening now that even the regional RIRs have started to run out of addresses.
My bad experience is generally rooted in the above criteria. It's particularly problematic on news sites. For example, if I click a news article with certain political association, I keep seeing only news from that political side. This creates an echo chamber which makes uninformed believe that 'whole world' is thinking that way. It definitely make sense to increase more engagement, but might not be totally appropriate in all instances. Curious to see how it works in live (if the apple search does become a reality)
I'm wondering if the same holds true when that experiment is conducted in isolation. For example, if we take any particular news site, if a user is repeatedly visiting the same news site, then my hypothesis is that the kind of news one sees will converge onto a specific political view point. It might vary based on each news site - google news vs apple news based on how they are optimizing for users.
In reality, the isolated experiment might be moot as users do end up on news site from SNS which ultimately leads to recommendation algos picking up diverse signals to show diverse results. Given that, whether the algo itself is diverse in isolation or not wouldn't matter as the environment that feeds it is diverse. Does it make sense?
A lot of YouTube channels and types of content could be replaced by video podcasts, and be more convenient for viewers.
As for monetization, Apple wouldn't even have to build an ads platform; content creators could embed ads themselves into the content, as they currently do for audio podcasts.
Everyone appears to be nervous because of the supposed search engine. I believe this was part of their initiative to improve Siri results; when the service was even worse.
Is that pretty much normal?
https://gs.statcounter.com/browser-market-share/desktop/worl...
As a user, I tend to be quite happy with the search results in the App Store, I don't remember any time where I could not find the thing I am looking for.
There are reports that right now Applebot is being used to scrape the content used by Siri. If youre already scraping the web, indexing it and building search and analytics services on top of it for Siri, user facing search is just a short step to take.
Apple has always sided with privacy and while DDG toots it, they’re too small to build an independent service.
I am thrilled, excited and perhaps surprised why it took so long for the most used website for large majority of the people in the world to see some competition.
Please don’t fuck this up, Apple.
Edit: see here @jd20 thread for insights and confirming this.
The bottom of the page says "Apple Search may take the following into account when ranking web search results:", is this another confirmation that Apple is developing a search engine?
If proxies, what is their source for proxies.
Shouldn't it be "robots.txt"? It even says so below.
Might do that same for all iOS traffic.
But.
If Apple wants to embrace the openness of the web that they benefit from, I feel they should not cripple it on their mobile devices.