Last week, Apple quietly released notes about the existence of an Apple-specific web crawler, creatively dubbed Applebot. The notes provided by Apple are brief and vague, stating that Applebot is used by “products including Siri and Spotlight Search” (italics are mine, more on this below), that it respects Robots.txt files and Robots meta tags, and if AppleBot is not specified, it will follow rules for Googlebot. Outside of providing an email for questions or concerns, that’s about all that’s mentioned in the update. And I would guess that email’s getting pounded, as this brief update leaves many questions up in the air.
In Apple’s note, they don’t specify exactly what Applebot will be powering. Sure, they mention Siri and Spotlight Search, but only as examples of the products that will use it. Do other products use it or will they in the future? What other products and how will this impact user experience on iOS devices? Can Apple truly compete with Google in this area, the company that essentially defined online search and controls 70% of the market? If Apple is creating a broader search engine, will it be an ad-free environment? What influence will brands have over organic search visibility, if any?
Let’s get some of the facts out on the table, and allow me to speculate about some of them.
Apple’s contract with Google is up at the end of the year. And most agree that they won’t renew. This leaves an opening for a default engine used by Safari on Apple’s devices. Will it be Bing, Wolfram Alpha, DuckDuckGo, or are they truly building their own?
Siri and Spotlight Search are currently powered by Bing and Wolfram Alpha. Obviously, this will change in the future. Is Apple dissatisfied with their results, or do they simply want more control?
Mobile search recently overtook desktop on Google. We’ve been expecting it for some time, but now it’s a reality. Mobile is big. Mobile is growing. Apple knows mobile. Are they uniquely qualified to deliver a full mobile experience inclusive of hardware and software? Let’s all consider how well this has worked with Apple Maps.
Apple acquired social search company Topsy in 2013. This was a somewhat curious acquisition at the time, but now it’s starting to make sense. The Topsy team is now leading Apple’s internal search group, which is said to be rapidly expanding (confirmed by the existence of several job postings for search positions). Apple also announced that tweets will soon be included in Spotlight Search results. Creating a search crawler is a major task, but how big of a team do they really need if it’s only going to power Siri and Spotlight Search?
Apple has stated that they make products, not software. That’s their bread and butter, and they wouldn’t have it any other way. Except they might need to. Hardware is becoming increasingly commoditized. Apple may lead the pack as the top smartphone manufacturer, but Android is the most used platform with 52.4% of market share, 9.8% higher than iOS. To diversify their products and continue driving exorbitant amounts of revenue, further diving into software products (and generating ad revenue) may be a smart move.
We’ll have to wait and see what Apple’s full intentions for Applebot are, but for the time being, it’s nice to see that they’re providing at least some instructions for how the new crawler will access websites. I personally look forward to the diversity and competition that a new, legitimate rival to Google could bring.
