Evidence shows Apple is operating a mysterious web crawler


A developer has discovered that Apple appears to crawl HTML websites with its own automated bot, but the exact purpose of the mysterious homemade software remains unknown.

Evidence of Apple’s web crawler, written in the Go programming language, has been discovered for the first time by developer Jan Moesen, who found hits for the software dating back to October 15. Others have discovered significant traffic to their websites from Apple’s mysterious robot, which appears to be shut down from time to time.

Apple bot requests are for HTML pages only and do not access CSS, JavaScript, or image files on a website. Moesen discovered that Apple’s web crawler identifies itself as “Mozilla / 5.0 (compatible; Fetcher / 0.1)”.

The crawler was found from an IP address beginning with “17.” which would indicate that the bot originated from Apple’s servers.

It is not known if the Apple bot is an official company project or if an employee unofficially built and run the system from a company machine.

Apple’s efforts may be related to Spotlight enhancements in OS X Yosemite and iOS 8. Both were updated this fall with enhancements that allow users to directly access web search content, without the need to access a third-party search tool such as Google or Bing.

While Apple does not compete directly with Google or Microsoft Bing, the company has made an effort to provide specific information directly in Siri’s search results. This makes it unnecessary to provide potential links through Siri’s official web search partner, Bing.



Rosemary S. Bishop