Google will allow sites to disable web crawling after the FTC agreement expires

  • Google’s agreement to allow sites to opt out of its web crawling expires on December 27.
  • In a letter to the FTC, Google promised to maintain the deal going forward, which was originally put in place in 2012 after an antitrust investigation.
  • It’s a win for sites like Yelp that claim Google is using non-Google content on search results pages.

google said tuesday that it will maintain an agreement to allow sites to opt out of web crawling by the search giant, even if the agreement expires on December 27.

In a letter to the FTC, Google said, “We believe these policies provide additional flexibility for developers and websites, and we will maintain them as policies after the Commitments expire.”

Google originally entered into the legally binding deal in 2012 following an antitrust investigation against the company. The deal was part of his settlement with the US government.

Websites like Yelp have complained over the years that Google unfairly “trims” content from their websites and monetizes that content on search results pages. The agreement allows sites to prevent Google from harvesting their content for use on search results pages.

But Yelp accused Google in September of violating the agreement. In a letter to the FTC, Yelp said it found thousands of instances where Yelp images appeared in search results.

Rosemary S. Bishop