DEMO: 80legs promises web crawling for the masses


In technology, it is all the rage to offer something “as a service” – to deliver it online, usually for a subscription fee. We’ve seen software as a service, platform as a service, data as a service, etc. Now a startup called 80legs says it sells web crawling as a service.

Web crawling means browsing and indexing online content in an automated way. Its most important use is to build databases for search engines like Google, but it’s important for anyone who wants to find content on the web, like a movie studio that wants to find pirated footage or an ad network that wants to find content on the web. wants to see where their ads are placed. For now, the main options are to build your own web crawler, usually using your own data center, or take advantage of online services like Amazon Elastic MapReduce, says Shion Deysarkar, CEO of 80legs.

80-leg comparableIt’s much easier to use 80legs (which is launching today at DEMOfall 09, the tech conference co-produced by VentureBeat) – just choose from several menus, telling 80legs where you want them to be. it crawls and what you want it to search for, and it returns a data file with your results. Innovations such as the use of distributed computing technology from Plura also make 80legs more powerful. Deysarkar says that where Amazon can crawl 100 million pages per day, 80 Legacies can crawl 2 billion. (I’ve included a comparison chart created by 80legs.) All of this makes web crawling accessible to small businesses. Deysarkar compares the service to the provision of a “mini-Google”.

80legs also opens an app store, where developers sell apps that further refine web crawling results. For example, according to Deysarkar, developers could sell apps that perform sentiment analysis, search for video fingerprints, or analyze sentence structure.

“Hopefully, we can become something of a de facto web crawl provider,” Deysarkar says. “No one has thought of crawling the web as a market. We want to control this market.

80legs charges $ 2 / million crawled pages and $ 0.03 per CPU hour used. He is based in Houston. The company raised $ 400,000 from Creeris Ventures.



VentureBeat’s mission is to be a digital public place for technical decision-makers to learn about transformative technology and conduct transactions. Our site provides essential information on data technologies and strategies to guide you in managing your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the topics that interest you
  • our newsletters
  • Closed thought leader content and discounted access to our popular events, such as Transform 2021: Learn more
  • networking features, and more

Become a member


Rosemary S. Bishop