The Ethicality of Web Crawlers

Search engines largely rely on web crawlers to collect information from the web. This has led to an enormous amount of web traffic generated by crawlers alone. To minimize negative aspects of this traffic on websites, the behaviors of crawlers may be regulated at an individual web server by implemen...

Full description

Saved in:
Bibliographic Details
Published in:2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology Vol. 1; pp. 668 - 675
Main Authors: Sun, Yang, Councill, Isaac G., Giles, C. Lee
Format: Conference Proceeding
Language:English
Published: IEEE 01.08.2010
Subjects:
Costs
Crawlers
ethicality
Ethics
Guidelines
Privacy
Regulation
Robots
robots.txt
Search engines
Vectors
web crawler ethics
ISBN:9781424484829, 1424484820
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Search engines largely rely on web crawlers to collect information from the web. This has led to an enormous amount of web traffic generated by crawlers alone. To minimize negative aspects of this traffic on websites, the behaviors of crawlers may be regulated at an individual web server by implementing the Robots Exclusion Protocol in a file called "robots.txt". Although not an official standard, the Robots Exclusion Protocol has been adopted to a greater or lesser extent by nearly all commercial search engines and popular crawlers. As many web site administrators and policy makers have come to rely on the informal contract set forth by the Robots Exclusion Protocol, the degree to which web crawlers respect robots.txt policies has become an important issue of computer ethics. In this research, we investigate and define rules to measure crawler ethics, referring to the extent to which web crawlers respect the regulations set forth in robots.txt configuration files. We test the behaviors of web crawlers in terms of ethics by deploying a crawler honeypot: a set of websites where each site is configured with a distinct regulation specification using the Robots Exclusion Protocol in order to capture specific behaviors of web crawlers.We propose a vector space model to represent crawler behavior and a set of models to measure the ethics of web crawlers based on their behaviors. The results show that ethicality scores vary significantly among crawlers. Most commercial web crawlers receive fairly low ethicality violation scores which means most of the crawlers' behaviors are ethical; however, many commercial crawlers still consistently violate or misinterpret certain robots.txt rules.
ISBN:9781424484829
1424484820
DOI:10.1109/WI-IAT.2010.316