The Ethicality of Web Crawlers

Search engines largely rely on web crawlers to collect information from the web. This has led to an enormous amount of web traffic generated by crawlers alone. To minimize negative aspects of this traffic on websites, the behaviors of crawlers may be regulated at an individual web server by implemen...

Full description

Bibliographic Details

Published in:	2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology Vol. 1; pp. 668 - 675
Main Authors:	Sun, Yang, Councill, Isaac G., Giles, C. Lee
Format:	Conference Proceeding
Language:	English
Published:	IEEE 01.08.2010
Subjects:	Costs Crawlers ethicality Ethics Guidelines Privacy Regulation Robots robots.txt Search engines Vectors web crawler ethics
ISBN:	9781424484829, 1424484820
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!