A soft computing approach for benign and malicious web robot detection

•We propose a method called SMART (Soft computing for MAlicious RoboT detection).•The method detects benign and malicious robots, and human visitors to a web server.•SMART selects its features on a particular web server by fuzzy rough set theory.•A graph-based clustering algorithm classifies session...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Expert systems with applications Jg. 87; S. 129 - 140
Hauptverfasser: Zabihimayvan, Mahdieh, Sadeghi, Reza, Rude, H. Nathan, Doran, Derek
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Elsevier Ltd 30.11.2017
Elsevier BV
Schlagworte:
Automation
Clustering
Cybersecurity
File servers
Fuzzy Rough Set Theory
Heuristic methods
Intrusion detection systems
Malicious web agents
Markov clustering algorithm
Privacy
Robots
Set theory
Soft computing
State of the art
Traffic information
Web crawler
Web Robot Detection
Web Robot Detection
Web crawler
Fuzzy Rough Set Theory
Malicious web agents
Markov clustering algorithm
ISSN:0957-4174, 1873-6793
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:•We propose a method called SMART (Soft computing for MAlicious RoboT detection).•The method detects benign and malicious robots, and human visitors to a web server.•SMART selects its features on a particular web server by fuzzy rough set theory.•A graph-based clustering algorithm classifies sessions into the three agent types.•Analyses on web logs suggest state-of-the-art results to detect both robot types. The accurate detection of web robot sessions from a web server log is essential to take accurate traffic-level measurements and to protect the performance and privacy of information on a Web server. Moreover, the irrecoverable risks of visits from malicious robots that intentionally try to evade web server intrusion detection systems, covering-up their visits with fabricated fields in their http request packets, cannot be ignored. To separate both types of robots from humans in practice, analysts turn to heuristic methods or state-of-the-art soft computing approaches that have only been tuned to the specification of a kind of web server. Noting that the landscape of web robot agents is ever changing, and that behavioral patterns and characteristics vary across different web servers, both options are lacking. To overcome this challenge, this paper presents SMART, a soft computing system that simultaneously detects benign and malicious types of robot agents from web server logs and can automatically adapt to the session characteristics of a web server. The results of experiments over some access log file servers, each servicing different domains of the web, demonstrate outperformance of the proposed method on state-of-the-art ones for benign and malicious robot detection.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0957-4174
1873-6793
DOI:10.1016/j.eswa.2017.06.004