A configurable rule based classful token bucket filter network request scheduler for the lustre file system

HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remo...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) S. 1 - 12
Hauptverfasser: Qian, Yingjin, Li, Xi, Ihara, Shuichi, Zeng, Lingfang, Kaiser, Jürgen, Süß, Tim, Brinkmann, André
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: New York, NY, USA ACM 12.11.2017
Schriftenreihe:ACM Conferences
Schlagworte:
Bandwidth
File systems
Filtering algorithms
High performance computing
Information systems > Information storage systems > Storage architectures > Distributed storage
Information systems > Information storage systems > Storage management
Limiting
Lustre
Parallel File Systems
QoS
Quality of service
Real-time systems
Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > File systems management
parallel file systems
quality of service
QoS
lustre
ISBN:9781450351140, 145035114X
ISSN:2167-4337
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remote Procedure Call (RPC) rate limitations based on (potentially complex) Quality of Service (QoS) rules. The QoS rules are enforced in Lustre's Object Storage Servers, where each request is assigned to an automatically created QoS class. The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC rate even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
ISBN:9781450351140
145035114X
ISSN:2167-4337
DOI:10.1145/3126908.3126932