A configurable rule based classful token bucket filter network request scheduler for the lustre file system

HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remo...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) s. 1 - 12
Hlavní autoři: Qian, Yingjin, Li, Xi, Ihara, Shuichi, Zeng, Lingfang, Kaiser, Jürgen, Süß, Tim, Brinkmann, André
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: New York, NY, USA ACM 12.11.2017
Edice:ACM Conferences
Témata:
Bandwidth
File systems
Filtering algorithms
High performance computing
Information systems > Information storage systems > Storage architectures > Distributed storage
Information systems > Information storage systems > Storage management
Limiting
Lustre
Parallel File Systems
QoS
Quality of service
Real-time systems
Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > File systems management
parallel file systems
quality of service
QoS
lustre
ISBN:9781450351140, 145035114X
ISSN:2167-4337
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remote Procedure Call (RPC) rate limitations based on (potentially complex) Quality of Service (QoS) rules. The QoS rules are enforced in Lustre's Object Storage Servers, where each request is assigned to an automatically created QoS class. The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC rate even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
ISBN:9781450351140
145035114X
ISSN:2167-4337
DOI:10.1145/3126908.3126932