A configurable rule based classful token bucket filter network request scheduler for the lustre file system

HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remo...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) s. 1 - 12
Hlavní autori: Qian, Yingjin, Li, Xi, Ihara, Shuichi, Zeng, Lingfang, Kaiser, Jürgen, Süß, Tim, Brinkmann, André
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: New York, NY, USA ACM 12.11.2017
Edícia:ACM Conferences
Predmet:
Bandwidth
File systems
Filtering algorithms
High performance computing
Information systems > Information storage systems > Storage architectures > Distributed storage
Information systems > Information storage systems > Storage management
Limiting
Lustre
Parallel File Systems
QoS
Quality of service
Real-time systems
Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > File systems management
parallel file systems
quality of service
QoS
lustre
ISBN:9781450351140, 145035114X
ISSN:2167-4337
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remote Procedure Call (RPC) rate limitations based on (potentially complex) Quality of Service (QoS) rules. The QoS rules are enforced in Lustre's Object Storage Servers, where each request is assigned to an automatically created QoS class. The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC rate even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
ISBN:9781450351140
145035114X
ISSN:2167-4337
DOI:10.1145/3126908.3126932