A configurable rule based classful token bucket filter network request scheduler for the lustre file system

HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remo...

Full description

Saved in:
Bibliographic Details
Published in:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) pp. 1 - 12
Main Authors: Qian, Yingjin, Li, Xi, Ihara, Shuichi, Zeng, Lingfang, Kaiser, Jürgen, Süß, Tim, Brinkmann, André
Format: Conference Proceeding
Language:English
Published: New York, NY, USA ACM 12.11.2017
Series:ACM Conferences
Subjects:
Bandwidth
File systems
Filtering algorithms
High performance computing
Information systems > Information storage systems > Storage architectures > Distributed storage
Information systems > Information storage systems > Storage management
Limiting
Lustre
Parallel File Systems
QoS
Quality of service
Real-time systems
Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > File systems management
parallel file systems
quality of service
QoS
lustre
ISBN:9781450351140, 145035114X
ISSN:2167-4337
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remote Procedure Call (RPC) rate limitations based on (potentially complex) Quality of Service (QoS) rules. The QoS rules are enforced in Lustre's Object Storage Servers, where each request is assigned to an automatically created QoS class. The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC rate even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
ISBN:9781450351140
145035114X
ISSN:2167-4337
DOI:10.1145/3126908.3126932