CAPTURE: Memory-Centric Partitioning for Distributed DNN Training with Hybrid Parallelism

Deep Learning (DL) model sizes are increasing at a rapid pace, as larger models typically offer better statistical performance. Modern Large Language Models (LLMs) and image processing models contain billions of trainable parameters. Training such massive neural networks incurs significant memory re...

Full description

Bibliographic Details

Published in:	Proceedings - International Conference on High Performance Computing pp. 76 - 86
Main Authors:	Dreuning, Henk, Verstoep, Kees, Bal, Henri E., van Nieuwpoort, Rob V.
Format:	Conference Proceeding
Language:	English
Published:	IEEE 18.12.2023
Subjects:	Costs Deep Learning GPU Graphics processing units HPC Hybrid Parallelism Memory Memory management Predictive models Tensors Throughput Training
ISSN:	2640-0316
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!