When and why PINNs fail to train: A neural tangent kernel perspective

•We analyze the training dynamics of PINNs using neural tangent kernel theory.•We derive the NTK of PINNs and study its limiting behavior.•Our analysis reveals a remarkable discrepancy in the convergence rate of PINNs loss functions.•We propose a novel NTK-guided gradient descent algorithm to effect...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Journal of computational physics Jg. 449; S. 110768
Hauptverfasser: Wang, Sifan, Yu, Xinling, Perdikaris, Paris
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Cambridge Elsevier Inc 15.01.2022
Elsevier Science Ltd
Schlagworte:
Algorithms
Computational physics
Convergence
Eigenvalues
Empirical equations
Gradient descent
Inverse problems
Kernels
Lenses
Multi-task learning
Neural networks
Partial differential equations
Physics-informed neural networks
Scientific machine learning
Spectral bias
Training
Spectral bias
Gradient descent
Scientific machine learning
Multi-task learning
Physics-informed neural networks
ISSN:0021-9991, 1090-2716
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:•We analyze the training dynamics of PINNs using neural tangent kernel theory.•We derive the NTK of PINNs and study its limiting behavior.•Our analysis reveals a remarkable discrepancy in the convergence rate of PINNs loss functions.•We propose a novel NTK-guided gradient descent algorithm to effectively resolve this pathology.•We demonstrate significant and consistent accuracy improvements across a range of benchmarks. Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/PINNsNTK.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0021-9991
1090-2716
DOI:10.1016/j.jcp.2021.110768