Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms

We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods-Bayesian neural networks (BNNs), concrete dropout (CD), and deep ensembles (DEs) - are compared t...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Machine learning: science and technology Ročník 2; číslo 1; s. 15002 - 15010
Hlavní autoři: Caldeira, João, Nord, Brian
Médium: Journal Article
Jazyk:angličtina
Vydáno: Bristol IOP Publishing 01.03.2021
Témata:
Algorithms
Bayesian inference
Deep learning
ensemble of neural networks
Error analysis
Machine learning
Neural networks
Noise prediction
Physical sciences
Statistical methods
Training
Uncertainty
uncertainty quantification
ISSN:2632-2153, 2632-2153
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods-Bayesian neural networks (BNNs), concrete dropout (CD), and deep ensembles (DEs) - are compared to the standard analytic error propagation. We discuss this comparison in terms endemic to both machine learning ('epistemic' and 'aleatoric') and the physical sciences ('statistical' and 'systematic'). The comparisons are presented in terms of simulated experimental measurements of a single pendulum-a prototypical physical system for studying measurement and analysis techniques. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.
Bibliografie:MLST-100132.R2
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
arXiv:2004.10710; FERMILAB-PUB-20-157-SCD
AC02-07CH11359
USDOE Office of Science (SC), High Energy Physics (HEP)
ISSN:2632-2153
2632-2153
DOI:10.1088/2632-2153/aba6f3