Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders

Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful informati...

Full description

Saved in:
Bibliographic Details
Published in:PLoS computational biology Vol. 17; no. 9; p. e1009439
Main Authors: Whiteway, Matthew R., Biderman, Dan, Friedman, Yoni, Dipoppa, Mario, Buchanan, E. Kelly, Wu, Anqi, Zhou, John, Bonacchi, Niccolò, Miska, Nathaniel J., Noel, Jean-Paul, Rodriguez, Erica, Schartner, Michael, Socha, Karolina, Urai, Anne E., Salzman, C. Daniel, Cunningham, John P., Paninski, Liam
Format: Journal Article
Language:English
Published: United States Public Library of Science 22.09.2021
Public Library of Science (PLoS)
Subjects:
Algorithms
Animal behavior
Animals
Artificial Intelligence - statistics & numerical data
Behavior, Animal
Biology and Life Sciences
Brain research
Cameras
Computational Biology
Computer and Information Sciences
Computer Simulation
Computer vision
Datasets
Encoders
Engineering and Technology
Feature extraction
Information processing
Machine learning
Machine vision
Markov Chains
Medicine and Health Sciences
Mice
Models, Statistical
Nervous system
Neural networks
Neural Networks, Computer
Principal components analysis
Production methods
Social Sciences
Supervised Machine Learning - statistics & numerical data
Unsupervised Machine Learning - statistics & numerical data
Video compression
Video data
Video Recording - statistics & numerical data
United States
ISSN:1553-7358, 1553-734X, 1553-7358
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.
Bibliography:new_version
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
The authors have declared that no competing interests exist.
ISSN:1553-7358
1553-734X
1553-7358
DOI:10.1371/journal.pcbi.1009439