DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels

Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We cre...

Full description

Saved in:
Bibliographic Details
Published in:eLife Vol. 10
Main Authors: Bohnslav, James P, Wimalasena, Nivanthika K, Clausing, Kelsey J, Dai, Yu Y, Yarmolinsky, David A, Cruz, Tomás, Kashlan, Adam D, Chiappe, M Eugenia, Orefice, Lauren L, Woolf, Clifford J, Harvey, Christopher D
Format: Journal Article
Language:English
Published: England eLife Sciences Publications Ltd 02.09.2021
eLife Sciences Publications, Ltd
Subjects:
Animal behavior
Animals
behavior analysis
Biomedical research
Classification
computer vision
Datasets
deep learning
Drosophila melanogaster
Female
Grooming
Humans
Image Processing, Computer-Assisted
Kinetics
Learning algorithms
Machine learning
Male
Mars
Mice
Mice, Inbred C57BL
Motor Activity
Neural networks
Neural Networks, Computer
Neuroscience
Pattern Recognition, Automated
Reproducibility of Results
Researchers
Social Behavior
Software
Supervised Machine Learning
Tools and Resources
Walking
deep learning
mouse
D. melanogaster
neuroscience
behavior analysis
computer vision
ISSN:2050-084X, 2050-084X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram’s rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram .
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ObjectType-Undefined-3
ISSN:2050-084X
2050-084X
DOI:10.7554/eLife.63377