A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard

The vision system of a fruit picking robot must perform two difficult tasks, the accurate pixel-wise instance segmentation of fruit and the correct detection of picking points. Due to the unsatisfactory performance of the vision system on these tasks, the picking performance of the robot is insuffic...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biosystems engineering Jg. 206; S. 32 - 54
Hauptverfasser: Zheng, Chan, Chen, Pengfei, Pang, Jing, Yang, Xiaofan, Chen, Changxin, Tu, Shuqin, Xue, Yueju
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier Ltd 01.06.2021
Schlagworte:
algorithms
Complex scene
data collection
End-to-end framework
Fruit picking robot
fruits
Instance segmentation
lighting
mangoes
Occlusion and overlap
orchards
Picking-point detection
vision
Instance segmentation
Fruit picking robot
Occlusion and overlap
End-to-end framework
Picking-point detection
Complex scene
ISSN:1537-5110, 1537-5129
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The vision system of a fruit picking robot must perform two difficult tasks, the accurate pixel-wise instance segmentation of fruit and the correct detection of picking points. Due to the unsatisfactory performance of the vision system on these tasks, the picking performance of the robot is insufficient and its application is not prevalent. In this study, performing fruit instance segmentation and the localization of picking points, a vision algorithm using RGB images was designed in a deep learning framework for a potential visual system of mango picking robots. The algorithm not only performed these two tasks successfully, but also integrated them into an end-to-end network, in which parallel branches performed the two tasks simultaneously. The average precision (at IoU of 0.75) and the average recall of instance segmentation reached 0.947 and 0.929 respectively, and the best precision and recall of picking-point detection reached 0.984 and 0.908 respectively. In addition, the tasks face various illumination and background interference in outdoor orchards, along with complex problems in terms of occlusion, overlap and object scale. In this study, the performance of the vision system was analysed in detail on several datasets and subsets of the various complex conditions, and the major factors that affect the performance were discussed. The results demonstrated that this system was robust against various illuminations and complex backgrounds, and yielded satisfactory segmentation and picking-point detection performances for minor and medium occlusion or overlap, and for medium and large mangoes. The model visualization and the influence analysis of model training demonstrated the training process and modelling effect of the deep learning network. •Designed an end-to-end vision system for a mango picking robot.•Instance segmentation and pick point detection are performed simultaneously.•It is robust to various illumination and background interference scenarios.•Good performances in minor and medium occlusion, various overlap and object scale.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1537-5110
1537-5129
DOI:10.1016/j.biosystemseng.2021.03.012