Robotic Vision for 3D Modeling and Sizing in Agriculture

Abstract

Obtaining accurate perceptual information is a critical component in agricultural robotics since there is a heavy need for interaction with the environment to perform tasks such as pruning, harvesting, and phenotyping. In this thesis, we tackle the problem of perception and 3D modeling in agriculture through the use of stereo cameras in the context of three applications: apple fruitlets sizing, online 3D reconstruction through in-hand camera manipulation, and sorghum field 3D modeling.

First, a deep-learning-based vision system aimed at measuring apple fruitlets in the field is presented achieving an accuracy of less than 1mm compared to ground truth measurements.

The problem of sorghum field 3D modeling is then tackled using Simultaneous Localization and Mapping (SLAM) techniques. An object-level feature association algorithm is proposed that enables the creation of 3D reconstructions robustly by taking advantage of the structure of robotic navigation in agricultural fields. An object-level SLAM system is presented that utilizes recent advances in deep learning-based object detection and segmentation algorithms to detect and segment semantic objects in the environment which are used as landmarks for SLAM. The SLAM system does not use inertial sensory measurements and only relies on visual odometry from a stereo camera capturing images at the frame rate of 5Hz. The object-based feature association algorithm enabled mapping 78% of a sorghum range on average in contrast with traditional visual features which have an average mapped distance of 38%. The system is also compared against ORB-SLAM2, a state-of-the-art visual SLAM algorithm, and shows significant performance improvement in the average mapped distance metric. Finally, we tackle the problem of 3D reconstruction and mapping through the use of an in-hand camera attached to a mobile robot. A planning strategy to perform robotic arm scanning, a 3D reconstruction system, and preliminary apple fruitlet mapping strategies are proposed. The system was deployed in the field and used to autonomously scan tree canopies, collect datasets, and build 3D reconstructions in apple orchards.