Computer Vision-Based Phenotyping in Agriculture: Leveraging Semantic Information for Non-Destructive Small Crop Analysis

Abstract

Fast and reliable non-destructive phenotyping of plants plays an important role in precision agriculture, as the information enables farmers to make real-time crop management decisions without affecting yield. These decisions encompass a wide range of tasks, including harvesting, disease and pest management, quality control, and scientific breeding.

To non-destructively phenotype crops, computer and stereo-vision based methods are commonly used, as they are low-cost and resolve finer details compared to other systems such as LiDAR. However, most approaches are targeted towards large and sparsely populated crops, where occlusions, wind, and sensor error pose less of a challenge.

In this thesis, we tackle the problem of using computer vision to non-destructively phenotype smaller crops by leveraging semantic information. First, we present a method for creating 3D models of sorghum panicles by using seeds as semantic 3D landmarks. To evaluate performance, we develop an unsupervised metric to assess point cloud reconstruction quality in the absence of ground truth. We then use the model to estimate seed count, and demonstrate that this method outperforms extrapolating counts from 2D images, a common approach used in similar applications.

Next, we present a computer vision-based method to measure sizes and growth rates of apple fruitlets. With images collected by a hand-held stereo camera, our system fits ellipses to fruitlets to measure their diameters. To measure growth rates, we utilize an Attentional Graph Neural Network to associate fruitlets across days. We provide quantitative results on data collected in an apple orchard, and demonstrate that our system is able to predict abscise rates within 3% of the current method with a 7 times improvement in speed, while requiring significantly less manual effort.

Finally, we build upon our sizing pipeline by designing a robotic system to make the sizing process fully autonomous. We present a next-best-view planning approach targeted towards sizing smaller fruit. We utilize semantically labeled regions of interest to sample viewpoint candidates, along with an attention-guided information gain mechanism to generate optimal camera poses. Additionally, a dual-map representation is used to improve speed. When sizing, a robust estimation and clustering approach is introduced to associate fruit detections across images. We demonstrate that our system can effectively size small fruit in occluded environments.