Photorealistic Arm Robot Simulation for 3D Plant Reconstruction and Automatic Annotation using Unreal Engine 5

Xingjian Li, Jeremy Park, Chris Reberg-Horton, Steven Mirsky, Edgar Lobaton, Lirong Xiang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5480-5488

Robotics paired with computer vision are widely used in precision agriculture. Simulations are critical for safety and performance estimation by verifying their routine in a virtual world before real-world testing and deployment. However many simulators used in agricultural robots lack photorealism in their virtual worlds compared to the real world. We implemented Unreal Engine 5 (UE5) and the Robot Operating System (ROS) to develop a robot simulator tailored to agricultural tasks and synthetic data generation with RGB segmentation and depth images. We designed a method for assigning multiple segmentation labels within a single plant mesh. We experimented with a semi-spherical routine for two robot arms to perform 3D point cloud reconstruction across 10 plant assets. We showed our simulator produces much more accurate segmentation images and reconstruction compared to existing UE5 solutions. We extend our results with Neural Radiance Field (NeRF) reconstructions. The packaged simulator UE5 project and ROS package with the Python routine can be found at https://github.com/NCSU-BAE-ARLab/AgriRoboSimUE5.