Deep Learning-Based Approach for Optical Microrobot Tracking, Pose Prediction and Trapping Points Localisation

Abstract

Mobile micro-robots enables precise manipulation of micrometer scale biological objects, and offer new opportunities in biomedical applications. Optical microrobots are micron-sized structures actuated in a liquid medium by an external laser. This paper presents a deep learning-based approach for optical robots tracking, with the aim of implementing visual control-loops for task automatisation. The system is designed to allow the detection and tracking of robots, of trapping positions (TPs) and measuring the microrobot rotation angles under an optical microscope. YOLOv7 and Deep SORT algorithms was integrated and specially optimized for this specific environment. The proposed model demonstrated 3% higher mean average precision (mAP) at the 0.5:0.95 Intersection over Union (IoU) threshold for training and also sim3% and sim 11% increase in accuracy for both microrobot and TPs, respectively at the 0.95 IoU threshold in our test set. To validate the effectiveness of our TPs detection results, we compared them with potential circle detection methods, demonstrating a success rate of 99% in trapping events with zero false positive detections. Finally, our proposed rotation angle tracking results were verified successfully in two different case scenarios, involving deformed and undeformed microrobots at different speed profiles. Our analysis reveals that the error rate is significantly increased by velocity-induced drag force. © 2023 Elsevier B.V., All rights reserved.