Over the past decades, the exponential growth of technology has brought around a new field called robotic vision, which is a subfield of robotics. Its main idea is to give robots the ability to gain vision by capturing images and processing them. A simple example of robotic vision in everyday life would be QR code scanning for purchases in the supermarket. However, this is nowhere close to its potential application. As a result, it is deeply researched and it is now utilized in fields such as biomedical engineering, and its application will only get broader as more research is conducted.

Setup of Object Detection and Robot Arm Workflow

Stage 1: Dataset collection

Stage 2: Image and Feature Extraction

Stage 3: Coordinate Callibration and Target coordinate processing

OpenCV is used to grab a single frame in the stack of images, which further creates the live-streaming through the camera and processing each frame through the object detector trained using the FasterRCNN ResNet 50 [2] COCO model.

Image and Feature Extraction: Faster-RCNN ResNet

Using the Faster-RCNN ResNet model, the images were used as input and trained for about 4 hours to reach a loss of about below 0.05. 

Inverse Kinematics: Target Coordinate Processing

Using the 2D inverse kinematics library, FABRIK2D in Arduino, which is adapted from the iterative inverse kinematics solver FABRIK [4], the coordinates parsed are mapped into the minimum and maximum coordinates of the eye from the calibration and passed into the solve function of FABRIK2D to get the individual angles of the joints. Defining PWM (pulse width modulation) for each servo the angles are mapped to an equivalent 0 to 180 degrees in terms of PWM. Updating the arm from each change of coordinates is done through changing the servo angles each time the coordinates are parsed and processed.

Conclusion

The medical world can be greatly improved with the aid of robotic vision, and this project is only a basic demonstration of it. From aiding patients suffering paralysis to a substitute for a prosthetic arm, the opportunity of further development can be helpful in complex situations such as surgeries, which could be life-changing.