EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation

Perception of transparent and deformable objects is a challenging problem for computer vision, especially in environments like hospitals where tasks like handling transfusion bags require robust grasping.

To address this problem, we developed EfficientPPS, a neural network architecture that performs part-aware panoptic segmentation in a single, efficient model. EfficientPPS builds upon the EfficientPS architecture to combine semantic, instance and part segmentation. Crucially, we introduce a novel part-panoptic fusion module that adaptively blends outputs from different heads to resolve conflicts while leveraging synergies between part and scene-level predictions.

Training data is another big challenge for transparent objects. To reduce annotation effort, we developed an unsupervised data collection pipeline using a robot arm-mounted camera. By acquiring data from multiple viewpoints with varied backgrounds, our method can generate training datasets without human labeling.

We evaluated EfficientPPS on a new dataset containing real hospital items like transfusion bags. Results show our model achieves state-of-the-art part-panoptic segmentation, especially for multipart objects and overlapping scenes. We incorporated EfficientPPS into a robot grasping demonstration, where it reliably predicts grasp points on transfusion bags from different poses.

Overall, our work takes an important step towards robust perception of deformable and transparent objects. EfficientPPS runs efficiently and could enable new applications like hospital logistics robots. Future work includes expanding our model and datasets to handle wider object varieties and more complex scenes.