Conference Paper
Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations
We are happy to announce that our paper “Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations” has been accepted at the 20th International Conference on Principles of Knowledge Representation and Reasoning (KR 2023).
The full text of the paper is accessible in the conference proceedings (Link) and on ArXiv (Link).
Authors: Benjamin Alt, Franklin Kenghagho Kenfack, Andrei Haidu, Darko Katic, Rainer Jäkel, Michael Beetz
There is a growing need for assistance robots that can perform a wide range of everyday manipulation tasks. However, it is impossible to pre-program robots for every situation they may encounter. Instead, robots should be able to learn new tasks from human instruction.
Our system allows non-experts to demonstrate manipulation tasks in virtual reality and then automatically generates executable robot programs from these demonstrations. We leverage techniques from knowledge representation and reasoning to interpret the human actions at a semantic level as sequences of tasks. We represent tasks using a hybrid approach, where task types are defined in an ontology and task semantics are specified as logic rules. This allows us to reason about what tasks may have been demonstrated and what a robot needs to do to achieve them.
Once we interpret the VR demonstration as one or more candidate task sequences, we translate these into underspecified robot plans. We then leverage our cognitive robotics architecture to gradually specialize these plans into executable programs via knowledge-based perception, motion planning, and code generation. This allows our approach to generalize robot programs to new environments and object configurations without retraining.
We experimentally validate our system on two challenging real-world manipulation tasks: fetching objects from supermarket shelves and hanging products. Our quantitative evaluations show that our approach can robustly synthesize working robot programs for a wide variety of human demonstrations. Due to the explicit ontology-based knowledge representation, the same background knowledge supported the synthesis of robot programs for both tasks.
In future work, we aim to extend the range of tasks our system can learn and to incrementally improve programs through interactive learning with humans. We believe our cognition-inspired approach is an important step towards making robots more intuitive to program for nonexpert users.
Conversion of human demonstrations in virtual reality (top right) to executable robot programs by leveraging semantic task knowledge (center right) and knowledge-augmented perception (bottom).