Tactile, acoustic, and vision sensors provide robots with perceptual capabilities, enabling them to explore and analyze their environment in order to behave more intelligently. The intelligent robot system KAMRO uses several camera systems to generate an environment model in order to perform complex assembly tasks. To interact with this robot system, it would be an insufficient proposal to use natural language just as a command-language for the operator of a robot system. The reason is that incomplete information must may completed by the human operator if the robot is not able to generate the missing information from its sensors or knowledge base. Four main situations of human-machine interaction can be distinguished in the context of natural language access to the Kamro system:
- Task specification: Operations and tasks which are to be performed by the robot can be specified on different levels. On high level the commands are like “assemble Canfield Benchmark” The lowest level are explicit elementary operations as “fin emotion” or “grasp”. This is useful if an operator likes to control the robot or its components directly like teleportation. On a higher level, the operator can also specify a suitable assembly sequence, in order to perform a certain task. Several different descriptions can lead towards an assembly precedence graph generated automatically from the sentences. The most abstract specification would just provide a qualitative description of the desired state, i.e., the final positions of the different assembly parts.
- Execution monitoring: The capability of autonomous behavior gives the robot systems the ability to work up an assembly mission in different orders. Therefore, it is important to inform the operator what part of the mission the robot performs. Depending on the demands formulated by the user, it is possible to give the descriptions and explanations in more or less detail.
- Explanation of error recovering: The capability of recovering from error situations leads to dynamic adjustment of assembly plans during execution. This feature makes it more difficult for the operator to predict and understand the behavior of the robot. Natural language explanations and descriptions of why and how a certain plan was changed would increase the cooperativeness of the intelligent robot. Similar descriptions are even more important in error situations which can not be autonomously handled by the robot.
- Updating the environment representation: Because of the dynamic nature of the environment, geometric and visual information can never be complete, even for an autonomous robot equipped with several sensors. In a natural-language dialogue, user and robot can aid each other by providing additional information about the environment and world model.
To completely implement the interaction possibilities on a high level, it would be necessary to take all four above mentioned situations into account.