Since the early 2000 he has been working on the integration of sensorimotor information with the conceptual system, bridging the gap between signals and symbols. This led to the introduction of language tools into the Robotics community. During the past five years his research is supported by the European Union under the cognitive systems program in the projects POETICON and POETICON++ , by the National Science Foundation under the Cyber Physics Systems Program in the project Robots with Vision that find objects and by the National Institues of Health in the project Human Activity Languages.
Here is an example of going from language to action (ask a robot to do something). Note how the robot announces that he has to think for a moment, before performing the action but does not reveal its thinking. Here, some of the thinking is revealed.
For the dual problem of going from action to language (observing an activity and describing in natural language what is going on), see our demos in the Telluride Neuromorphic Cognition Engineering workshops .