CHARLIE: An adaptive robot design with hand and face tracking for use in autism therapy

Laura Boccanfuso and Jason M. O'Kane
International Journal of Social Robotics
vol. 3
no. 4
pp. 337–347

Abstract Basic turn-taking and imitation skills are imperative for effective communication and social interaction. Recently, research has demonstrated that interactive games using turn-taking and imitation have yielded positive results with autistic children who have impaired communication or social skills. This paper describes a robot that plays interactive imitation games using hand and face tracking. The robot is equipped with a head and two arms, each with two degrees of freedom, and a camera. We trained a human hands detector and subsequently, used this detector along with a standard face tracker to create two autonomous interactive games: single-player ("Imitate Me, Imitate You") and two-player ("Pass the Pose".) Additionally, we implemented a third setting in which the robot is teleoperated by remote control. In "Imitate Me, Imitate You", the robot has both passive and active game modes. In the passive mode, the robot waits for the child to initiate an interaction by raising one or both hands. In the second game mode, the robot initiates interactions. The "Pass the Pose" game engages two children in cooperative play by enlisting the robot as a mediator between two children alternately initiating and imitating poses. These games are designed to increase attention, promote turn-taking skills and encourage child-led verbal and non-verbal communication through simple imitative play. This research makes two specific contributions: (1) We present a low-cost robot design which measures and adapts to a child's actions during interactive games and, (2) we train, test and make freely available, a new hand detector, based on Haar-like features, which is usable in various kinds of human-robot interactions. We present proof-of-concept experiments with a group of typically developing children.

  author = {Laura Boccanfuso and Jason M. O'Kane},
  journal = {International Journal of Social Robotics},
  number = {4},
  pages = {337--347},
  title = {{CHARLIE}: An adaptive robot design with hand and face
           tracking for use in autism therapy},
  volume = {3},
  year = {2011}

O'Kane's home page
O'Kane's publication list
Last updated 2024-07-02.