American Sign Language Recognition in Game Development for Deaf Children

Summary

The authors of this paper created CopyCat, an American Sign Language game, which uses gesture recognition to help young deaf children learn sign language. One of the main problems in this area is continuous, user independent sign language recognition in classroom settings. They implemented a Wizard of Oz version of CopyCat and collected data from deaf children who used the system. They attempted to overcome the problems in continuous signing, such as clothing and skin tone differences and changes in illumination in the classroom. The dataset consisted of 541 phrase samples and 1959 individual sign samples.

Their solution used color histogram adaptation for hand segmentation and tracking. The children would wear colored gloves with wireless accelerometers. Both data is used to train hidden Markov models for recognition. The game is interactive. It has tutorial videos to demonstrate the correct signs, live video, and an animated character mimicking what the child is doing. They evaluated the result using the leave-one-out technique, by iterating through each child and removing him from the data set, training the data on the four other children, and testing it on the remaining child's data. They achieved between 73 to 92% accuracy.

Commentary

I think this is a great application for the research area. It is helpful to teach deaf children how to do sign language, especially if their parents aren't fluent in it as well. I think future work will improve on the interface and allow the programmers to get away from the Wizard. Also, the dataset was filtered out to include only the clear signs. In everyday use, that would not be possible. Also, the software needs to deal with individuality in signs. Not everyone would do the signs the exact same way. There are always variations. Overall, though, I enjoyed this paper. A very useful application for HCI.

---------------------------------------------

Helene Brashear, Valerie Henderson, Kwang-Hyn Park, Harley Hamilton, Seungyon Lee, Thad Starner. American Sign Language Recognition in Game Development for Deaf Children. ASSET 2006.