Development of a prototype vision-based Malaysian sign language recognition system

Abstract

Everyday communication with the hearing population poses a major challenge to those with hearing loss. Sign Language (SL) is used as the preferable language for many people who were either born with hearing/speech impairment or became hard of hearing at an early age. Automatic SL translators are a challenging problem in the domain of image processing and computer vision which have tremendous efforts for translating the lexical form of sign gestures and developing the algorithms that scale effectively to large vocabularies. In this research, a vision-based automatic sign language translator for Malaysian Sign Language (MSL) is developed. The system utilizes the MSL database developed in this research in which the videos were collected from native signers. The MSL recognition system uses SL images and videos with bare hands and has been developed through four stages; face and hands detection and tracking, Human Upper Body (HUB) detection, feature extraction, and real-time SL recognition using Hidden Markov Model (HMM). A novel technique for combining appearance-based method with the colour space YCbCr techniques for the achievement of real-time blobs detection and tracking have been developed. In addition, the system is able to match and compare the input sign trajectory with each of the prototype sign trajectories contained in the database with lower error rate. This is achieved by extracting seven geometric and eight motion features from head, right hand and left hand. Furthermore, the location of hand with respect to the head and other Human Upper Body (HUB) parts conveys a lot of meanings in understanding SL. Therefore, a fast and robust algorithm for detecting and tracking HUB parts is introduced based on a figure adjusted for drawing artists. HMM was used for training and testing the developed system using isolated and continuous signs. Experiments were conducted over 100 times for 37 isolated signs using 38 feature vectors, making it in total of 3800 experiments. For continuous signs, experiments have been repeated 50 times for 202 and 172 sentence sets. The developed MSL recognition system was tested using 20 words and 20 sentences with lexicon of 37 words. The research observed that, the feature vector combination is much important than the feature vector dimension. Based on this, the system recognition accuracy reached up to 80% and 55% for isolated and continuous signs, respectively. The developed MSL recognition system has attained a significant performance in terms of recognition accuracy and speed that allows a real-time translation of signs into text and/or voice (in English).