Real-time Malaysian Sign Language recognition system using Microsoft Kinect 360 based on locally linear embedding and artificial neural network model

Abstract

Deaf people or people with hearing loss have a major problem in everyday communication. Sign Language (SL) is a common communication method for deaf people. Many attempts have been made with SL translator to solve of communication gap between normal and deaf people and ease communication for deaf people. The system is able to match and compare the input sign trajectory with each of the prototype sign trajectory contained in the database with lower error rate. This is achieved by extracting a number of static and dynamic features from right hand and left hand. This contribution tries to introduce an SL translator, especially for static and dynamic MSL by using Kinect 360 technology and Native signers with MSL database which have been created in this research. Iterative method has been used for data denoising for depth information. The result for denoising data has been reduce from 307200 to 160000 value. HOG and GA are used as feature extraction for static sign recognition. SVM classifier is used for training and testing the developed system using static signs. Accuracy result for static signs using HOG is 99.37%, GA is 62.92% and GA+HOG is 93.14%. LLE and PCA feature extraction has been used for dynamic sign recognition which improved accuracy result much better (it is mentioned that LLE features have been used for the first time for dynamic sign recognition). Three types of classifier such as MLP, CFNN and SVM are used to test and implement dynamic sign recognition. Accuracy results are 92.30%, 88.50% and 82.70% for MLP, CFNN and SVM respectively. The developed MSL recognition system was tested using 10 dynamic words and 24 static alphabets. The developed MSL recognition system has attained a significant performance in terms of recognition accuracy and speed that allow a real time translation of signs into text.