Deep learning models for open-world RGB-D face recognition

Abstract

RGB-D based Face Recognition (FR) has grown popular with low-cost depth cameras like Microsoft Kinect, Intel RealSense, and Zed. However, these advances are insufficient for Open-World Face Recognition (OWFR) because the recognition model must identify individuals for whom the FR model was not trained. Developing robust open-world face recognition systems is critical for many practical applications such as cobots, law enforcement, security, and surveillance. Existing FR methods require extensive fine-tuning, classifier retraining, or global metric learning to improve the performance for effective domain adaptation in the open world. These steps are computationally expensive and time consuming. The recognition performance will also significantly degrade when presented with unique individuals. Therefore, it is necessary to develop robust multimodal open-world FR systems using RGB-D cameras without incurring substantial downtime. This thesis proposes three main contributions to the research in RGB-D face recognition. Firstly, the thesis investigates and proposes an RGB-D based FR models suited for the open world, termed CuteFace3D. These robust FR models are built using a multimodal CNN and RGB-D face dataset. The various CNN backbones are investigated for the task. The close-set evaluation on the Intellifusion test dataset is used as the criterion to select a more discriminative FR model as a feature extractor for OWFR. The selected models are then extensively analyzed for an open world on a large dataset of 3D faces. The results imply that deeper networks alone are not discriminative enough for OWFR. The storage is optimized by eliminating the need to save raw RGB-D images, reducing model inference time, and improving data security. A complete FR pipeline is also implemented using a RealSense D435 depth camera. In addition, embeddings are utilized for open-world and unseen domain adaptation with the KNN classifier and k-fold validation, which achieved 99.997% for the open set RGB-D pipeline for domain adaptation. Early fusion with multichannel RGB-D input makes the proposed models robust and accurate in open-world scenarios, with performance equivalent to close-set FR models. Secondly, for OWFR, a fast and efficient adaptive threshold algorithm is developed using an effective Region of Interest (ROI) setting for metric learning. It uses five different ROI schemes to find an adaptive threshold in real-time. After new enrolment the algorithm determines the FR model’s quality and usability. To establish the effectiveness, then benchmarked the proposed method against various threshold-finding strategies for face recognition algorithms for open-world adaptation on different datasets. Experimental results demonstrated that the proposed ROI-based method is up to 12 times faster than the best threshold search algorithm, reporting higher accuracy and fewer errors. Thirdly, this thesis also addresses the performance degradation of the FR model in an open-world setting. A novel performance evaluation metric for FR algorithms on imbalanced datasets is proposed. The proposed metric with an adaptive threshold is more effective than conventional fixed threshold strategies. Thus, this thesis alludes that FR algorithms should be benchmarked for accuracy at the highest F1-score in an open-world. In conclusion, all three contributions increase the effectiveness and efficiency of the proposed FR in terms of computational cost, storage, and security. The proposed method also reduces computational time, making existing FR models operational for OWFR in real-time.