Enhancing Biometric Security with Bimodal Deep Learning and Feature-level Fusion of Facial and Voice Data

Authors

DOI:

https://doi.org/10.26636/jtit.2024.4.1754

Keywords:

biometric recognition, deep learning, multimodal systems, SincNet, voice modality

Abstract

Recent research in biometric technologies underscores the benefits of multimodal systems that use multiple traits to enhance security by complicating the replication of samples from genuine users. To address this, we present a bimodal deep learning network (BDLN or BNet) that integrates facial and voice modalities. Voice features are extracted using the SincNet architecture, and facial image features are obtained from convolutional layers. Proposed network fuses these feature vectors using either averaging or concatenation methods. A dense connected layer then processes the combined vector to produce a dual-modal vector that encapsulates distinctive user features. This dual-modal vector, processed through a softmax activation function and another dense connected layer, is used for identification. The presented system achieved an identification accuracy of 99% and a low equal error rate (EER) of 0.13% for verification. These results, derived from the VidTimit and BIOMEX-DB datasets, highlight the effectiveness of the proposed bimodal approach in improving biometric security.

Downloads

Download data is not yet available.

References

[1] S.A. Abdulrahman and B. Alhayani, "A Comprehensive Survey on the Biometric Systems Based on Physiological and Behavioral Characteristics", Materials Today: Proceedings, vol. 80, pp. 2642-2646, 2023.
View in Google Scholar DOI: https://doi.org/10.1016/j.matpr.2021.07.005

[2] S.K.S. Modak and V.K. Jha, "Multibiometric Fusion Strategy and its Applications: A Review", Information Fusion, vol. 49, pp. 174-204, 2019.
View in Google Scholar DOI: https://doi.org/10.1016/j.inffus.2018.11.018

[3] D. Patel, S. Patel, A.A. Thadeshwar, and R. Chaturvedi, "Multimodal Biometric Systems: A Review", International Journal of Advanced Research in Computer Science, vol. 9, no. 2, pp. 361-365, 2018.
View in Google Scholar DOI: https://doi.org/10.26483/ijarcs.v9i2.5742

[4] H. Mandalapu et al., "Audio-visual Biometric Recognition and Presentation Attack Detection: A Comprehensive Survey", IEEE Access, vol. 9, pp. 37431-37455, 2021.
View in Google Scholar DOI: https://doi.org/10.1109/ACCESS.2021.3063031

[5] M. Singh, R. Singh, and A. Ross, "A Comprehensive Overview of Biometric Fusion", Information Fusion, vol. 52, pp. 187-205, 2019.
View in Google Scholar DOI: https://doi.org/10.1016/j.inffus.2018.12.003

[6] N. Alay and H.H. Al-Baity, "Deep Learning Approach for Multimodal Biometric Recognition System Based on Fusion of Iris, Face, and Finger Twenty Traits", Sensors, vol. 20, no. 19, art. no. 5523, 2020.
View in Google Scholar DOI: https://doi.org/10.3390/s20195523

[7] S. Shakil, D. Arora, and T. Zaidi, "Feature Based Classification of Voice Based Biometric Data Through Machine Learning Algorithm", Materials Today: Proceedings, vol. 51, pp. 240-247, 2022.
View in Google Scholar DOI: https://doi.org/10.1016/j.matpr.2021.05.261

[8] N.D. Al-Shakarchy, H.K. Obayes, and Z.N. Abdullah, "Person Identification Based on Voice Biometric Using Deep Neural Network", International Journal of Information Technology, vol. 15, no. 2, pp. 789-795, 2023.
View in Google Scholar DOI: https://doi.org/10.1007/s41870-022-01142-1

[9] N.K. Benamara, E. Zigh, T.B. Stambouli, and M. Keche, "Towards a Robust Thermal-visible Heterogeneous Face Recognition Approach Based on a Generative Cycle Adversarial Network", International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 4, pp. 132-145, 2022.
View in Google Scholar DOI: https://doi.org/10.9781/ijimai.2021.12.003

[10] D.M. Jiménez-Bravo et al., "Edge Face Recognition System Based on One-shot Augmented Learning", International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 6, pp. 31-44, 2022.
View in Google Scholar DOI: https://doi.org/10.9781/ijimai.2022.09.001

[11] A. Alcaide et al., "LIPSNN: A Light Intrusion-proving Siamese Neural Network Model for Facial Verification", International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 4, pp. 121-131, 2022.
View in Google Scholar DOI: https://doi.org/10.9781/ijimai.2021.11.003

[12] V. Talreja, M.C. Valenti, and N.M. Nasrabadi, "Multibiometric Secure System Based on Deep Learning", 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Montreal, Canada, 2017.
View in Google Scholar DOI: https://doi.org/10.1109/GlobalSIP.2017.8308652

[13] Q. Zhang, H. Li, Z. Sun, and T. Tan, "Deep Feature Fusion for Iris and Periocular Biometrics on Mobile Devices", IEEE Transactions on Information Forensics and Security, vol. 13, no. 11, pp. 2897-2912, 2018.
View in Google Scholar DOI: https://doi.org/10.1109/TIFS.2018.2833033

[14] Y. Xin et al., "Multimodal Feature-level Fusion for Biometrics Identification System on IoMT Platform", IEEE Access, vol. 6, pp. 21418-21426, 2018.
View in Google Scholar DOI: https://doi.org/10.1109/ACCESS.2018.2815540

[15] V.V. Khryashchev, A.I. Topnikov, A.F. Stefanidi, and A.L. Priorov, "Bimodal Person Identification Using Voice Data and Face Images", Eleventh International Conference on Machine Vision (ICMV 2018), Munich, Germany, 2019.
View in Google Scholar DOI: https://doi.org/10.1117/12.2523138

[16] A. Abozaid, A. Haggag, H. Kasban, and M. Eltokhy, "Multimodal Biometric Scheme for Human Authentication Technique Based on Voice and Face Recognition Fusion", Multimedia Tools and Applications, vol. 78, pp. 16345-16361, 2019. -0.3pt
View in Google Scholar DOI: https://doi.org/10.1007/s11042-018-7012-3

[17] O. Olazabal et al., "Multimodal Biometrics for Enhanced IoT Security", 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, USA, 2019.
View in Google Scholar DOI: https://doi.org/10.1109/CCWC.2019.8666599

[18] X. Zhang et al., "An Efficient Android-based Multimodal Biometric Authentication System with Face and Voice", IEEE Access, vol. 8, pp. 102757-102772, 2020.
View in Google Scholar DOI: https://doi.org/10.1109/ACCESS.2020.2999115

[19] E. Al Alkeem et al., "Robust Deep Identification Using ECG and Multimodal Biometrics for Industrial Internet of Things", Ad Hoc Networks, vol. 121, art. no. 102581, 2021.
View in Google Scholar DOI: https://doi.org/10.1016/j.adhoc.2021.102581

[20] M. Leghari et al., "Deep Feature Fusion of Fingerprint and Online Signature for Multimodal Biometrics", Computers, vol. 10, no. 2, art. no. 21, 2021.
View in Google Scholar DOI: https://doi.org/10.3390/computers10020021

[21] C.F.F. Costa-Filho, J.V. Negreiro, and M.G.F. Costa, "Multimodal Biometric System Based on Autoencoders and Learning Vector Quantization", Brazilian Congress on Biomedical Engineering, Vitoria, Brazil, 2020.
View in Google Scholar

[22] C. Kamlaskar and A. Abhyankar, "Feature Level Fusion Framework for Multimodal Biometric System Based on CCA with SVM Classifier and Cosine Similarity Measure", Australian Journal of Electrical and Electronics Engineering, vol. 20, no. 2, pp. 205-218, 2023.
View in Google Scholar DOI: https://doi.org/10.1080/1448837X.2022.2129147

[23] Z. Zhang, H. Lu, P. Sang, and J. Wang, "MultiBioGM: A Hand Multimodal Biometric Model Combining Texture Prior Knowledge to Enhance Generalization Ability", in: Biometric Recognition (CCBR 2023), pp. 106-115, 2023.
View in Google Scholar DOI: https://doi.org/10.1007/978-981-99-8565-4_11

[24] V. Gurunathan and R. Sudhakar, "Multimodal Biometric System Using Palm Vein and Ear Images", Proceeding of International Conference on Computer Visions and Robotics, pp. 439-451, 2023.
View in Google Scholar DOI: https://doi.org/10.1007/978-981-99-4577-1_36

[25] T. Hafs, H. Zehir, A. Hafs, and A. Nait-Ali, "Multimodal Biometric System Based on the Fusion in Score of Fingerprint and Online Handwritten Signature", Applied Computer Systems, vol. 28, no. 1, pp. 37-49, 2023.
View in Google Scholar DOI: https://doi.org/10.2478/acss-2023-0006

[26] M. Ravanelli and Y. Bengio, "Speaker Recognition from Raw Waveform with SincNet", 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece, 2018.
View in Google Scholar DOI: https://doi.org/10.1109/SLT.2018.8639585

[27] Y. Badr, P. Mukherjee, and S.M. Thumati, "Speech Emotion Recognition using MFCC and Hybrid Neural Networks", Proceedings of the 13th International Joint Conference on Computational Intelligence, pp. 366-373, 2021.
View in Google Scholar DOI: https://doi.org/10.5220/0010707400003063

[28] A.K. Dubey and V. Jain, "Comparative Study of Convolution Neural Network's ReLU and Leaky-ReLU Activation Functions", in: Applications of Computing, Automation and Wireless Systems in Electrical Engineering, pp. 873-880, 2019.
View in Google Scholar DOI: https://doi.org/10.1007/978-981-13-6772-4_76

[29] D.B. Jadhav, G.S. Chavan, V.C. Bagal, and R.R. Manza, "Review on Multimodal Biometric Recognition System Using Machine Learning", Artificial Intelligence and Applications, vol. 20, pp. 1-7, 2023.
View in Google Scholar

[30] C. Sanderson and B.C. Lovell, "Multi-region Probabilistic Histograms for Robust and Scalable Identity Inference", in: Advances in Biometrics (Conference Proceedings), pp. 199-208, 2009.
View in Google Scholar DOI: https://doi.org/10.1007/978-3-642-01793-3_21

[31] D. Snyder, D. Povey, and G. Chen, "MUSAN: A Music, Speech, and Noise Corpus", ArXiv, 2015.
View in Google Scholar

[32] A. Zelinsky, "Learning OpenCV-Computer Vision with the OpenCV Library", IEEE Robotics & Automation Magazine, vol. 16, no. 3, p. 100, 2009.
View in Google Scholar DOI: https://doi.org/10.1109/MRA.2009.933612

[33] M. Wang, Z. Wang, and J. Li, "Deep Convolutional Neural Network Applies to Face Recognition in Small and Medium Databases", 2017 4th International Conference on Systems and Informatics (ICSAI), Hangzhou, China, 2017.
View in Google Scholar DOI: https://doi.org/10.1109/ICSAI.2017.8248499

[34] P. Ke, M. Cai, H. Wang, and J. Chen, "A Novel Face Recognition Algorithm Based on the Combination of LBP and CNN", 2018 14th IEEE International Conference on Signal Processing (ICSP), Beijing, China, 2018.
View in Google Scholar DOI: https://doi.org/10.1109/ICSP.2018.8652477

[35] Q. Xu and N. Zhao, "A Facial Expression Recognition Algorithm Based on CNN and LBP Feature", 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chongqing, China, 2020.
View in Google Scholar DOI: https://doi.org/10.1109/ITNEC48623.2020.9084763

[36] A.B. Jung et al., "Imgaug", GitHub: San Francisco, USA, 2020 (https://github.com/aleju/imgaug).
View in Google Scholar

[37] J.-M. Cheng and H.-C. Wang, "A Method of Estimating the Equal Error Rate for Automatic Speaker Verification", 2004 International Symposium on Chinese Spoken Language Processing, Hong Kong, China, 2004.
View in Google Scholar

[38] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition", IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016.
View in Google Scholar DOI: https://doi.org/10.1109/CVPR.2016.90

[39] I. Aliyu, M.A. Bomoi, and M. Maishanu, "A Comparative Study of Eigenface and Fisherface Algorithms Based on OpenCV and Sci-kit Libraries Implementations", International Journal of Information Engineering & Electronic Business, vol. 14, no. 3, pp. 30-40, 2022.
View in Google Scholar DOI: https://doi.org/10.5815/ijieeb.2022.03.04

[40] D. Snyder et al., "X-vectors: Robust DNN Embeddings for Speaker Recognition", 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, Canada, 2018.
View in Google Scholar DOI: https://doi.org/10.1109/ICASSP.2018.8461375

[41] Y. Kortli, M. Jridi, A. Al Falou, and M. Atri, "Face Recognition Systems: A Survey", Sensors, vol. 20, no. 2, art. no. 342, 2020.
View in Google Scholar DOI: https://doi.org/10.3390/s20020342

[42] A. Verma, A. Goyal, N. Kumar, and H. Tekchandani, "Face Recognition: A Review and Analysis", in: Computational Intelligence in Data Mining (Conference Proceedings), pp. 195-210, 2022.
View in Google Scholar DOI: https://doi.org/10.1007/978-981-16-9447-9_15

[43] Z. Bai and X. L. Zhang, "Speaker Recognition Based on Deep Learning: An Overview", Neural Networks, vol. 140, pp. 65-99, 2021.
View in Google Scholar DOI: https://doi.org/10.1016/j.neunet.2021.03.004

[44] A.Q. Ohi, M.F. Mridha, M.A. Hamid, and M.M. Monowar, "Deep Speaker Recognition: Process, Progress, and Challenges", IEEE Access, vol. 9, pp. 89619-89643, 2021.
View in Google Scholar DOI: https://doi.org/10.1109/ACCESS.2021.3090109

[45] M. Gofman et al., "Multimodal Biometrics via Discriminant Correlation Analysis on Mobile Devices", International Conference on Security and Management (SAM), Las Vegas, USA, 2018.
View in Google Scholar

[46] R. Ramachandra et al., "Smartphone Multimodal Biometric Authentication: Database and Evaluation", arXiv:1912.02487, 2019.
View in Google Scholar

[47] G. Antipov, N. Gengembre, O.L. Blouch, and G.L. Lan, "Automatic Quality Assessment for Audio-visual Verification Systems: The LOVe Submission to NIST SRE Challenge 2019", arXiv:2008.05889, 2020.
View in Google Scholar DOI: https://doi.org/10.21437/Interspeech.2020-1434

[48] S.O. Sadjadi et al., "The 2019 NIST Audio-visual Speaker Recognition Evaluation", The Speaker and Language Recognition Workshop: Odyssey 2020, Tokyo, Japan, 2020.
View in Google Scholar DOI: https://doi.org/10.21437/Odyssey.2020-38

[49] M. Liu et al., "Exploring Deep Learning for Joint Audio-visual Lip Biometrics", arXiv:2104.08510, 2021.
View in Google Scholar

[50] G. Fenu and M. Marras, "Demographic Fairness in Multimodal Biometrics: A Comparative Analysis on Audio-visual Speaker Recognition Systems", Procedia Computer Science, vol. 198, pp. 249-254, 2022.
View in Google Scholar DOI: https://doi.org/10.1016/j.procs.2021.12.236

[51] M.S. Saeed et al., "Single-branch Network for Multimodal Training", 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023.
View in Google Scholar DOI: https://doi.org/10.1109/ICASSP49357.2023.10097207

[52] G.P. Rajasekhar and J. Alam, "Audio-visual Speaker Verification via Joint Cross-attention", International Conference on Speech and Computer, Dharwad, India, 2023.
View in Google Scholar DOI: https://doi.org/10.1007/978-3-031-48312-7_2

[53] R. Tao et al., "Multi-stage Face-voice Association Learning with Keynote Speaker Diarization", arXiv:2407.17902, 2024.
View in Google Scholar DOI: https://doi.org/10.1145/3664647.3688980

[54] M. Abdrakhmanova et al., "One Model to Rule Them All: A Universal Transformer for Biometric Matching", IEEE Access, vol. 12, pp. 96729-96739, 2024.
View in Google Scholar DOI: https://doi.org/10.1109/ACCESS.2024.3426602

[55] A. Farhadipour, M. Chapariniya, T. Vukovic, and V. Dellwo, "Comparative Analysis of Modality Fusion Approaches for Audio-visual Person Identification and Verification", arXiv:2409.00562, 2024.
View in Google Scholar

[56] C. Wang, H. Zhu, and L. Xu, "Research on the Improvement of the Target Speaker Recognition System Based on Dual-Modal Fusion", 2024 5th International Conference on Computer Vision, Image and Deep Learning (CVIDL), Zhuhai, China, 2024.
View in Google Scholar DOI: https://doi.org/10.1109/CVIDL62147.2024.10603613

[57] Y. Jiang et al., "Target Speech Diarization with Multimodal Prompts", arXiv:2406.07198 , 2024.
View in Google Scholar DOI: https://doi.org/10.1109/ICASSP48485.2024.10446072

[58] C. Peng, L. He, and D. Su, "Fuse after Align: Improving Face-voice Association Learning via Multimodal Encoder", arXiv:2404.09509, 2024.
View in Google Scholar

Downloads

Submitted

2024-07-22

Accepted

2024-09-18

Published

2024-12-31

Issue

Section

RESEARCH ARTICLES

How to Cite

[1]
K. Merit and M. Beladgham, “Enhancing Biometric Security with Bimodal Deep Learning and Feature-level Fusion of Facial and Voice Data”, JTIT, vol. 98, no. 4, pp. 31–42, Dec. 2024, doi: 10.26636/jtit.2024.4.1754.

Most read articles by the same author(s)