A CNN-Based Hand Gesture and Facial Recognition Interface for Contactless Human-Computer Interaction

Authors

  • Prasad SR Dept. of CSE, Shri Dharmasthala Manjunatheshwara Institute of Technology, Ujire-574240, Affiliated to Visvesvaraya Technological University, Belagavi, Karnataka, India https://orcid.org/0000-0002-3801-5529
  • Charan K Dept. of CSE, Shri Dharmasthala Manjunatheshwara Institute of Technology, Ujire-574240, Affiliated to Visvesvaraya Technological University, Belagavi, Karnataka, India https://orcid.org/0009-0007-5445-9115
  • Ghanashyama K P Makkithaya Dept. of CSE, Shri Dharmasthala Manjunatheshwara Institute of Technology, Ujire-574240, Affiliated to Visvesvaraya Technological University, Belagavi, Karnataka, India https://orcid.org/0009-0000-4306-2445
  • Keerthana M S Dept. of CSE, Shri Dharmasthala Manjunatheshwara Institute of Technology, Ujire-574240, Affiliated to Visvesvaraya Technological University, Belagavi, Karnataka, India https://orcid.org/0009-0003-6968-0411
  • Larine Theresa P Dept. of CSE, Shri Dharmasthala Manjunatheshwara Institute of Technology, Ujire-574240, Affiliated to Visvesvaraya Technological University, Belagavi, Karnataka, India https://orcid.org/0009-0008-1877-502X

DOI:

https://doi.org/10.26438/ijcse/v13i8.1120

Keywords:

Gesture-Based Control, Convolutional Neural Networks (CNNs)

Abstract

Traditional input devices such as the mouse and keyboard, while precise and widely adopted, present limitations in terms of physical strain, accessibility, and adaptability to diverse user needs. To address these challenges, this study proposes a comprehensive, gesture-driven human-computer interaction (HCI) system that integrates four critical components. First, a combination of advanced computer vision algorithms and libraries was employed for real-time skin detection, hand tracking, feature extraction, and gesture-based cursor control. Convolutional Neural Networks (CNNs) were central to the gesture recognition pipeline, enabling accurate and adaptive interpretation of visual input. Second, a secure authentication and authorization mechanism was developed using CNN-based facial recognition in conjunction with hand tracking, ensuring personalized and protected user access. Third, the system features a responsive graphical user interface (GUI), designed using Python’s Tkinter framework, which supports customizable gesture-to-command mappings and provides real-time user feedback. Finally, the system's adaptability and robustness were validated across multiple computing environments. This work contributes to the evolving field of contactless interaction by presenting an integrated framework that emphasizes usability, security, and interoperability, with potential applications in assistive technologies, smart environments, and touch-free computing systems

References

[1] P. Niu, “Convolutional neural network for gesture recognition human-computer interaction system design”, PLoS One, Vol.20, No.2, pp.e0311941, 2025. http://dx.doi.org/10.1371/journal.pone.0311941

[2] H. Kim, J. Lee, and J. Park, “Dynamic hand gesture recognition using a CNN model with 3D receptive fields”, Proc. Int. Conf. Neural Netw. Signal Process., Nanjing, China, pp.14–19, 2008. IEEE.

[3] P. K. Pisharady, P. Vadakkepat, and A. P. Loh, “Attention based detection and recognition of hand postures against complex backgrounds”, International Journal of Computer Vision, Vol.101, pp.403–419, 2013.

[4] M. Bhat, N. Kumar, P. Y. Poojitha, S. Thulasi, and V. Arvind, “CNN based facial recognition with age invariance”, International Journal of Research in Applied Science and Engineering Technology, Vol.11, pp.1061–1065, 2023. http://dx.doi.org/10.22214/ijraset.2023.56680

[5] M. Sharma, G. Akilesh, V. V. Vishwaa, S. F. P. Sharon, and A. Kala, “Virtually controlling computers using hand gesture and voice commands”, Journal of Current Research in Engineering and Science, Vol.5, No.17, 2022.

[6] S. Song, D. Yan, and Y. Xie, “Design of control system based on hand gesture recognition”, Proc. IEEE Int. Conf. Networking, Sensing and Control (ICNSC), 2018.

[7] R. Damdoo, K. Kalyani, and J. Sanghavi, “Adaptive hand gesture recognition system using machine learning approach”, Biosciences Biotechnology Research Communications, Vol.13, No.14, pp.106–110, 2020.

[8] J. J. Beom, S.-K. Kim, and S. Kim, “Enhancing virtual and augmented reality interactions with a MediaPipe–based hand gesture recognition user interface”, Ingénierie des Systèmes d’Information, Vol.28, No.3, pp.633, 2023.

[9] M. Al-Hammadi et al., “Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation”, IEEE Access, Vol.8, pp.192527–192542, 2020.

[10] J. Yu, M. Qin, and S. Zhou, “Dynamic gesture recognition based on convolutional neural network and feature fusion”, Scientific Reports, Vol.12, No.1, pp.4345, 2022.

[11] X. Wang and J. Tanaka, “GesID: 3D gesture authentication based on depth camera and one-class classification”, Sensors, Vol.18, No.10, pp.3265, 2018.

[12] A. Jha et al., “Gessure: A robust face-authentic enabled dynamic gesture recognition GUI application”, International Journal of Cybernetics and Informatics (IJCI), Vol.11, No.11, pp.19, 2022.

[13] J. Shin, M. A. M. Hasan, and M. Maniruzzaman, “Hand gesture authentication using optimal feature selection and dynamic time warping based K-nearest neighbor”, Proc. Int. Conf. Electron., Commun. Control Eng., 2022.

[14] S. Parikh, S. Banka, I. Lautrey, I. Gupta, and D. Yedurkar, “Human-computer interaction using dynamic hand gesture recognition to conveniently control the system”, International Journal of Engineering and Applied Sciences Technology, Vol.5, No.9, 2021. ISSN: 2455-2143.

[15] S. K. Shareef, I. V. S. L. Haritha, Y. L. Prasanna, and G. K. Kumar, “Deep learning based hand gesture translation system”, Proc. 5th Int. Conf. Trends Electron. Informatics (ICOEI), pp.1531–1534, 2021. http://dx.doi.org/10.1109/ICOEI51242.2021.9452947

[16] S. Shahi et al., “Vision-based hand gesture customization from a single demonstration”, arXiv preprint, 2024. https://doi.org/10.1145/3654777.3676378

[17] O. Köpüklü, A. Gunduz, N. Kose, and G. Rigoll, “Real-time hand gesture detection and classification using convolutional neural networks”, Proc. 14th IEEE Int. Conf. Automatic Face & Gesture Recognition (FG), Lille, France, pp.1–8, 2019. http://dx.doi.org/10.1109/FG.2019.8756576

[18] H. Choi and H. Park, “A multimodal user authentication system using faces and gestures”, BioMed Research International, Vol.2015, pp.343475, 2015. http://dx.doi.org/10.1155/2015/343475

[19] S. Krishna and N. Sinha, “Gestop: Customizable gesture control of computer systems”, arXiv preprint, 2020. https://doi.org/10.1145/3430984.3430993

[20] P. Ramanahally, S. Gilbert, T. Niedzielski, D. Velázquez, and C. Anagnost, “Sparsh UI: A multi-touch framework for collaboration and modular gesture recognition”, Proc. ASME Conf. Virtual Environ. Human-Computer Interact. (WINVR), 2009. http://dx.doi.org/10.1115/WINVR2009-740

[21] O. Köpüklü, A. Gunduz, N. Kose, and G. Rigoll, “Real-time hand gesture detection and classification using convolutional neural networks”, arXiv preprint, 2019.

[22] A. Filipowska, W. Filipowski, P. Raif, M. Pieni??ek, J. Bodak, P. Ferst, K. Pilarski, S. Sieci?ski, R. J. Doniec, J. Mieszczanin, J. Skwarek, K. Bryzik, M. Henkel, and M. Grzegorzek, “Machine learning-based gesture recognition glove: Design and implementation”, Sensors, Vol.24, No.18, pp.6157, 2023. http://dx.doi.org/10.3390/s24186157

[23] J. Wu, P. Ren, B. Song, R. Zhang, C. Zhao, and X. Zhang, “Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism”, PLoS One, Vol.18, No.11, pp.e0294174, 2023. http://dx.doi.org/10.1371/journal.pone.0294174

[24] H. Lee, J. K. Mandivarapu, N. Ogbazghi, and Y. Li, “Real-time interface control with motion gesture recognition based on non-contact capacitive sensing”, 2022.

Downloads

Published

2025-08-31
CITATION
DOI: 10.26438/ijcse/v13i8.1120
Published: 2025-08-31

How to Cite

[1]
P. SR, C. K, G. K. P. Makkithaya, K. M. S., and L. T. P, “A CNN-Based Hand Gesture and Facial Recognition Interface for Contactless Human-Computer Interaction”, Int. J. Comp. Sci. Eng., vol. 13, no. 8, pp. 11–20, Aug. 2025.

Issue

Vol. 13 No. 8 (2025): IJCSE August Edition

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.