Abstract
The usage of hand gestures to operate digital devices has grown in recent years, and developing accurate gesture recognition systems using computer vision technology is becoming easier. Voice assistants have also become more common, with developments in artificial intelligence (AI) and natural language processing (NLP) allowing for more accurate voice recognition. Our goal is to create a system which combines both gesture-controlled mouse functionality and voice assistant to execute things that a regular mouse and keyboard can. The GCVA (Gesture Controlled Voice Assistant) comes with a gesture-controlled mouse and a voice assistant. A gesture-controlled mouse uses OpenCV and Media-Pipe, whereas a voice assistant uses AI and NLP concepts to recognize the voice commands given by the user. The system should be used by performing hand gestures and executing the right click, left click, drag, drop, volume control and computer cursor functions, thereby omitting the need for a physical mouse. The success of this system is observed in terms of practicality, stability, and compatibility with physical mechanisms. The voice assistant works and responds to pre-defined commands present in the system. The GCVA system was tested by four different users in order to generate the accuracy scores of mouse gestures and response time graphs for voice assistant. The system yields an accuracy rate of 98%. The system’s accuracy and usability showed that hand gesture detection and voice assistants have the potential to replace traditional input devices in the future.
Keywords: Human Computer Interaction, Computer vision, Artificial Intelligence, Voice Assistant, Media-Pipe