Real Time Hand Gesture Recognition Based Control of Arduino Robot

Real Time Hand Gesture Recognition Based Control of Arduino Robot

• 1.
  International Journal ofTrend in Scientific Research and Development (IJTSRD) Volume 6 Issue 4, May-June 2022 Available Online: www.ijtsrd.com e-ISSN: 2456 – 6470 @ IJTSRD | Unique Paper ID – IJTSRD49934 | Volume – 6 | Issue – 4 | May-June 2022 Page 79 Real-Time Hand Gesture Recognition Based Control of Arduino Robot Dr. Sunil Chavan1 , Prof. Revti Jadhav2 , Jayesh Mankar3 , Suyash Thakur3 , Shahid Ansari3 , Vatsal Panchal3 1 Principal, 2 Professor, 3 Student, 1, 2, 3 Smt. Indira Gandhi College of Engineering, Navi Mumbai, Maharashtra, India ABSTRACT The main objective of this project aims at delivering the need for one of many applications of the Internet of Things domain. It is a combination of various domains such as image processing and robotics for development in the field of the Internet Of Things (IoT) now also termed as Internet Of Everything (IoE) after seeing its involvement in almost everything happening in our day to day lives. With increased dependency on home automation and wireless controlled equipment and gadgets, this project also shares some scope in home automation. To improve the functionality of gadgets and to increase their efficiency we thought of creating a device to cater to these needs and explore alongside our own interest in the field of the Internet of Things. This paper presents the design, functioning, and successful testing of a rover controlled wirelessly with help of hand gestures. Gesture Recognition has played important role in the field of Human-Computer Interaction (HCI). Vision-based hand gesture recognition provides a great solution to various machine vision-based applications by providing an easy interaction channel. For various automated machine control applications, an effective real-time communication approach is required. In this paper, a vision- based hand gesture-based approach is presented for providing a real- time control of Arduino-based robot. Combining all of these concepts into a single device that can recognize hand gestures and sends data wirelessly to remote devices for surveillance and other purposes. KEYWORDS: Computer vision, Human-computer interface (HCI), Hand Gesture Recognition (HGR), Image processing, IoT, Arduino Uno How to cite this paper: Dr. Sunil Chavan | Prof. Revti Jadhav | Jayesh Mankar | Suyash Thakur | Shahid Ansari | Vatsal Panchal "Real-Time Hand Gesture Recognition Based Control of Arduino Robot" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456- 6470, Volume-6 | Issue-4, June 2022, pp.79-83, URL: www.ijtsrd.com/papers/ijtsrd49934.pdf Copyright © 2022 by author(s) and International Journal of Trend in Scientific Research and Development Journal. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0) (http://creativecommons.org/licenses/by/4.0) INTRODUCTION With an increase in digitalization and automation in many industries, various problems and jobs which were once considered laborious for humans are now becoming effortless. Robots' functionality and accuracy have proven to be efficient, clean as well as productive in various industrial and security sectors. This along with the help of other domains such as image processing and wireless connectivity over the internet is now becoming a major part contributing to the betterment of people in the coming future. The main objective of this project aims at delivering the above saying. It is a combination of various domains such as computer vision, HCI, and robotics for development in the field of Internet of Things (IoT) now also termed Internet of Everything (IoE) after seeing its involvement in almost everything happening in our day to day life. With increased dependency on home automation and wireless controlled equipment and gadgets, this project also shares some scope in home automation. To improve the functionality of gadgets and to increase their efficiency we thought of creating a device to cater to the needs and explore alongside our interest in the field of the Internet of Things. This paper uncovers Human-computer interaction using Hand Gesture Recognition. It presents the design, functioning, and successful testing of a rover controlled wirelessly with help of hand gestures. With IJTSRD49934
• 2.
  International Journal ofTrend in Scientific Research and Development @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD49934 | Volume – 6 | Issue – 4 | May-June 2022 Page 80 wireless connectivity and Bluetooth support for easy data transmission, this rover can be monitored and controlled on multiple devices. The camera module capture and send real-time footage to users remotely here as an Ip address link for live tracking of the environment surrounding the rover. Adding motor control gave us maneuverability and Arduino functions as the brains of this rover. To instruct this rover we took the help of serial communication to coordinate tasks between Arduino and python programming language. Literature Survey: In past, a lot of research work has been done on recognizing hand gestures efficiently. Systems were developed with applications in robotic control, gaming applications, and many more. There has been much research in the field of hand gesture recognition for controlling robots. Rafiqul Zaman Khan et al [2] discussed the issues in hand gesture recognition and provided the literature review on different techniques used for Hand Gesture Recognition. R. Pradipa et al [4] performed an analysis of different Hand Gesture Recognition Algorithms and provided their advantages and disadvantages. A Computer Vision-based approach for gesture recognition includes four phases commonly Image Acquisition, Pre-processing, Feature Extraction, and Classification as shown in Fig. 1. Firstly, the image of the user providing gestures is collected through a webcam and is followed by some preprocessing. Then, the hand portion is extracted and classified into the category of a particular gesture [5]. F. Arce et al [6] illustrated the use of an accelerometer based on 3 axes to recognize Hand Gestures Proposed System: The system hardware includes Arduino Uno microcontroller, Dc Motors, L298N motor Driver, HC-05 Bluetooth Module, LED, and Esp3- Cam.Esp32 camera is a small size, low power consumption camera module based on ESP32. It is widely used in IoT applications such as wireless video monitoring, Wifi image upload, and QR identification. The battery is required to power everything in this project from Arduino to motors. The system software includes Arduino IDE, and Python IDE (Pycharm). Pyqt5 designer (GUI toolkit) for designing a user interface. Libraries Used: Pyserial, OpenCV, Matplotlib, NumPy, Cvlib, Urlib, SoftwareSerial library. Mediapipe: MediaPipe is a framework that is used for applying in a machine learning pipeline, and it is an open-source framework of Google. The MediaPipe framework is beneficial for cross-platform development since the framework is made using statistical data. The MediaPipe framework is multimodal, where this framework is often applied to varied audios and videos. Methodology: A. Hand gesture recognition The proposed system presented in this paper performs real-time hand gesture recognition which is done using various image processing techniques. The overall system architecture of the hand gesture recognition system using media pipe is shown in fig B. System Architecture
• 3.
  International Journal ofTrend in Scientific Research and Development @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD49934 | Volume – 6 | Issue – 4 | May-June 2022 Page 81 In the block diagram, we can observe the serial communication between Python and Arduino Uno. Here Esp32 camera is working Independently, the live stream can be accessed by any device available if an IP address is known. Before running the Python code, First, enable the Bluetooth application and pair the Bluetooth HC05 device with a PC or laptop. When we run the python program it first establishes a Serial connection between PC to Arduino Uno. It first checks for the available connection, after paring it starts capturing video from the Webcam for live interaction between user and Robot. Then it detects for HandLandmarks of the hand and when it is detected successfully it counts the number of fingers raised and Displays it accordingly. If two fingers are raised it will show 2 counts. Depending on the number of fingers counted it will receive certain instructions to send commands like to go forward, backward, left, right, and stop. Later Arduino Uno receives the commands through serial communication between PC and HC05 Bluetooth. In Arduino Uno, various libraries are included like a Software serial that reads the received data. Here we first initialized the pins of Arduino Uno and then Begin Serial communication. It reads the instruction that is received and Begins commands like the Forward, Backward, Left, Right, and Stop. Result The esp32 camera can be accessed remotely using its IP address for monitoring the robot's surroundings. This can be used for surveillance, monitoring, and other wide applications. The hand gesture recognition process captures the hand gestures of the user. The video of the user is captured through the webcam of a laptop or computer. This process involves converting the video to static frames which can be used for image processing. These frames are obtained through the function of the OpenCV library
• 4.
  International Journal ofTrend in Scientific Research and Development @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD49934 | Volume – 6 | Issue – 4 | May-June 2022 Page 82 The movement of the robot is controlled by these five gestures. The captured video goes through image processing to detect Multi-hand landmarks. When landmarks are detected it counts for the number of figures that are raised. When 5 fingers are raised it sends a forward command. When 4 fingers are raised it sends a backward command. When 2 fingers are raised it sends the Right command. When 1 finger is raised it sends the left command. When 0 fingers are raised it sends a stop command. I Advantages: The main advantage of this system is that it does not require any additional hardware implementation as the web cameras of the laptop can be used to receive gestures from the user. It requires only a web camera on the laptop and hence can be used with any of the systems for gesture recognition. There is no need for contact with the computer physically and the user can communicate with the system from a few meters distance. The robotic systems controlled by hand gestures provide many useful applications in various fields. Real-time vision-based gestures can be used in gaming technology for designing interactive gaming applications. Applications: In recent years, HGR technology has made its way into various industries as advances in Deep Learning, Computer Vision, machine learning, Cameras, and sensors have made it more available, reliable, and accurate. The top four fields actively adopting hand tracking and gesture recognition are given below. Automotive and Robotics Industry A gesture recognition solution from Sony Depth Sensing Solutions features a time-of-flight feature that measures the time it takes for a gesture to "travel" from the infrared sensor to the object and back. The AI is trained to differentiate main gestures from gestural noise and to work under any lighting conditions. The BMW 7 Series features a built-in HGR system that recognizes five gestures and can control music and incoming calls, among other things. Less interaction with the touchscreen makes the driving experience safer and more convenient. Machine learning methods, artificial intelligence, and complex algorithms are used by robotic control systems to perform a particular task, enabling natural communication of robotic systems with the environment and making autonomous decisions. Research studies suggest that combining computer vision technology with a robot can be used to create assistive devices which can be used by elderly. Another study uses computer vision to allow a robot to ask a human for a proper path within a house. The hand gesture can be used to design a surveillance robot used to spy on the enemies in the areas where humans are restricted. The dynamic feature of the surveillance robot increase the range of area under surveillance. The small size and portability of the robot allows it to reach inconvenient places easily. Search and rescue operations can be performed using Hand gesture controlled robots in the areas affected by natural calamities. These robots play a very important role to perform rescue operations in the areas which are not safe for humans Healthcare Industry Operating rooms and emergency room may be chaotic, with lots of noise machines and personnels. In such environments, voice commands can be less effective than gestures. Touchscreens are not an option because there’s a strict boundarybetween what is and is not sterile. But access to information and images during surgery or another manipulation is possible with HGR tech. Various companies have proved it and many more research is ongoing on this system. Gesture can provide doctors with the ability to check Imagery, MRI and CT scans with simple gestures while performing their task. Patient monitoring in the hospitals can be done by using robots which can be moved from one ward to another using hand gestures. The robot will perform surveillance and keep an eye on the patients wherever they are present. Virtual reality Leap Motion (acquired by Ultrahaptics) presented updated HGR software that allows users to track
• 5.
  International Journal ofTrend in Scientific Research and Development @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD49934 | Volume – 6 | Issue – 4 | May-June 2022 Page 83 gestures in virtual reality in addition to controlling a PC. The Leap Motion controller is a USB device that observes the area of about one meter with the help of two IR camera sensors and three infrared LEDs. A hand tracking application developed by ManoMotion recognizes gestures in three dimensions using just a smartphone camera (on both Android and iOS) and can be applied in VR and AR environments. The use cases for this technology include IoT devices, VR- gaming and interaction with virtual simulations. Consumer electronics Home automation is another vast field within the consumer electronics domain in which gesture recognition can be used for various applications. uSens has developed a hardware and software to make smart TVs sense finger movements and hand gestures. Gestoos is a AI Platform in which gestures can be created and assigned by a smartphone or another device, and one gesture can be used to enable various commands. It also offers touch less control over lighting and audio systems The consumer market is open to accept and try new HMI technologies, and hand gesture recognition technology can be said as evolution from touchscreens. Demand for more smooth and means of interaction with devices as well as a concern for Safety of driver are pushing the adoption of HGR. Future scope: A limited (five) number of hand gestures have been considered here. Though the number of gestures can be extended by adding different algorithm, the computation will be increased. OpenCV library is preferred here as it is applicable for real time and execution is faster. It may be used on a variety of robotic systems with a set of gesture commands appropriate to that system. Further the robot can be designed to perform various other tasks and gestures can be designed accordingly. The algorithm improved further working in various light conditions. The performance analysis of the algorithm will be done by taking images of gestures at different conditions like non uniform background, poor light conditions. Conclusion: The proposed system was achieved by using a webcam or a built-in camera which detects the hand gestures. The main advantage of the system is that it does not require any physical contact with humans and provides a natural way of communicating with the robots. The algorithm implementation does not require any additional hardware. The proposed system can be extended further for a wide range of applications. Movement of robot is controlled using these five gestures, which were implemented. The arduino robot moves in the respective direction with the help of wheels attached to the DC motors. References: [1] T. Jha, B. Singh, A. Sharma, S. Sharma and R. Kumar, "Real Time Hand Gesture Recognition for Robotic Control," 2018 Second International Conference on Green Computing and Internet of Things (ICGCIoT), 2018, pp. 371-375, doi: 10.1109/ICGCIoT.2018.8752966. [2] Rafiqul Zaman Khan and Noor Adnan Ibraheem, "Hand Gesture Recognition: a Literature Review", International Journal of Artificial Intelligence & Applications (IJAIA), July 2012. [3] N. Mohamed, M. B. Mustafa and N. Jomhari, "A Review of the Hand Gesture Recognition System: Current Progress and Future Directions," in IEEE Access, vol. 9, pp. 157422-157436, 2021, doi: 10.1109/ACCESS.2021.3129650. [4] Pradipa, R. and S N Kavitha. “Hand Gesture Recognition - Analysis of Various Techniques, Methods and Their Algorithms.” (2014). [5] Ebrahim Aghajari and Damayanti Gharpure, "Real Time Vision-Based Hand Gesture Recognition for Robotic Application", International Journal, vol. 4, no. 3, 2014 [6] F. Arce and J. M. G. Valdez, "Accelerometer- Based Hand Gesture Recognition Using Artificial Neural Networks", Soft Computing for Intelligent Control and Mobile Robotics Studies in Computational Intelligence, 2011 [7] J. Hossain Gourob, S. Raxit, and A. Hasan, "A Robotic Hand: Controlled With Vision-Based Hand Gesture Recognition System," 2021 International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI), 2021, pp. 1-4, doi: 10.1109/ACMI53878.2021.9528192. [8] Shaikh, S.A., Gupta, R., Shaikh, I., & Borade, J. (2016). Hand Gesture Recognition Using Open CV. [9] Joseph Howse. OpenCV computer vision with python. Packt Publishing Birmingham, 2013. [10] Sudhakar Kumar, Manas Ranjan Das, Rajesh Kushalkar, Nirmala Venkat, Chandrashekhar Gourshete, and Kannan M Moudgalya. Microcontroller programming with Arduino and python. 2021. [11] Google, MP, https://ai.googleblog.com/2019/08/on-device- real-time-hand-tracking-with.html. [12] https://learnopencv.com/introduction-to- mediapipe