by Kirti Sahu & Ashish Kumar Khare
|
ABSTRACT
Gesture Recognition is one of the most important part of research today. Many new algorithms are being developed recently in today‟s upcoming technologies. In the day to day life, mobile devices like phones or tablets are very common and being widely used among all people of world. These devices are connected with high speed networks and provide strong communications. These devices are often an enormous help for the people that aren't ready to communicate properly and even in emergency conditions. For a disabled one that isn't able to speak or an individual who speaks a special language, these devices are often a boon as understanding, translating and speaking systems for these people. This chapter discusses a transportable android based hand sign recognition system which may be employed by disabled people. This paper presents a comprehensive review on vision-basedhand gesture recognition, with a stress on dynamic hand gestures. First, a quick introduction of the essential concepts and the classification of hand gesture recognition techniques are given. Then, variety of popular related technologies and interesting applications are reviewed. Finally, we give some discussion on the present challenges and open questions during this area and mean an inventory of possible directions for future work.
Keywords: Python, NumPy, TensorFlow, Tflearn, Keras, Convolutional Neural Network, Training, Classification.
INTRODUCTION
Sign Language may be a well-structured code gesture, every gesture has meaning assigned thereto. Sign Language is that the only means of communication for deaf people. With the advancement of science and technology many techniques are developed not only to attenuate the matter of deaf people but also to implement it in several fields. But if the pc are often programmed in such how that it can translate signing to text format, the difference between the traditional people and therefore the deaf community can be minimized. We have proposed asystem which is in a position to acknowledge the varied alphabets of Indian signing for Human-Computer interaction giving more accurate results a minimum of possible time. It will not only benefit the deaf and dumb people of India but also might be utilized in various applications within the technology field.
LITERATURE SURVEY
The contributions of various scholars are studied for survey and analysing the merits and demerits in order to enhance the consequences for making the system work better.
In Paper [1], Abhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama H S have proposed a system on Hand Gesture Recognition using Machine Learning Algorithms. The main focus of this is to recognize the human gestures using mathematical algorithms for human computer interaction. Only a few modes of Human-Computer Interaction exist, they are: through keyboard, mouse, touch screens etc. Each of these devices has their own limitations when it comes to adapting more versatile hardware in computers. Gesture recognition is one among the essential techniques to create user-friendly interfaces. Usually gestures are often originated from any bodily motion orstate, but commonly originate from the face or hand. Gesture recognition enables users to interact with the devices without physically touching them. This paper describes how hand gestures are trained to perform certain actions like switching pages, scrolling up or down in a page. The importance of gesture recognition lies in building efficient human-machine interaction.
In paper [2], Jay Prakash, Uma Kant Gautam has proposed a Hand Gesture Recognition using Computer Vision Based Approach, Hand Gesture Recognition, Human Computer Interface (HCI), Instrumented Glove, Non-Verbal language. Hand Gesture Recognition System works like this : first user gives input to the system by making hand gestures, then system scanned the gestures by using cam or sensor and deducts it into signal and passes the program, now its program responsibility to first accept the signal then examine what is the input given using gestures, then check if there is any corresponding data is saved into dataset then result will be obtained in the output device.
In paper [3], Amit Chaurasia and Harshul Shire have proposed a system SNCHAR: Sign language Character Recognition using Keras, TensorFlow, Scikit, and Pyttsx3. This project "SNCHAR: Sign language Character Recognition" system is a python-based application. It uses live video as input, and predicts the letters the user is gesturing in the live feed. It captures the gestures, and recognizes the area of hand gesture skin colour intensity object. It separates the gesture area from the rest of the frame, and feeds that part to their trained model. This pre-trained model, using the hand gesture as input predicts a value that represents an alphabet. This alphabet is displayed on the screen. User can hear the text predicted on the screen by pressing “P” on the keyboard. The predicted text can be erased if required by using “Z” from the keyboard. At one hand, the project is capable of capturing the live feed and converting the gestures into the corresponding alphabets.
In Paper [4], D. Nagajyothi, M. Srilatha and V. Jyothi have proposed a Hand Gesture Method to Speech Conversion using Image Segmentation and Feature Extraction Algorithm. In this system, the detection of skin colour and region segmentation is performed during the segmentation stage. RGB colour space, cbr colour space, HS colour space, Normalized RGB & HSV are skin colour segmentation techniques. From these values the skin colour is detected. The RGB values lies in between a boundary for skin pixels and it varies for non-skin pixels. With this RGB ratio they can identify whether the skin pixel belong to the skin region or not. Skin region detection algorithm is applied for each gesture and it is applied to skin region to find the colour. This system not only recognizes gesture indications it develops speech system. From the results they have obtained accuracy up to 80%.
In paper [5], T. Chandraleka, Balasubramanian R, Balasubramanian S, Karthikeyan S and Jayaraj R have proposed a system on Hand Gesture Robot Car using ADXL 335. In this System, Arduino, Microcontroller, Transmitter, Receiver are used. The outer frame work was done using tyres and supporting board is fixed to it and the tyres are each other with steel road of suitable capacity and which the tyres are connected to the board using wires and also the motors are fixed to the tyres for rotation purpose. Radio signals are transmitted using transmitter module Without any physical connection, the embedded system is used to interact with each other. After successful completion the working loads were improving the project. Even the mounting of ultrasonic sensor and other sensors for the complete information about the place where the car is being operated & make it useful for the society. The most important feature is to interact with the application from the distance object without any physical contact.
In paper [6], Sankara Gomathi.S, Amutha. S, Sridhar.G and Jayaprakasan.M have proposed a system Interpretation of Formal Semantics from Hand Gesture to Text using Proficient Contour Tracing Technique. In this system, Contour Tracing, Hand gesture, SVM, Feature Extraction, TOF, IoT are used. In this project, semantics are classified by support vector machine with trained datasets. The recognised hand gestures are displayed as text. Their main objective is to resolve the problem of facing interviewer for vocally impaired individuals. This helps them to build their confidence and eradicate their inferiority complex compared to other methods. In the interpretation of framework, conversion of sign to text, Image captured from camera is binaries, noise is expelled, boundaries of finger is detected and corresponding text is displayed as an output to the receiver.
In paper [7], Abdul Khader, Muhammad Thouseef, Akbar Ali and Ahamad Irfan have proposed a system on Efficient Gesture based Language Recognition using SVM and Lloyd‟s Algorithm. In this work, they have actualized a presumable exact strategy to perceive static gestures or image frames from a live camera or video data. As Hand Gesture Recognition is identified with two noteworthy fields of image processing and AI (machine learning). APIs that can be utilized to implement different strategies and methods in these fields.
In paper [8], Rajesh George Rajan and M Judith Leo have proposed a Comprehensive Analysis on Sign Language Recognition System. The human- machine interaction is developed through the gesture recognition system. In the previous years, most of the researchers had done their research in static hand gesture recognition. Some works have been reported for recognition of dynamic hand gesture. Also, facial expressions aren't included in most generally used systems. Developing systems which are capable of recognizing both hand and facial gestures may be a key challenge during this area. In this paper they have discussed different sign language recognition approaches using different acquisition methods. By using the different data acquisition methods like sensor-based gloves, Kinect, leap motion controller etc.
In paper [9], S. Shivashankara and S. Srinath have proposed a system on American Sign Language Recognition System using Bounding Box and Palm FEATURES Extraction Techniques. Bounding Box Technique, Canny Edge Detector, CIE Colour Model are used. This research paper exhibits an inventive framework, to achieve the transliteration of 24 static alphabets (Letter J and Z not included as they involve hand movement) of American Sign Language into English text and achieved an average recognition rate of 98.21% which is the best in recent (papers published in year 2017, and 2018) existing traditional work carried out. This paper also summarizes the system architecture, state of art, data collection for the proposed work, proposed system design, and the detailed results evaluation by showing comparative graphical depiction of the proposed technique with the existing techniques average recognition rate and also depicts the average gesture recognition rate chart by considering various factors like background complexity, background colour, location, time, distance, angle, mobile camera resolution, and illumination. This paper also highlights on face detection and edge detection technique, and also the various hand / palm features extraction techniques.
In paper [10], Shreyas Rajan, Rahul Nagarajan, Akash Kumar Sahoo, M. Gowtham Sethupati have proposed a system on Interpretation and Translation of American Sign Language for Hearing Impaired Individuals using Image Processing. This project mainly focuses on the development of software that can convert American Sign Language to Communicative English Language and vice-versa. This is accomplished via Image- Processing. The latter is a system that does a few activities on a picture, to acquire an improved picture or to extricate some valuable data from it. Image processing in this project is done by using MATLAB, software by MathWorks. The latter is programmed in a way that it captures the live image of the hand gesture. The captured gestures are put under the spotlight by being distinctively coloured in contrast with the black background.
In paper [11], S. Chandrasekhar and N.N. Mhala have proposed a system on High-speed Integration of Kinect V2 Data for Identification of Hand Gesture in Real time Movements. Hand gesture recognition is extremely critical for human-PC connection. This manuscript presents a narrative constant strategy for human-hand gesture recognition. There a framework for the discovery of quick gesture movement by utilizing a direct indicator of hand developments utilizing information combination technique. In their system, the hand area is removed from the foundation with the foundation subtraction strategy. At long last, the framework has been approved by methods for the Kinect v2 application actualized. The time requirement is recognized and the recognition is quick contrasted with other ongoing minutes. The timing analysis is compared, and the average time using data fusion method is 63ms. By using fast integration of data, the average time is 45ms. The time taken for recognition of hand gesture is been improved.
In paper [14], Suthagar S., K. S. Tamilselvan, P. Balakumar, B. Rajalakshmi and C. Roshini have proposed a system on Translation of Sign Language for Deaf and Dumb People. Their project objective isto analyse and translate the sign language that is hand gestures into text and voice. For this process, Realtime Image made by deafmute people is captured and it is given as input to the pre-processor. Then, feature extraction process by using algorithm and classification by using SVM (support Vector Machine) can be done. After the text of corresponding sign has been produced. The obtained output is converted into voice with use of MATLAB. Thus, hand gestures made by deaf-mute people has been analysed and translated into text and voice for better communication. In this proposed model an attempt has been made to design a system which can recognize the sign language of alphabets and number.
In paper [15], V. Padmanabhan, M. Sornalatha have proposed a system for dumb people Hand gesture recognition and voice conversion system. In this system, Gesture, Flex sensor, accelerometer, microcontroller, TTS are used. This project aims to lower the communication gap between the mute community and additionally the quality world. The projected methodology interprets language into speech. The system overcomes the required time difficulties of dumb people and improves their manner. Compared with existing system the projected arrangement is compact and is feasible to hold to any places. This system converts the language in associate passing voice that'swell explicable by blind and ancient people.
Table 1: Comparison on Various Methods Used in Hand Gestures
S. No | Paper | Technique | Result | Issues |
1 | Hand Gesture Recognition using Machine Learning Algorithms | Gesture Recognition, Human Computer Interaction, User- friendly Interface. | Each of these devices has their own limitations when it comes to adapting more versatile hardware in computers. | They are interpreted as gestures by the computer to perform actions like switching the pages, scrolling up or down the page. The system is built using OpenCV and TensorFlow object detector. |
2 | Hand Gesture Recognition | Computer Vision Based Approach, Hand Gesture Recognition, Human Computer Interface (HCI), Instrumented Glove, Non-Verbal language | Hand Gesture Recognition System works like this: first user give input to the system by making hand gestures, then system scanned the gestures by using cam or sensor and deducts it into signal and passes the program, now its program responsibility to first accept the signal | Examine what is the input given using gestures, then check if there is any corresponding data is saved into dataset then they will get their result. |
3 | SNCHAR: Sign language Character Recognition | Keras, TensorFlow, Scikit, and Pyttsx3 | Different images were tested and found that the new technique of TensorFlow was found to show some results. | Moreover, there were difficulties to attain a 57% accuracy. |
4 | Hand Gesture Method to Speech Conversion using Image Segmentation and Feature Extraction Algorithm | HSV colour model, Pattern Recognition, Tracking and Segmentation. | The RGB values lies in between a boundary for skin pixels and it varies for non-skin pixels. With this RGB ratio they can identify whether the skin pixel belong to the skin region or not. Skin region detection algorithm is applied for each gesture and it is applied to skin region to find the colour. | The issue is the system was not able to achieve the proper image capturing and colour detection problems. |
5 | Interpretation and Translation of American Sign Language for Hearing Impaired Individuals using Image Processing | Feature Extraction, Edge Detection, Segmentation | Their system translates the detected gesture into actions such as opening websites and launchingapplications like VLC Player and PowerPoint. The dynamic gesture is used to shuffle through the slides in presentation. Our results show that an intuitive HCI can be achieved with minimum hardware requirements. | System that did not utilize any markers, hence making it more user friendly and low cost. In this gesture recognition system, they have aimed to provide gestures, covering almost all aspects of HCI such as system functionalities, launching of applications and opening some popular websites. |
6 | High speed Integration of Kinect V2 Data for Identification of Hand Gesture inReal timeMovements | Gesture Recognition, Human Computer Interaction, Kinect V2 system | The time requirement is recognized and the recognition is quick contrasted with other ongoing minutes. The timing analysis is compared, and the average time using data fusion method is 63ms | Outcome of the module is inappropriate. |
7 | Sign Language Recognition | SVM, CNN, HSV colour model | A dataset containing all the gestures are present. Each gesture folder consists of 2400 images which is used for training and testing the model. There are 47 gestures but more can be added by the users. | As the hand segmentation is dependent on the colour of the hand, if the objects in the background match the skin colour, it could distort the binarized threshold image. Due to similar gestures that exist in ASL, the final accuracy of classification depends on the environment and image processing techniques. |
8 | SVM, MATLAB | Hand detection, Segmentation and Hand Tracking | An attempt has been made to design a system which can recognize the sign language of alphabets and number. 11 different features from image has been extracted to make a feature vector database. SVM and neural network is used for classifying the different sign- language word and hence for recognition. | The result obtained for the system is not appropriate and could recognise the images properly. |
9 | Hand Gesture Recognition and Voice conversion system for dumb people | Gesture, Flex Sensor, TTS, Microcontroller | The language interprets into some text kind displayed on the digital display screen, to facilitate the deaf people. | The main issue is recognition algorithm is reduced to 60% - 80%. |
CONCLUSION AND FUTURE WORK
In this project, we present a hand tracking and segmentation algorithm that is both accurate and computationally efficient. The importance of gesture recognition lies in building efficient human- machine interaction. This paper describes how the implementation of the system is completed based upon the pictures captured, and the waythey're interpreted as gestures by the pc to perform actions like switching the pages, scrolling up or down the page. They were able to create a robust gesture recognition system that did not utilize any markers, hence making it more user friendly and low cost. In this gesture recognition system, we have aimed to provide gestures, covering almost all aspects of HCI such as system functionalities, launching of applications and opening some popular websites. In future we would like to improve the accuracy further and add more gestures to implement more functions. Finally, we target to extend our domain scenarios and apply our tracking mechanism into a variety of hardware including digital TV and mobile devices. We also aim to extend this mechanism to a range of users including disabled users.
REFERENCES
[1] Abhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama H S “Hand Gesture Recognition using Machine Learning Algorithms” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8, Issue-1, May 2019.
[2] Jay Prakash, Uma Kant Gautam “Hand Gesture Recognition”, International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277- 3878, Volume-7 Issue-6C, April 2019.
[3] Amit Chaurasia, Harshul Shire, “SNCHAR: Sign language Character Recognition”, International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-3, September 2019.
[4] D. Nagajyothi, M. Srilatha, V. Jyothi “Hand Gesture Method to Speech Conversion using Image Segmentation and Feature Extraction Algorithm” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-4, November 2019
[5] T. Chandraleka, Balasubramanian R, Balasubramanian S, Karthikeyan S, Jayaraj R “Hand Gesture Robot Car using ADXL 335” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-4, November 2019.
[6] Sankara Gomathi.S, Amutha. S, Sridhar.G, Jayaprakasan.M “Interpretation of Formal Semantics from Hand Gesture to Text using Proficient Contour Tracing Technique” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277- 3878, Volume-8, Issue-2S11, September 2019.
[7] Abdul Khader, Muhammad Thouseef, Akbar Ali, Ahamad Irfan “Efficient Gesture based Language Recognition using SVM and Lloyd‟s Algorithm” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8, Issue-2S3, July 2019.
[8] Rajesh George Rajan, M Judith Leo “A comprehensive Analysis on Sign Language Recognition System” International Journal of RecentTechnology and Engineering (IJRTE) ISSN: 2277- 3878, Volume-7, Issue-6, March 2019.
[9] S. Shivashankara, S. Srinath “An American Sign Language Recognition System using Bounding Box and Palm FEATURES Extraction Techniques” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-7 Issue-4S, November 2018.
[10] Shreyas Rajan, Rahul Nagarajan, Akash Kumar Sahoo, M. Gowtham Sethupati “Interpretation and Translation of American Sign Language for Hearing Impaired Individuals using Image Processing” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-4, November 2019.
[11] S. Chandrasekhar, N.N. Mhala “High-speed Integration of Kinect V2 Data for Identification of Hand Gesture in Real time Movements” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-4, November 2019.
[12] E. Padmalatha, S. Sailekya, R. Ravinder Reddy, Ch. Anil Krishna, K. Divyarsha “Sign Language Recognition” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277- 3878, Volume-8 Issue-3, September2019.
[13] L. LATHA, M. KAVIYA “A Real Time System for Two Ways Communication of Hearing and Speech Impaired People” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277- 3878, Volume-7 Issue-4S2, December 2018.
[14] Suthagar S., K. S. Tamilselvan, P. Balakumar, B. Rajalakshmi, C. Roshini “Translation of Sign Language for Deaf and Dumb People” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-5, January 2020.
[15] V. Padmanabhan, M. Sornalatha “Hand gesture recognition and voice conversion system for dumb people” International Journal of Scientific & Engineering Research, Volume 5, Issue 5, May-2014.
