IMouse: Eyes Gesture Control System

—A high number of people, affected with neuro-locomotor disabilities or those paralyzed by injury cannot use computers for basic tasks such as sending or receiving messages, browsing the internet, watch their favorite TV show or movies. Through a previous research study, it was concluded that eyes are an excellent candidate for ubiquitous computing since they move anyway during interaction with computing machinery. Using this underlying information from eye movements could allow bringing the use of computers back to such patients. For this purpose, we propose an imouse gesture control system which is completely operated by human eyes only. The purpose of this work is to design an open-source generic eye-gesture control system that can effectively track eye-movements and enable the user to perform actions mapped to specific eye movements/gestures by using computer webcam. It detects the pupil from the user’s face and then tracks its movements. It needs to be accurate in real-time so that the user is able to use it like other every-day devices with comfort.


INTRODUCTION
Many people nowadays are falling victim to diseases that impair them physically, like Paraplegia, as a result of which the person is unable to use his body from neck down. In most of the OECD (Organization for Economic Co-operation and Development) countries, females are more affected by disabilities than males [1]. The only organ that can generate different actions is their eyes. In 7 billion population, the total of 518 million people reported having a disability in Census 2011. Currently, in Feb 7, 2018, around 10% (about 650 million) of the world"s population individuals live with disabilities.
A considerable portion of people affected with Amyotrophic lateral Sclerosis (ALS) [3], or those paralyzed are not able to use computers for basic day to day tasks. Even when it comes to eating they need help from another individual to feed them. These individuals need assistance for their day to day activities. Currently, individuals with disabilities usually type on the computer keyboard by holding long sticks in their mouth. The technique that we present, will help handicaps to be independent in their lives. It will give them a chance to entertain, socialize and work in their lives.
Creative and advance techniques of HCI are being produced rapidly. Many experts are actively working in this research field. Human eyes bear a lot of information which can be obtained and can be utilized in various applications [2] (i.e. interacting with Computers). Eye gesture shows an individual"s point of interest. Eye gesture tracking aims to keep track of gestures from a human eye. By capturing Eye movements and using it as control signals to enable direct interaction with interfaces without requiring input from the keyboard or mouse.
Existing computer input devices like a mouse, keyboard, and other type of input devices have been used for interaction with digital instruments. Individuals with disabilities cannot use these computer input devices by themselves. In this research work, a computer input device that is controlled only by human eyes is developed for individuals suffering from disabilities and also for wearable computing [4]. Furthermore, such information could be used to produce necessary outputs for controlling a computer like moving commercially available robotic machinery such as the robotic arm or wheelchairs to enable these patients to feed themselves. This will physically enable them and make them contributing members of the society.
The purpose of this research is to explore and improve upon existing avenues in the eye gesture tracking system. Particularly those areas which can help physically disable individuals, enabling them to use computers and programmable controlled systems. Thus, such individuals could still take on their responsibilities, improve the quality of their lives and continue with their day to day tasks often without the need for a helping hand. In present times, most eye tracking systems utilizes the use of real-time video-based tracking of the pupil. We have adopted the same technique, technologies and improved upon them developing a more robust and accurate system. We used a high-definition, small, portable Microsoft LifeCam HD-6000 this is easily available at low cost. This camera can easily attach with any computer or laptop through the USB port.

A. Proposed System
The eye gesture control system directly interacts with the vision of the human eyes and then controls the system. Eye gesture, a real-time gesture assurance programming which controls a computer mouse cursor by using the user"s eye gestures [5]. Only requirement to operate the imouse system is, individuals having at least one eye with good vision and ability to control the computer. Its users can be grown-ups and youngsters with cerebral paralysis, spinal rope wounds, mental wounds, ALS, different sclerosis, Brainstem strokes, and so on. Eye gesture control system can be utilized as a part of homes, workplaces, schools, healing centers, and long-haul mind offices. By looking at the control of a system that is display on a screen, a person can run a computer software,  Simulating mouse functions, performing different mouse functions such as left click, right click, double click and so on using their eyes.
This paper is organized as follows: Section II elaborates the literature we studied, Section III presents questions regarding this research, Section IV describes the methodology of the proposed system, Section V presents various Tools and techniques that are utilized in our system, Section VI illustrate the algorithms that are used in the proposed system, Section VII describes the design of the proposed system, Section VIII presents implementation, results and tests for the proposed system, Section IX presents contributions of this research work, Section X is the conclusion of our system and finally the last Section XI shows recommendations and future work for the proposed system.

II. LITERATURE REVIEW
The literature was studied to address the aims, understanding of the research area, focus on the research questions, planning of the data collection approach, clarification of the meaning of the terms and proper identification of the framework. The most important task was to understand the research domain in which eyes detection and cursor movement of a mouse is involved.
Going through the literature, the focus was on how to develop a system which can fulfill the needs of physically impaired individuals and the system should be very easy to understand.
A group over at MIT [6] has created a system titled "The sixth sense", the system aims to enhance human-computer interaction by using gestures from the hands and eyes. The entire system is mountable on the user"s head, so that it can be projected on to smooth surfaces (like walls) and used anywhere in the world. The problem is that, it doesn"t provide enhanced assistance and accessibility to the disabled nor does it produce a system that can interact with other compatible devices.
Though Drewes, Heiko (2009) [7], presents a comprehensive overview, it was noted that most algorithms needed further refinement as they took tedious and longwinded approaches to calibration. A no-nonsense, agile approach was defined Schmidt, Jochen [8], in using structure from motion algorithm for human-computer interaction. This was exploited by Kassner, Moritz Philipp, and William Rhoades Patera [9], in using the same sfm (structure from motion) algorithm, optimizing it and extending its use as an efficient algorithm for pupil tracking. In order to achieve this goal, they developed a framework by the name of PUPIL, to critically inquire the relationship between a human subject and space to visualize this unique spatial experience and to enable its use for gaze gesture tracking.
In 2018 [10], an eye tracking algorithm based on Hough transform was developed. This system detects the face and eyes of a person. It uses a webcam to detect user"s face and eyes. The system is based on Matlab. The issue in this system is of real-time tracking and time-speed issue. The system is quite slow [11] and it needs a high-quality computer system to work properly which is costly.
In 2017 a better system was introduced by the authors. This system is developed for the paralytic patients. This system uses webcam through MATLAB and moves the mouse cursor by using the pupil of a person [12], [13]. The issue in this system is that it takes a lot of time in detecting the pupil of a person. It uses a lot of algorithms and techniques to detect the pupil.
In 2016 [14] a Vision-based wearable eye-gaze tracking system was introduced. This system works using a high infrared camera. It detects the eyes of the person through the infrared cam. The issue in this system was that it is slow and costly [15].
In 2015 [16], a Pupil center coordinate detection using the circular Hough transform technique was introduced. In this system, the webcam uses Hough Transform Techniques to detect the pupil of a person [17]. The issue in this system was that it takes a lot of time and is not a real-time system. It first captures the body after that, it move to face then eyes and finally to the pupil taking a lot of time.
In 2014, a face and eye-controlled system were developed which were based on MATLAB [18], [19]. It uses a webcam to control the mouse by eye and face movement. The issue in this system is that this system only works in a few centimeter radius.
In 2013 [20], a system was developed which used eye tracking system, this system is based on the pictogram selection. It uses different eye-tracking techniques to make the system reliable. The issue in this system is that if any liquid is found in eyes, it will not work. Like female use eyeliner or mascara in their eyes, so the system stops working in those situations.

A. The Human Eye
The eye of a human [21], works on a two-lens system in a fluid called the vitreous humor, to project the waves of light from different objects in the world onto the retina exterior. Fig. 1 gives the structure of the human eye with the various aspects labeled. There is an area on the retinal surface of the eye that is thickly pressed with cones in the extent to poles. Once the rays pass through the lens, they cast on the retinal surface after traveling through the vitreous humor. The fovea is measured as 1/4000th of the retinal surface area [22] or a rawboned distance of between 0.3 degrees and 2 degrees. This zone is called fovea and appears as a minor yellow segment on the retinal outside. www.ijacsa.thesai.org The retinal surface contains two sorts of photoreceptors, bars and cones [23]. The fovea is thickly stuffed with cones, roughly 161,900 for each square millimeter, taking into account high-determination shading vision. The composition of a retinal exterior demonstrates to us that around is just a little portion of our visual field that we can resolve in high resolution [24]. The rest of the retina isn't visually impaired as the separation increments, the fovea thickness of pinecones and visual sharpness are enormously decreased.
The first [24], optical element is the Cornea, to be precise, a thin layer of tear fluid that covers the curved corneal surface. The encompassing zone is populated by bars, thickly pressed around the fovea. One function of the aperture is to control how much light can pass into the lens or camera. This gap can revolute in the estimate, becoming biggerexpansion in little light circumstances to permit all the further.
The eye focal point can change, fit as a fiddle or distort keeping in mind the end goal to center around light originating from various depths [24]. The retinal surface is secured with photoreceptor cells, so that the density of the light should be such that it is not too high that it affects the eye and cause the eye lens to contract and also it should not be too low that the camera becomes unable to track the movements of the pupil.    Fig. 2 shows the system overview and model of the system. The System prototype takes input from a camera and recognizes the user"s pupil and tracks it in real-time [25]. This "tracking" information can then be used by computers or micro-controllers to perform various tasks, some of these tasks that the project aims to achieve is to track the pupil-movement [26] and then store that tracked eye movement to control the mouse pointer of a computer, so that someone with a disability like say Amyotrophic Lateral Sclerosis can use, to communicate with others.

B. Analysis of Literature Review
It includes a carefully positioned high definition web camera, an open-platform and easy to install software module compatible with all window-based operating systems and easily installed on all modern laptops or desktop personal computers. This system can be thought of as a holistic transition from the concept to design to proof of concept. It consists of part research paper implementation and part working with the open-source community on designing and then building the prototype, all the while ensuring that only open-source, cheap, readily available and commercially off the shelf (COTS) products are used.

1) Provide a cheap eye-tracking system. 2) To control the cursor of a computer with eyes. 3) Allow physically disabled people to use computers. 4) To control a computer and communicate with other systems.
5) To provide a real-time accurate eyes gesture control system. 6) To provide a hand free mouse control system. 7) To provide a complete generic eye-gesture mouse control system. 8) To provide a complete wire free mouse control system. 9) Easy to control cursor movement of a mouse.

A. Python
Python, an average unique programming language, is progressively utilized as a part of numerous application areas. Dynamic features in python enable programmers to change code at runtime [27]. Some unique features, for example dynamic compose checking, have a functioning impact in performance. Along these lines dynamic component code is frequently changed to provide efficient programming development.
Some particular features of python are as per the following:  Deciphered programming. As opposed to e.g. C or FORTRAN, one does not arrange python code before executing it. Likewise, Python can be utilized intelligently: numerous python translators are accessible, from which contents can be executed.  An extremely discernable language with the clear nonverbose linguistic structure  A language for which an expansive assortment of topnotch bundles are accessible for different applications, from web structures to logical processing.
In our system we use various packages of python which are discussed below.

1) NumPy
NumPy [28], is a low-level library written in C++ (and FORTRAN) for exceptional state scientific capacities. NumPy cunningly beats the issue of running slower algorithms on Python by utilizing multidimensional model and structures that works on clusters. Any algorithm would then be able to be communicated as a structure of models, enabling the algorithm to run rapidly.

2) Scipy
Scipy [28], is a library that utilizes NumPy for more scientific work. SciPy utilizes NumPy model as the essential information structure, and accompanies modules for different tasks in logical programming, including straight variable based math, joining (analytics), normal differential condition settling and flag processing (eyes signals preparing through webcam)), straight variable based math, joining (analytics), normal differential condition settling and flag processing (eyes signals preparing through webcam)).

3) OpenCV
At first all the new modules ought to be produced independently, and distributed in the opencv_contrib archive [29]. Afterward, when the module develops and picks up estimations, it is moved to the focal OpenCV archive and in this manner they should not be freed as a piece of control for OpenCV [30] dispersion, since the library keeps up parallel similarity, and attempts to give fair execution and stability improvements. www.ijacsa.thesai.org

4) PyautoGUI
The reason [31], for PyAutoGUI, is to give a cross-stage Python module for GUI automation for developers. The API is intended to be as basic as conceivable with reasonable errors. PyAutoGUI can simulate moving the mouse, clicking the mouse, dragging with the mouse, pressing keys, pressing and holding keys, and pressing keyboard hotkey combinations. On Windows, PyAutoGUI has no dependencies (other than Pillow and some other modules, which are installed by pip along with PyAutoGUI). It does not need the pywiin32 module installed since it uses Python"s type modules.

VI. ALGORITHMS
In eyes, gesture control system uses two main different algorithms.

A. Harr-cascade Algorithm
Haar-cascade is a protest detection algorithm used to find faces, people on foot, items, outward appearances in a picture and primarily utilized for face detection. In Haar-course, the framework is given a few quantities of constructive pictures (like appearances of changed people at various foundations) and pessimistic (pictures that do not face but rather can be whatever else like the seat, table and divider and so forth).
Human face detection [32], noted to be a testing problem in the zones of image handling and patter acknowledgment. Another algorithm for human face detection by crude Haar course algorithm, joined with three extra frail classifiers is presented in this work. The three powerless classifiers depend on tint histogram coordinating, detecting a mouth and detecting eyes. To start with, pictures of individuals are handled by a crude Haar course classifier, without false human face dismissal (low rate of false negative), yet with few false acknowledgment (false positive). Besides, a powerless classifier a view of face skin tint histogram coordinating is connected and a lion's share of non-human countenances are expelled, to dispose of falsely acknowledged non-human faces. Next, another powerless classifier a view of eyes detection is attached and other lingering non-human appearances are resolved and dismissed.
Finally, a mouth detection activity is used for the rest of the non-human countenances and the possibility of false positive is additionally diminished. With the assistance of OpenCV, test starts to run on pictures of individuals under impediments and understanding and some level of preface and turns, in both preparing set and test set demonstrate that the proposed algorithm is successful and accomplishes best in class execution. Moreover, it is proficient as a result of its ease and effortlessness of execution.

B. Hough Transform Algorithm
The Hough transform [33], algorithm is a worldwide strategy for discovering straight lines, picture examination, computer vision, and advanced imaging preparing. The reason for this method is to discover flawed occurrences of courses classified, a specific lesson of figures through a polling technique. The Hough transforms are connected for the inquiry of a typical focus of round or mostly roundabout segments exhibit in a picture. The effectiveness has been significantly improved by a persistent refresh of a rundown of a polling focuses, in conjunction with the advancement of the aggregator size and position. The technique was executed as a module for the logical open source picture handling bundle Image. Although at first intended for X-beam, diffraction examination, various different applications are cited in various other logical fields, in picture estimation strategies, machine vision and biometric, i.e. for iris restriction.

VII. SYSTEM DESIGN
Imouse system is design in python and following modules of python are imported for working of this system.  PyautoGUI: is a cross-platform GUI automation module that works on Python. In this, you can control the mouse and keyboard as well as you can perform basic image recognition to automate tasks on your computer. In 0 shows use case diagram; the system accomplishes to have the following steps.

1) Software running. 1) Open the webcam on the laptop and show the image of a person.
2) Face detection action is performed.
3) The system detects the eyes of a person. 4) After the above action system move on to the next operation.
(IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 9, No. 9, 2018 548 | P a g e www.ijacsa.thesai.org 5) In the next step the system detects eyes and face through webcam of a laptop. 6) Pupils detected now a person can control mouse cursor through his eye movements. Curosr mevement is shown on the home screen of the computer.

1) Open the webcam and capture the video. 2) The system performs an action and detects the face. 3) The system performs eyes detection. 4)
The system detects the eye pupil. 5) With only the image of face from the webcam, the system will locate the eyes and perform geometry translations.

6) Perform action
Mouse Control: detects a gesture, moves the mouse cursor and translate the coordinates to the user"s screen. Then perform the following the action.

1) Scroll vertical 2) Scroll horizontal
3) Scroll diagonal 4) Repeating the whole system cycle and the system pause or system end.

C. System Sequence Diagram
(IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 9, No. 9, 2018 549 | P a g e www.ijacsa.thesai.org In Fig. 5 the system sequence diagram, elaborates the six basic modules of our system. In the first module, the system detects the pupil of a person through the webcam using detection algorithms. Then the system detects the face. After that the system detects and captures the eyes. Then the system detects the pupils. In the last module, the system starts moving the mouse cursor by tracking pupil movements.

A. Work On System
In this approach, a system is developed using python ,named imouse. First it will open the camera and start capturing video, the video consists of frames, it will select a frame and then convert it to a grayscale image as it converts images to binary form and it becomes easy to detect objects from that image. It will then detect the face from it using Haar-cascade. Haar-cascade is a cascade function which is trained from a lot of positive and negative images and then it detects objects from other images. It will detect the face from the given frame and then crop that frame and will pass on for further processing. Then from the obtained frame it will detect eyes using Haar-cascade and crop that from. Haar-cascade that detects eye is called eye-cascade. A four variable array (i.e supported by numpy) is defined that will give us four values at a time x, y, w, and h. x and y are for the starting point from where the camera detects eyes, w is for the width of eye detected and h gives us the height. By using these variables we draw a rectangle around the eye and then crop that image as shown in Fig. 7. The rectangle starts from the point x and ends at ends at x+w in the horizontal direction, and start from y and ends at y+h in the vertical direction. The Output is shown in the Fig. 7. The resulting image will be of a very small size, the program will blur it to make it clear for our further processing. After that, it will detect the edges of the pupil with the help of the Hough Transform and then draw a circle around the pupil of the eye. But, sometimes the camera recognizes dark circles near the eyes as a pupil.
To overcome this problem, the system detects the dark spots that are only circular in shape and are found in the center of the rectangle drawn. This way the camera will only detect the pupil of the eye and track its movements. Define two variables x and y using pyautogui for cursor movement. Y is for the vertical movement, x is for horizontal movement of the mouse. At the start, both of them are given a random value so when the code will run the mouse will start moving from that point. To detect the movement of the eye, note the position of the pupil from two different frames and check if the distance is increased. The human eye is not still, a little bit of movement is always there. To overcome this problem a check is performed that if the difference in the position of the eye is less than 5 pixels then consider eye is still.
If difference in the horizontal position of the eye is greater than 5 pixels and in vertical direction less than 5 pixels then this means the eye is moving in the horizontal direction, If difference in the vertical position of the eye is greater than 5 pixels and in horizontal direction less than 5 pixels then this means the eye is moving in the horizontal direction and If difference in both horizontal and vertical position of the eye is greater than 5 pixels then this means the eye is moving diagonally. When we move the eye vertically, the mouse cursor will move in the upper/down direction and when we will move our eye horizontal then the cursor will move right/left and when we move our eye in diagonal then the cursor will move in diagonal.

B. The System Analysis & Evaluation
We start by presenting the results of the working contour detecting algorithm that works by using Haar-cascade functions and Hough Transform.

C. Harr-cascade Algorithm
In this algorithm, the system performs two functions detect the face and eyes of the persons Fig. 8 illustrate the facecascade function. It first detects the face of the user from an image using face-cascade. It draws a box around the face and crops the Image for further processing. After we extract the face from the image it detects the eyes of the user from that image. As shown in the Fig. 9 below.

D. Hough Transform Algorithm
In Hough Transform pupil is detected from this cropped image and a circle is drawn around it to track its movements. As you can see in the Fig. 10 below that this system detects the pupil and ignores other dark spots and tracks only the pupil of the eye. Eye coordinates are then located in each frame. If there is a change in the coordinates of the eyes then the eye is considered to be moving otherwise the eye is considered still. The delay of all this processing is less than one second. Which means this project can be used in real life situations.

E. Histograms for Mouse vs Eye
The histograms for mouse vs. eye control are shown below, please note that the readings were taken for ideal lighting conditions and may vary in other situations if pupil detection is erroneous. This has been illustrated in the 0and 0.  Note that during a blink, the transmitted coordinates are null, signifying the absence of a pupil, this fact is exploited in the click gesture, which has been defined in two ways. One, where the user can perform a click by staring at a point for a couple of seconds, or two, blink for an abnormal duration.

IX. RESULT AND DISCUSSION (RQ1)
Individuals having a disability, in most cases they are not able to perform basic tasks using computers, in this situation the system will detect the pupil of a person automatically. It detects the pupil of a person and starts a connection between the mouse and eye gestures. The mouse cursor will start move as pupil move. The clicking of the mouse is based on the eye blinking.
For our system, we conduct some tests by changing the eyes position and adding liquid to the eyes. We also test this system through short and long distances. The results were different in term of distance. The result can be seen in   TABLE III shows the results from various distances. We measure the results at different distances. The maximum distance of the system is 50cm, our system works properly and in real-time at that distance. At 10cm the system is working very fast, its max value at this point is 6.458533cm while its min value is 4.50689cm. The mouse is moving at an average of 1.0965423cm.
At 20cm the average value of the mouse movement is 0.5898252cm. At that point, the mouse is working properly. At 30cm the average value is 0.4281392cm and mouse is working properly. As we getting far from the system the mouse speed is reducing. At 50cm the speed of mouse is little low but it works slowly at 50cm.

X. RESEARCH CONTRIBUTION (RQ2)
Many systems exist that are using eye pupil to move the mouse cursor and also some embedded systems. We aim to implement such a system which is reliable and also have better performance. We developed a complete hand free mouse control system. The system uses webcam of a computer and tracks the eye pupil of a person in real-time to move mouse cursor. The system uses a webcam and detects the face of a person after detecting face it moves toward the eye than it detects the eye pupil. The mouse is moving through pupil movement using webcam infrared. The system provides help to the handicapped and other diseases patients who are not able to use their hands properly.
The recommended system will provide many additional performances than the prevailing systems.
 Suitable for all handicapped persons.
According to the survey, almost 17.7% of the world population is handicapped. Our system is specially designed for these type of individuals, so they can use computer systems by themselves.
 Directly detects the pupil of the user.
The webcam directly targets the pupil of the user. It focuses directly on the pupil of the user. In other systems, the webcam did not detect the pupil of the user  Real-time pupil tracking.
The pupil of an individual is tracked in the real time. There is no delay while tracking a user"s pupil.
 Fast and easy mouse cursor movement.
The movement of the mouse is fast and accurate and all the movement is done in real time.
 Fast clicking of the mouse The clicking is done fast and immediate. The clicking is done by using pupil through the webcam XI. CONCLUSION As a final review in this research, the system aims to deliver a low-cost eye-tracker that will allow the user to control the mouse cursor of a computer system. The designed system is low cost and efficient, utilizing only a camera (Microsoft LifeCam Hd-6000), software modules coded in C++ and Python Programming language. If required, the spatial field of view history can also be drawn on the world process, showing eye movements and where the user spent a lot of time looking, to tweak the interface or to just retrieve spatial attention data for the purposes mentioned in the "future applications" section. Finally, we note that the project is operable in variable environmental conditions, only a few tweaks in the brightness and contrast setting need to be applied for it to maintain its robustness. This is an impressive feat for such a low-cost eye-tracking system.

A. Present Work
Eye tracking has become an important field of research recently. Sony is working on developing and incorporating eye tracker in their popular PlayStation platform. A Berlin developer introduced this technology through SensoMotoric Instruments [34]. It uses RED-OEM platform among its different functions which highlights the things you are looking at, enlarging the list of options which helps to simply track and respond to your queries. Oculus Rift is a virtual reality HCI interface which is the brainchild of Kickstarter.com and relies on heavy user gaze data to reposition the field of view to give the illusion of reality.

1) Search Engine Optimization
Search engine optimization (SEO) for websites and businesses can be done effectively by employing eye tracking based information and data analytics. This potential area of research is being spearheaded by Microsoft and Google [35], [36]. Results from Google search engine can be seen in Fig. 14 and in 0.

2) Market Research and Advertising Testing
Eye-tracker can play a major role in marketing. To understand the demands of the customer eye trackers are placed in marker and they determine for how much time the user is starting at the product and what is its reaction to the qualities or price of the of the product. This will help the manufacturers to make products that meet with the customer"s requirements and in return, their revenue will increase.
A commercial eye tracker scans and zeros in on which products the customer is scanning and where on those products is his gaze concentrating. An example of a dishwasher is shown in 0 with blue and pink markers showing where the customer"s gaze concentrated and the radius of marker depicts the duration for which the customer"s gaze was hooked on to that particular position. We can see the widest markers and most markers, in general, are concentrated towards the product label and this gives important information to manufacturers and advertisers alike that they should make the labels visually appealing, among other customer behavior information.