Mobile Phone Operations using Human Eyes Only and its Applications

Mobile phone operations using human eyes only is proposed together with its applications for cooking with referring to recipes and manufacturing with referring to manuals, production procedure, so on. It is found that most of mobile phone operations can be done without touching the screen of the mobile phone. Also, mobile phone operation success rate based on the proposed method is evaluated for the environmental illumination conditions, visible or near infrared (NIR) cameras, the distance between user and mobile phone, as well as pupil size detection accuracy against the environmental illumination changes. Meanwhile, the functionality of two typical applications of the proposed method is confirmed successfully. Keywords—Mobile phone operations; line of sight estimation; gaze estimation; wearable computing; pupil detection


INTRODUCTION
Evaluation of users' impact for using the proposed Eye Based Human-Computer Interaction: EBHCI with moving and fixed keyboard by using EEG signals is investigated [1].Electric wheel chair controlled by human eyes only with obstacle avoidance is proposed [2].Robot arm control with human eyes only and its application to help having meal for patients is proposed [3].These methods and systems are overviewed (Human-Computer Interaction with human eyes only and its applications) [4].
A new keyboard for improving accuracy and minimizing fatigue effect is invented [5] followed by moving keyboard for EBHCI [6].Service robot which is controlled by human eyes only with voice communication capability is developed as an example of EBHCI [7] followed by eye-based domestic robot allowing patient to be self-services and communications remotely [8].
Method for psychological status estimation by gaze location monitoring using EBHCI is proposed [9] followed by method for psychological status monitoring with line of sight vector changes (Human eyes movements) detected with wearing glass [10].It becomes wearable computing system with input output devices based on EBHCI allowing location based web services [11].
Speed and vibration performance as well as obstacle avoidance performance of electric wheel chair controlled by human eyes only is evaluated [12].Service robot with communication aid together with routing controlled by human eyes is overviewed [13].On the other hand, information collection service system by human eyes for disable persons is developed [14].Meanwhile, the relation between psychological status and eye movement is clarified [15].
Method for 3D image representation with reducing the number of frames based on characteristics of human eyes is proposed [16].On the other hand, error analysis of line of sight estimation using Purkinje images for EBHCI is made [17].Then, new service robot controlled by human eye which allows patients in hospitals self-services remotely is overviewed [18].
In the paper, mobile phone operations using human eyes only is proposed together with its applications for cocking with referring to recipes and manufacturing with referring to manuals, production procedure, so on.Although EBHCI is computer based method and systems, the proposed system EBHCI can be realized with mobile phone.Without touching the screen of the mobile phone, some of the operations can be done with human eyes only.
The following section describes the proposed system.Then, experiments are described with some experimental results of the mobile operation success rate followed by some applications of the proposed system.Finally, conclusion is described with some discussions.

A. Proposed Methods
Fig. 1 shows the proposed gaze estimation procedure.With Near Infrared: NIR camera, face image is acquired followed by face detection by using OpenCV.Then, eye detection is made with OpenCV followed by pupil detection with ellipsoidal shape approximation.Also, Purkinje image which is acquired with six NIR Light Emission Diodes: LEDs are used for estimation of cornea curvature and curvature center.
Example of the acquired Purkinje image (Red ellipsoid) and pupil (Blue ellipsoid) of which the shape is approximated with ellipsoid is shown in Fig. 2(a).The Purkinje images are aligned with ellipsoidal shape because the reflected light of six LEDs at the surface of the eye is aligned with the hexagonal shape as is shown in Fig. 2(b).The line of sight is defined as the line on the curvature center and pupil center.Using the estimated line of sight, gaze location is estimated.It, however, is not stable enough during the period of the gaze estimation, every 0.5 sec.It is shorter than 0.3 sec.for typical accidental blink time.Therefore, it is intentional blink if user close their www.ijacsa.thesai.orgeye for much longer than 0.3 sec.Thus, the line of sight can be estimated every 0.5 sec.Pupil detection procedure is illustrated in Fig. 3.The acquired eye image is binarized first, then edge is detected with Canny filter.After that, ellipsoidal shapes are selected from the edge images as candidates of the pupil.Then, the shorter and the longer radius is estimated followed by pupil center determination of which the two radius are crossing at the pupil center.

B. Mobile Phone Operation Selection
Fig. 4 shows the designed four mobile phone operations with the estimated gaze location and blinks.Due to the fact that line of sight estimation cannot be made accurately and stable, just four operations can be detected with the estimated line of sights.These four operations would be enough for choosing the required operations of the applications of the proposed method.It, however, is still difficult to identify the direction of which user is looking to.Therefore, the proposed method defines the direction by using the difference between the estimated eye center and pupil center.It is relatively easy to find the eye center because the two ends of eye can be identified using acquired eye images.Fig. 5 shows the estimated gaze location when user is looking to up and down directions as well as right and left directions.It is relatively easy to discriminate for to right and left directions while it is not so easy to discriminate for up and down directions.Therefore, just the discrimination of right and left directions is used.

B. Threshold Determination using Standard Deviation of the Difference between Eye Center and Pupil Center
Meanwhile, Fig. 7 shows the difference between eye center and pupil center of right and left eyes when user is looking to right and left directions.In the figure, average and standard deviation of the difference between eye center and pupil center are also shown with the black bars.It is relatively easy to discriminate the direction of which user is looking to.Thus, the direction of which user is looking to can be detected.
As shown in Fig. 7(a) and (b), it can be decided that the user is now looking to the left direction because the difference between eye center and pupil center is lower than the standard deviation for a while.Meanwhile, it also can be recognized the user is now looking to the right direction because the difference is larger than the standard deviation for a while as shown in Fig. 7(c) and (d).

C. Evaluation of Probability Density Function of the Difference between Eye Center and Pupil Center
Probability Density Function: PDF of the difference between eye center and pupil center is evaluated as a function of the distance between user and mobile phone NIR camera.As the result of the experiment, shown in Fig. 8, it is found that shortest distance of 15 cm shows the best performance in terms of identification if the direction of which the user is looking to.

D. Illumination Condition
On the other hand, the area and the short radius as well as the long radius of the ellipsoid of which pupil shape is approximated depend on the illumination conditions.Fig. 9(a) shows the area which is determined with the short radius over 2 multiplied by the long radius over 2 of the ellipsoid of which pupil shape multiplied byπis approximated for the right eye while those of the left eye is shown in Fig. 9(b), respectively.The grey colored line shows the right eye area for the illumination condition of 1 to 10 Lux, while red colored line shows that of 40 to 50 Lux.On the other hand, the blue colored line shows that of 140 to 150 Lux, respectively.Those are same thing for the left eye of the illumination condition.It is obvious that the area and the short radius as well as the long radius of the ellipsoid of which pupil shape is approximated is getting smaller when the brighter illumination condition.IV.EXAMPLES OF APPLICATION One of the applications of the proposed system is cocking with referring to a recipe.Fig. 11 shows an example of mobile phone operation with look direction and blink detections for the cocking with referring to the recipe.Without touching to mobile phone, user can use mobile phone operation, page forward, backward, showing the content, as well as the menu for cocking.Users' response is positive for all.Fig. 11(a) shows the start menu (main menu) while Fig. 11(b) shows the sub-menu when the user selected the second recipe from the main menu by using the right eye close (user select the desired downward direction of sub menu).
Another example is construction of houses and buildings with referring to design drawings as well as construction procedures without touching screen of the display of their mobile phone.Constructing workers can refer the documents and drawings when they working at construction sites in a safe manner.V. CONCLUSION Mobile phone operations using human eyes only are proposed together with its applications for cocking with referring to recipes and manufacturing with referring to manuals, production procedure, and so on.It is found that most of mobile phone operations can be done without touching the screen of the mobile phone.Also, mobile phone operation success rate based on the proposed method is evaluated for the environmental illumination conditions, visible or Near Infrared: NIR cameras, the distance between user and mobile phone, as well as pupil size detection Meanwhile, the functionality of two typical applications of the proposed method is confirmed successfully.
Further investigation is required for application of the proposed method and system.

Fig. 4 .
Fig. 4. Four mobile phone operations with the estimated gaze location and blinks.

Fig. 5 .Fig. 6 .
Fig. 5.Estimated gaze location when user is looking to up and down directions as well as right and left directions.III.EXPERIMENT A. Experimental Configuration Fig. 6(a) shows the experimental configuration.The distance between user and NIR camera and LED is set at 15, 20, 25, and 30 cm.Illuminations are set at three conditions, 40 to 50 Lux, 140 to 150 Lux.On the other hand, Fig. 6(b) shows the outlook of the experimental configuration.

Fig. 7 .
Fig. 7.Estimated gaze locations when user is looking to right and left directions.

FrameFig. 8 .
Fig. 8. Probability density function of the difference between eye center and pupil center as a function of the distance between user and the NIR camera attached to the mobile phone.

Fig. 10 Fig. 9 .
Fig. 10 also shows the estimated short radius and the long radius of the ellipsoid of which the acquired pupil shape is approximated.
Fig. 11.Example of mobile phone operation with look direction and blink detections.