Line of Sight Estimation Accuracy Improvement using Depth Image and Ellipsoidal Model of Cornea Curvature

Line of sight estimation accuracy improvement is attempted using depth image (distance between user and display) and ellipsoidal model (shape of user’s eye) of cornea curvature. It is strongly required to improve line of sight estimation accuracy for perfect computer input by human eyes only. The conventional method for line of sight estimation is based on the approximation of cornea shape with ellipse function in the acquired eye image. The proposed estimation method is based on the approximation of crystalline lenses and cornea with ellipsoidal function. Therefore, much accurate approximation can be performed by the proposed method. Through experiments, it is found that depth images are useful for improvement of the line of sight estimation accuracy. Keywords—Computer input just by sight; Computer input by human eyes only; Purkinje image; Cornea curvature


INTRODUCTION
There are some methods which allow gaze estimations and its applications for Human Computer Interaction: HCI [1]- [31].Paper [9] describes the method for gaze detection and line of sight estimation.In the paper, an error analysis is made for the previously proposed method.For the method, an expensive stereo camera is not needed, but only a cheap simple eye camera permits a motion of a user, and the method of determining the direction of a look from a pupil center and a cornea center of curvature is proposed without the calibration which forces a user a gaze of three points.
By specifically measuring an eyeball cornea curvature radius simply, the degree estimation of eyeball rotation angle which does not need a calibration is performed, details are extracted from a face picture, the posture of a head is detected from those relative spatial relationships, and a motion of a head is permitted.The light source of two points is used for measurement of the cornea curvature radius of an eyeball, and two Purkinje images obtained from the cornea surface were used for it.It is decided to also use together the near infrared light source which a camera has using the near-infrared camera which became budget prices, and to acquire the clear Purkinje image in recent years.
One of the weak points of the existing method for gaze estimation is that line of sight estimation accuracy is not so high when user moves away from the display and getting close to the display.Also, ellipse model of cornea shape is not so appropriate for human eyes.In the paper, these two problems are solved and overcome using raging image (Kinect acquires the depth between user and the display) and ellipsoidal shape model for estimation of cornea curvature.
The following section describes the proposed line of sight estimation accuracy improvement followed by some experiments.Then conclusions are described together with some discussions and future research works.

A. Eye Model
Fig. 1 (a) shows eye shape model while Fig. 1 (b) shows the definitions of Purkinje images of the first to the fourth Purkinje images.The size and the curvature of cornea, sclera, retina, and eyeball are different for everybody.Therefore, calibration is required before using computer input just by sight.It is possible to estimate the size and the curvature by using the locations of the first to the fourth Purkinje images.The line of sight is defined as the line starting from the cornea curvature center which is estimated with Purkinje images to pupil center.

B. Procedure for Estimation of Gaze Location on Display at Which User is Looking
The procedure for estimation of gaze location on display at which user is looking is as follows, 1) Cornea curvature radius is estimated with double Purkinje images 2) Pupil center is determined with ellipse approximation of pupil shape 3) Cornea curvature center is determined with geometric relations among eyeball, camera, display and light sources (See Appendix), 4) Line of sight is determined with the cornea curvature center and pupil center 5) Gaze location on the display is determined with the line of sight vector Fig. 2 shows the method for estimation of cornea curvature center and radius.L and H are given.The distance between two Purkinje images can be measured as follows, 1) binarize the acquired NIR image of the eye and its surroundings, 2) isolated noise pixels are removed by using morphological filter,

3) the distance between the locations of two Purkinje images is measured
This procedure is illustrated in the Fig. 3. Thus, the cornea curvature radius can be estimated.Distance between two light sources, Distance between camera and eyeball, Distance between two Purkinje images, Cornea curvature radius can be derived from the following equation representing the cornea curvature radius.

C. Improvement of Gaze Location Estimation Accuracy with
Depth Images Using Kinect Fig. 4 shows the set-up configuration of the proposed gaze location estimation with Kinect.Major specification and outlook of Kinect (v2) is shown in Table 1 and Fig. 5, respectively.Meanwhile, major specification of NIR camera of DC-NCR13U is shown in Table 2.

D. Preliminary Experiments
The proposed procedure for estimation of line of sight (Gaze location on display is as follows,  Distance between camera and pupil can be expressed in equation ( 2) which is related to the distance between Kinect and pupil.
In accordance with the Fig. 8, NIR image coordinate system can be converted to NIR camera coordinate system.Line of sight vector V is defined as the vector which is situated on the line G of pupil center and cornea curvature center O as shown in the equation ( 5).

G=O+tV
(5 where t denotes mediating variable.Z axis at the gaze location on display has to be zero.Therefore, 0=O z +tV z (6) Thus, the gaze location in unit of mm is expressed as the equation ( 7).where dpi, width, height is defined as dot per inch, display width and display height, respectively.

III. EXPERIMENTS
Cornea radius is assumed to be 7.92 mm for the previous experiences.The distance between iris and display is varied from 300, 310 and 320 mm.20 trials rare conducted for each distance.The estimated gaze locations and ideal viewpoint are scattered as shown in Fig. 10.Particularly, the gaze location estimation error is evaluated for mean and standard deviation.As a conclusion, it is found that estimation error of gaze location with Kinect (distance information can be used) is superior to that without Kinect by the factor of 10 to 100%.

IV. CONCLUSION
Line of sight estimation accuracy improvement is attempted using depth image (distance between user and display) and ellipsoidal model (shape of user's eye) of cornea curvature.Through experiments, it is found that depth images are useful for improvement of the line of sight estimation accuracy.Particularly, the gaze location estimation error is evaluated for mean and standard deviation.As a conclusion, it is found that estimation error of gaze location with Kinect (distance information can be used) is superior to that without Kinect by the factor of 10 to 100%.
Further investigations are required for simultaneous estimation of cornea curvature center and cornea radius, noise removal of the depth image.

APPENDIX: ELLIPSOIDAL APPROXIMATION OF THE SHAPE OF CHRISTALLINE LENSE AND CORNEA OF EYE
The shape of crystalline and cornea of the eye is assumed to be ellipsoid and can be approximated with the acquired eye image and Purkinje images based on the proposed ellipsoidal model shown in the following figure (Fig. A1).In the figure, 3D object of the shape of the extracted eye in the 3D coordinate system (x,y,z) can be expressed with λ 、φ、θ.The internal points of the ellipsoid is represented with the equation (A1).The second order moment of the ellipsoid around the origin point is then expressed with the equation (A4).M 0 =∫∫∫(a 2 +b 2 +c 2 ) da db dc = (1/5)V(A 2 +B 2 +C 2 ) (A4) Also, the second order moments of ellipsoid around a, b, c axis can be represented with the equation (A5).The volume can be represented with the equation (A12

L:Fig. 2 .
Fig. 2. Method for estimation of cornea curvature center and radius

Fig. 3 .
Fig. 3. Procedure of the cornea curvature radius measurement Fig. 4. Set-up configuration of the proposed gaze location estimation with Kinect On the other hand, Dlib which is developed by Davis E. King in 2002 is used for face detection function Also, Open_CV is used for image acquisition and manipulations.

1 )
IR image is acquired with DC-NCR-13U of NIR camera with NIR LED a) Pupil center and Purkinje image center is detected from the acquired image 2) NIR image and depth image is acquired with Kinect a) Iris is detected from the acquired depth image b) Distance between the iris and Kinect is estimated with the depth image c) Distance between the iris and display is estimated with the depth image 3) Cornea curvature center is estimated 4) Lune of sight (gaze location on the display) is estimated Fig.6 (a) shows an example of eye image extracted with Dlib software tool.The extracted eye image is binarized and labeled image is created from the binarized image.Then ellipse matching is performed through function matching.Finally, iris center is detected as shown in Fig.6 (b).Meanwhile, Purkinje center is detected with the binarized image derived from the acquired original eye image as shown in Fig.6 (c).

Fig. 7 .
Fig. 6.Examples of iris and Purkinje center detection

Fig. 8 .
Fig. 8. NIR image coordinate system can be converted to NIR camera coordinate system.Thus, X is calculated with the equation (3).X=D camera-pupile tan[(θ/2){(x-w/2)/w/2}] (3) Next thing we have to do is to estimate cornea curvature center.Using the geometrical relation among NIR camera, NIR LED and Purkinje image center which is illustrated in Fig.9, cornea curvature center is estimated in the equation (4) because cornea curvature center is situated on the line which divide the angle among Purkinje image center, NIR camera and NIR LED.PE=-CP-LP PO=-R(PE/|PE|) (4) CO=CP+PO

Fig. 10 .
Fig. 10.Estimated gaze location and ideal viewpoint It is found that the estimated gaze locations without Kinect are scattered much diversely compared to those with Kinect obviously.

TABLE III .
GAZE LOCATION ESTIMATION ERROR IN UNIT OF DEGREE (A) AND PIXEL (B) ).