Error Analysis of Line of Sight Estimation using Purkinje Images for Eye-based Human-Computer Interaction : HCI

Error analysis of line of sight estimation using Purkinje images for eye-based Human-Computer Interaction: HCI is conducted. Double Purkinje images which are obtained by two points of light sources are used in the proposed method for eye rotation angle estimation. It aimed at the improvement of the eyeball rotation angle accuracy by simply presuming the value of the curvature radius of the cornea. This technique is a cancellation of the time of the calibration that is the problem of past glance presumption. By presuming the size of the radius of curvature of the cornea. As a result, the eyeball rotation angle presumption accuracy of about 0.98deg was obtained 0.57deg and horizontally in the vertical direction without doing the calibration. Keywords—Computer input just by sight; Computer input by human eyes only; Purkinje image


I. INTRODUCTION
As for the computer input system with human eyes only based on an image-analysis method, many methods have being proposed so far.Matsuda et al. makes the line of sight which connects eyeball rotation center coordinates and a pupil center, and is performing gaze measurement [1].Eyeball rotation center coordinates are searched for by moving an eyeball in the various directions before gaze measurement.Therefore, since there is no necessity of showing an index, gaze measurement is possible in all places, but preparation takes time and there is a fault of not permitting head movement.Moreover, Ono et al. makes the line of sight which connects cornea center-of-curvature coordinates and a pupil center, and is performing gaze measurement [2].
Cornea center-of-curvature coordinates are the light source of one point, and installing a camera on the optical axis, and are searched for using the general cornea radius-of-curvature value.However, since they assumed models, such as Japanese typical eyeball form, the error remained in the direction estimation of a gaze not a little, and if these methods did not perform a calibration, when there were, they did not become.That is, in order to cancel the gaze estimation error based on the gap of a central fovea to an eyeball center, the refraction in a cornea, and the individual difference concerning the form of a cornea, the calibration which draws a gaze correction coefficient needed to be performed by gazing at two or more indices displayed on the display one by one [3].Moreover, since it was not what permits a motion of a user, the burden has been forced upon the user.The former presumes the point of regard on the display at the point of the look obtained from gaze estimation (three dimension measurement) of not only the direction of a gaze but both eyes, and uses the pupil center and the corneal reflex center for gaze estimation.The latter presumes a cornea center of curvature using two Purkinje images, makes a user gaze at the index of three points, presumes an eyeball rotation center, is the method of making a gaze the straight line which connects these, and is verifying accuracy by the experiment using a model eye [4].
There are some methods which allow gaze estimations and its applications for HCI [5]- [35].In particular, paper 13 describes the method for gaze detection and line of sight estimation.In this paper, an error analysis is made for the previously proposed method.For the method, an expensive stereo camera is not needed, but only a cheap simple eye camera permits a motion of a user, and the method of determining the direction of a look from a pupil center and a cornea center of curvature is proposed without the calibration which forces a user a gaze of three points.By specifically measuring an eyeball cornea curvature radius simply, the degree estimation of eyeball rotation angle which does not need a calibration is performed, details are extracted from a face picture, the posture of a head is detected from those relative spatial relationships, and a motion of a head is permitted.The light source of two points was used for measurement of the cornea curvature radius of an eyeball, and two Purkinje images obtained from the cornea surface were used for it.At this time, it decided to also use together the near infrared light source which a camera has using the nearinfrared camera which became budget prices, and to acquire the clear Purkinje image in recent years.When five subjects estimate the direction estimation accuracy of a look for this, a motion of the head of the 30 roll directions and the 15 directions of a pitch is permitted, and since it checked that the direction of a gaze could be presumed without a calibration with the error of 0.57 to 0.98 degrees, it reports here.
The angle estimation method of eyeball rotation angle of using two light sources and one near infrared camera is proposed first, the head angle detection method using the details in a face picture is described, and this paper estimates the validity of the proposed method by the direction estimation experiment of a look by five subjective examiners.www.ijarai.thesai.org The following section describes the proposed line of sight vector estimation with Purkinje images briefly followed by the proposed error analysis method.Then experimental results are described together with some remarks.Finally, conclusion is described with some discussions and future investigations.

A. Eye model
Fig. 1 (a) shows eye shape model while Fig. 1 (b) shows the definitions of Purkinje images of the first to the fourth Purkinje images.The size and the curvature of cornea, sclera, retina, and eyeball are different for everybody.Therefore, calibration is required before using computer input just by sight.It is possible to estimate the size and the curvature by using the locations of the first to the fourth Purkinje images.The line of sight is defined as the line starting from the cornea curvature center which is estimated with Purkinje images to pupil center.

B. Procedure for estimation of fixation point on display at which user is looking
The procedure for estimation of fixation point on display at which user is looking is as follows, 1) Cornea curvature radius is estimated with double Purkinje images 2) Pupil center is determined with ellipsoidal approximation of pupil shape 3) Cornea curvature center is determined with geometric relations among eyeball, camera, display and light sources 4) Line of sight is determined with the cornea curvature center and pupil center 5) Fixation point on the display is determined with the line of sight vector Fig. 2 shows the method for estimation of cornea curvature center and radius.L and H are given.The distance between two Purkinje images can be measured as follows, 1) binarize the acquired NIR image of the eye and its surroundings, 2) isolated noise pixels are removed by using morphological filter,

3) the distance between the locations of two Purkinje images is measured
This procedure is illustrated in the Fig. 3. Thus the cornea curvature radius can be estimated.Through this procedure, the relation between the distance between the eye and the camera and the number of pixels within 10cm of distance interval is shown in Fig. 4.Meanwhile, the estimated cornea curvature radius is shown in Fig. 5 as a function of the distance from the eye.These can be derived from the following equation representing the cornea curvature radius.
(1) Next, a pupillary zone is extracted using a dark pupil method, ellipse approximation of the pupil form is carried out, and a pupil center is searched for.With a dark pupil method, if an infrared floodlight is installed in the position distant from the lens optic axis of a camera and an eyeball is illuminated, the reflection of light of the pupil portion of the illuminated eyeball will be lost, and the pupil portion of the eyeball picture caught with the camera will use the character to become dark here.Image processing extracts this dark portion.www.ijarai.thesai.orgMoreover, in the extracted pupillary zone, since reflected lighting and the up-and-down portion of a pupil hide and are missing by the eyelid, the pupil may not be able to be extracted correctly.Therefore, edge is detected from the obtained temporary pupillary zone, and the edge concerned is approximated to an ellipse by a least squares method.However, when it calculates using except a pupil outline, there is a possibility of resembling the mistaken ellipse.Therefore, pupil ellipse approximation using the character of an ellipse was performed.
That is, in Fig. 6, straight lines l, m, and n shall be parallel, and l and n shall exist in the equal distance from m.The intersection of an ellipse and a straight line l is set to a and b, the intersection of an ellipse and a straight line n is set to c and d, and the middle point of the intersection of an ellipse and a straight line m is set to o.When the middle point of a line which connects the middle point of a and b and the middle point of c and d is set to o 0 , there is character in which o 0 overlaps with o.Least square method for elliptic approximation is illustrated in Fig. 7.The point of hitting o i0 (i = 1-N) of the group of the parallel lines is searched for.The point searched for is distributed on a straight line m.When there are few noises enough, the position in which most many points o 0 gathered is equivalent to the position of o.Since the point which is distant from the www.ijarai.thesai.orgposition will include the point which is not on the locus of an ellipse, it accepts them.o point a i which is alike and has sufficiently near o i0 ; b i ; c i ; Since d i is a point on an ellipse, ellipse approximation of an exact pupil outline can be performed by using them.
Next, a cornea center of curvature is searched for from the spatial relationship of the acquired cornea curvature radius and the light source of Fig. 8, an eyeball, and a camera, and the geometric expression of relations in a Figure .A cornea center of curvature exists here on the bisector which ties a camera, the Purkinje image, and lighting.It asks for a look vector from a pupil center and a cornea center of curvature.
Finally, the point of regard on a display is computed from a look vector.The vector which passes along two points, a cornea center of curvature and a pupil center, is made into a look vector, and the point of regard on a display is computed.It is considered as a camera position (0, 0, 0), and is considered as the main coordinates (0, 0z) of the picture picturized with the camera.If a line of sight vector is made into v = (x v ; y v ; z v ), a camera, lighting, and a display assume that it is being fixed and the head is also being fixed, and are z.The distance of the direction of an axis is known.When distance of an eyeball and a display is set to z h , point-of-regard coordinates t = on a display (x t ; y t ) is, (2) so that the gaze fixed point can be calculated as is shown in Fig. 9.Moreover, the picture acquired from the camera on the occasion of look measurement was processed in real time, and performed accuracy verification.

B. Preliminary Experimental Results
The measurement result of a cornea curvature radius is shown in Fig. 12.The spike in a Figure is based on incorrect detection of the Purkinje image, and these can be accepted from measurement of a cornea radius.Moreover, two lines in a Figure are the maximum errors of the cornea curvatureradius point estimate considered when movement of ±1.0cm has a head in the display direction.The average of the cornea curvature-radius point estimate except an edge portion is used for the point estimate of the cornea curvature radius used for the actual degree estimation of eyeball rotation angle.The value of a cornea curvature radius was set to R = 7.92 mm from this result.

C. Sensitivity and Stability
How to permit a motion of a head finally is shown.The both ends of eye and a mouth, the middle point between two ends of eye, and the middle point of the both ends of a mouth are detected from a face picture.Template matching is used for pursuit of details at these details extraction using OpenCV.By the middle point of both eyes, and the middle point of a mouth, three square shapes can be constituted and a plane can be defined.The normal direction of this plane is judged to be the posture of a head.Once extracting these details, by using template matching which can pursue most in a short time, it is devising so that the pursuit in real time may be possible.An example of details extraction and pursuit is shown in Fig. 13.The distance between the details in a Figure is found and the degree of rotation angle of a head can be presumed in comparison with the position of the details extracted in early stages by OpenCV.The blue line segment of Fig. 13 is a head posture (the plane normal direction is shown.).Moreover, a green line segment connects the middle point between the middle points of both eyes, and the middle point of the both ends of a mouth.
The geometric relation between the rotation center of a head and an eyeball rotation center is shown in Fig. 14, and can presume an eyeball rotation center position after this.If the method of presuming a look from an eyeball rotation center position is considered to be the same thing as the above-mentioned method, a motion of a head will be permitted and the direction estimation of a look will be www.ijarai.thesai.orgattained.At this time, that the both ends of both eyes and the both ends of a mouth are not occluded are the conditions which presume a head rotation center and an eyeball rotation center, and 30 degrees and about 15 degrees are rotation allowable limits in the roll direction and the direction of a pitch.
As described before, movement in the direction of z is compared with movement to x and a y direction, and has serious influence on eyeball angle detection.Therefore, I need to get a user to maintain distance with a computer display to some extent (separating beyond the defined distance and twisting like).As also described before, to rotation of x of a head, the parallel translation of a y direction, a roll, a pitch, and the direction of Yaw, it is detectable.As for parallel translation, it is highly accurate and can presume correctly about 100%.Therefore, the degree of setting head rotation angle and average estimation error by five subjects were checked by experiment here.An experimental result is shown in Fig. 15.Since the Yaw rotation of a head is rotation in a head plane, a presumed error is comparatively small.The presumed error of the direction of a pitch was the largest, and it turned out that the presumed error of about 5 times is produced also in rotation of about 1.0 degrees in the pitch rotation to a direction which especially bows.Even if it is relatively stout and rotates about 30 degrees to the pitch rotation to a direction which raises the head, the presumed error is settled in about 1.7 degrees.Moreover, the presumed error over rotation of the head of the roll direction was located in that middle, and when it was ±30 roll rotations, it has checked fitting in the presumed error of less than 1.7 degrees.Furthermore, one of eyes occlusion roll rotation exceeding 40 degrees, and a presumed error becomes large rapidly.Therefore, ±30-10~+30 degrees, even if the head rotated ±30 degrees in a roll, a pitch, and each direction of Yaw, it checked fitting in the presumed error of less than 1.7 degrees to them.A cornea curvature radius is presumed using the Purkinje image of two points obtained by using the two-point light source proposed in this paper, the proposed method of eyeball rotation angle estimation which does not need the calibration for individual difference dissolution of cornea curvature are 0.57 degree and y to the direction of x in the state where the head was fixed in simple.It turned out that the degree of eyeball rotation angle can be presumed with the error of about 0.98 degree in a direction.Moreover, since the degree estimation method of head rotation angle using the details of the face proposed in this paper is the presumed error of about 1.7 degrees under restriction of the degree of rotation angle, it assumes that rotation of these heads and rotation of an eyeball are independent, and it is RSS: Root Sum Square is taken, it turns out that the eyeball angle estimation accuracy in the case of permitting rotation of a head is 1.988 degrees in a x direction at 1.821 degrees and a y direction.
If a look is presumed and rotation is permitted for a head in the above-mentioned restriction range from the face picture acquired from the camera which separated 150mm, the direction of a look can be presumed in the error of about 2 times.When an intersection with a look is put on the computer www.ijarai.thesai.orgdisplay which left this 300mm, it is equivalent to about 10.472mm look and a display intersection position estimation error.When a with a pixel size (4 pixels / 10mm) computer display is assumed, this intersection position estimation error will be equivalent to about 4 pixels.Fig. 16 sets up a target on 7 pixels from a center as an example of look stability, gets five subjects to gaze at the target concerned for 30 seconds, and shows what plotted the presumed intersection position.Therefore, by making distance between adjacent keys into 10 pixels or more showed that rotation of a head was permitted and the degree estimation method of eyeball rotation angle which does not need a calibration could be realized.

D. Error Analysis
In the experiment, eyeball angle estimation accuracy was checked by seeing five indices displayed on the display one by one, where a head is fixed simply.The presumed result of the eyeball angle detection from which a cornea curvature radius differs is shown in Fig. 17.It is delta x and y when a cornea curvature radius is set to R = 7.92 mm at this time.Δ xy = it is 1.128.Near the measured cornea curvature radius has the highest degree detection accuracy of eyeball rotation angle.Therefore, it has checked that rather than led which uses the measured cornea curvature radius to improvement in the degree detection accuracy of eyeball rotation angle using the standard value of a cornea curvature radius.
Change of the look estimation result of survey by head parallel translation and rotation is shown in Fig. 18.There are two sets of the error analysis results, delta x,y,z and mean of delta x,y,z.Fig. 18 (a) and (b) show the two cases for cornea curvature radius estimation.On the other hand, the error analysis results for head angle estimation in directions of x,y,z are shown in Fig. 18 (c), (d) and (e), respectively.Meanwhile, sensitivity analysis results for delta x,y,z and mean of delta x,y,z in directions of roll, pitch and yaw angles are shown in Fig. 18

(f) to (i)
In measurement of a cornea curvature radius, it seems that the big accuracy fall has occurred when this is exceeded since it has measured in 3.0-5.0mmfrom the cornea center-ofcurvature part.Moreover, in the influence on the presumed accuracy by the head rotation to the direction of a pitch, since the influence of eyelashes becomes large in downward rotation of a head, pupil detection becomes difficult (measurement is impossible), and since the influence of upward rotation of a head of eyelashes decreases, its pupil detection accuracy improves.From this, the camera position at the time of measurement is understood that it is desirable to install by arrangement which looks up at a head.This paper proposed the estimation method of eyeball rotation angle which does not need the calibration for individual difference dissolution of cornea curvature by presuming a cornea curvature radius using the Purkinje image of two points obtained by using a two-point light source.As a result, in the state where the head was fixed in simple, they are 0.98deg(s) to 0.57deg(s) and the direction of y in the direction of x.The degree estimation accuracy of eyeball rotation angle was acquired with the error of a grade.Moreover, by the angle estimation method of the plane defined by the details of the face proposed since rotation of a head was permitted, when preparing restriction in each rotation of a roll, a pitch, and a yaw direction, it checked that the degree of rotation angle could be presumed with the error of about 0.5 degree.When it assumed that rotation of these heads and rotation of an eyeball were independent, it turned out that the eyeball angle estimation accuracy in the case of permitting rotation of a head is 1.988 degrees in a x direction at 1.821 degrees and a y direction.That is, since the degree of viewing angle beyond this error was detectable, when detaching and setting up the interval of a key rather than this, it turned out that it can guarantee that discernment of a key is possible.

L:Fig. 2 .
Fig. 2. Method for estimation of cornea curvature center and radius

Fig. 4 .Fig. 5 .
Fig. 4. Relation between the distance between the eye and the camera and the number of pixels within 10cm of distance interval

Fig. 7 .
Fig. 7. Least square method for elliptic approximation N which is in the equal distance about a temporary pupillary zone from the straight line m drawn in the center.The point of hitting o i0 (i = 1-N) of the group of the parallel lines is searched for.The point searched for is distributed on a straight line m.When there are few noises enough, the position in which most many points o 0 gathered is equivalent to the position of o.Since the point which is distant from the

Fig. 8 .
Fig. 8. Geometric relation among light source, camera and eyeball Fig. 9. Method for gaze fixed position During the process, ellipsoidal contour and feature points extraction can be done with the procedure as shown in Fig.10.

Fig. 10 .
Fig. 10.Procedure for ellipsoidal contour and feature points extraction Fig. Experimental configuration for error analysis The parameters of the experimental configuration are as follows, L = 1000 mm; HC = 670 mm; HT = 150 mm; HD = 380mm Software development environment is as follows, Microsoft Visual C++ Microsoft Visual studio.NET2 OpenCV 1.0

Fig. 13 .
Fig. 13.Method for head pose angle estimation using the extracted feature points from the acquired face image

Fig. 14 .
Fig. 14.Geometric relation between the rotation center of a head and an eyeball rotation center

Fig. 16 .
Fig. 16.Stability of the estimated location of fixed point

Fig. 17 .
Fig. 17. presumed result of the eyeball angle detection from which a cornea curvature radius differs