Autonomous Control of Eye Based Electric Wheel Chair with Obstacle Avoidance and Shortest Path Findings Based on Dijkstra Algorithm

—Autonomous Eye Based Electric Wheel Chair: EBEWC control system which allows handicap person (user) to control their EWC with their eyes only is proposed. Using EBEWC, user can move to anywhere they want on a same floor in a hospital autonomously with obstacle avoidance with visible camera and ultrasonic sensor. User also can control EBEWC by their eyes. The most appropriate route has to be determined with avoiding obstacles and then autonomous real time control has to be done. Such these processing time and autonomous obstacle avoidance together with the most appropriate route determination are important for the proposed EBEWC. All the required performances are evaluated and validated. Obstacles can be avoided using acquired images with forward looking camera. The proposed EBEWC system allows creation of floor layout map that contains obstacles locations in a real time basis. The created and updated maps can be share by the electric wheel chairs on a same floor of a hospital. Experimental data show that the system allows computer input (more than 80 keys) almost perfectly and electric wheel chair can be controlled with human eyes-only safely.


INTRODUCTION
Electric Wheel Chair: EWC for mobility support for handicap persons, in particular, is presented [1]- [5].There are some the proposed systems for mobility support using eye movements based on Electrooculography.EWC is controlled by eye movement which is acquired using Electrooculograph [1].Also there is the proposed integrated solution to motion planning and control with three input sources [2].User can specify their designated destination using a visual interface by looking at the designated functional keys displayed onto computer screen.This input makes the EWC control automatically and is generated by a deliberative plan incorporating with a prior knowledge.User also can use reactive controller to avoid obstacles and features that the sensor detect.User can directly provide velocity command using joystick or some other input devices as a basic level.There is the proposed vision-based navigation system for an electric wheelchair using ceiling light landmark [3].The EWC is equipped with two cameras those are used for self location and obstacle avoidance.The fluorescent ceiling lights are chosen as landmarks since they can be easily detected and do not require an additional installation.Also there is the proposed head gesture based control of an intelligent wheelchair [4].This system used Adaboost face detection 1 which is provided by OpenCV.By detecting frontal face and nose position, head pose and gesture are estimated and are used for control the EWC.There is the proposed EWC control by the detection of gaze direction and eye blinking [5].The gaze direction is expressed by horizontal angle of gaze.It is derived from the triangle form which is formed by the center position of eyes and nose.The gaze direction and eye blinking are used to provide the direction and timing command.The direction command related to the move direction of EWC and the timing command related to the time condition when EWC has to be moved.
In order to estimate gaze based on image analysis, it is common that gaze location is estimated with pupil location.The previous pupil detection methods can be divided into two categories: the active Infrared: IR-based approaches [7], [8], [16] and the traditional image-based passive approaches [9]- [15].Eye detection based on Hough transform is proposed [9]- [11].Hough transform is used for finding the pupil.Eye detection based on motion analysis is proposed [6], [7].Infrared light is used to capture the physiological properties of eyes (physical properties of pupils along with their dynamics and appearance to extract regions with eyes).Time series analysis of motion such as Kalman filter 2 and mean shift which are combined with Support Vector Machine: SVM3 used to estimate pupil location.Eye detection using adaptive threshold and morphologic filte4 r is proposed [12].Morphologic filter is used to eliminate undesired candidates for an eye and pupil.Hybrid eye detection using combination between color, edge and illumination is proposed [15].
We proposed the computer input system with human eyeonly and it application for EWC control [17].It is called Eye Based EWC: EBEWC hereafter.EBEWC works based on eye gaze.When user looks at appropriate angle/key, then computer www.ijacsa.thesai.orginput system will send command to EWC.EBEWC is controlled by three keys: left (Turn left), right (Turn right), and down (Move forward) in order to give command to electric wheelchair: turn left, turn right, and go forward [18].This three combination keys are more safely than others combination keys.The stop key is not required because EWC automatically stops when user changes the gaze.Gaze estimation performance is well reported in the previous paper [19] in terms of robustness against a variety of user types, illumination changes, vibration as well as gaze position accuracy, and calibration accuracy.Also EBEWC with obstacle avoidance control is described in the previous paper [20].In this paper, automatic EWC path finding algorithm and EWC control performance are discussed together with performance for ensure a real-time processing of all the required processes for autonomous control, in particular, obstacle avoidance with ultrasonic sensors and visible cameras.

II. PROPOSED EYE BASED ELECTRIC WHEEL CHAIR: EBEWC CONTROL
A. Hardware Configuration Fig. 1 shows the hardware configuration of the proposed EBEWC.The proposed system consists of (1) two cameras mounted glass, (2) NetBook PC, (3) Ultrasonic Sensor.Two cameras look at forward (finding obstacles) and backward (acquires users' eye image).The camera used 1.3 Mega pixel OrbiCam (Visible camera) and IR camera for gaze estimation.
The Yamaha JW-I type 5 of EWC is used.Netbook of Asus EeePC with 1.6 GHz Intel Atom processor, 1GB of RAM, 160GB hard drive, and run Windows XP Home edition is aslo used.We develop our software under C++ Visual Studio 2005 and Intel provided OpenCv image processing library [6].The proposed EBEWC system utilizes infrared web camera, NetCowBoy DC-NCR 131 as face image acquisition which works in a real time basis.This camera has IR Light Emission Diode: LED.Therefore, it is robust against illumination changes.Furthermore, pupil detection becomes much easy.In order to allow user movement and EWC vibration, IR camera mounted glass is used.The distance between camera and eye is set at 15.5 cm.The Yamaha JW-1 type of EWC is controlled by human eyes only through microcontroller of AT89S51 6 .This microcontroller can convert serial output from the Netbook to digital output for control.

B. EWC Control by Human Eyes Only
In order to control EWC, at least four keys, move forward, turn right, turn left and stop are required.For the safety reason, users have to look forward so that the key layout of Fig. 2 is proposed.Namely, key consists of 9 keys (3 by 3).Move forward and turn left/right are aligned on the middle row.Stop key is aligned on the other rows, top and bottom rows.Users understand the location of desired key so that it can be selected with users' eye-only without any computer screen.Basic actions of the EBEWC control are as follows, http://www.atmel.com/dyn/resources/prod_documents/doc2487.pdf  If eye detection fail then stop  If obstacle detected then stop  If user surprise then stop for safety reason.When user surprises, then typical eye shape becomes open widely and usually upward looking shape.
The backward looking camera whose resolution is 640 by 480 pixels acquires users' eye and its surrounding.Using OpenCV of eye detection and tracking installed on the Netbook PC, users' eye is detected and tracked.If the OpenCV cannot detect users' eye, then EWC is stopped for safety reason.EWC is also stopped when users look at the different location other than the three keys aligned on the center row.
Intentional blink can be detected if the eye is closed for more than 0.4 seconds because accidental blink is done within 0.3 seconds, typically.In this connection, it is easy to distinguish between intentional and accidental blink.Also, key selection can be done every 0.4 seconds.Thus the system recognizes user specified key every 0.4 seconds.In order to make sure the user specified key, 10 frames per seconds of frame rate is selected for backward looking camera.

C. Autonomous EWC Guide to Destinations
If user selects the top left corner of key, then the key layout is changed to Fig. 3.After that, user can select previously designated destination, 8 candidate destinations in this case.User selects the destination previously.For instance, nurse station is destination #1, rest room is destination #2 and so on.In this case, EBEWC goes to the destinations autonomously with a reference to the floor layout and with avoiding obstacles.Obstacle avoidance can be done with ultrasonic sensor and forward looking visible camera.The ultrasonic sensor is for transparent obstacle such as glass door, window, etc. and detection as well as avoidance in particular.When EWC detect the obstacle, it must understand where the best path should be chosen if EWC want to go to specific place.The most appropriate route will be chosen based on floor layout map and image map.Image map is created by acquiring background images in every 1 m as is shown in Fig. 4. Medical doctors, patients and nurses are obstacles.Even if the location cannot be detected with forward looking visible camera and ultrasonic sensor, their location can be acquired with wireless LAN connections.Access points of antennas are equipped at the four corners of a floor in a hospital.Therefore, the locations can be determined by using the delay times from the four antennas.These location information are gathered and are provided to the all the EWC on a floor.Thus EWC can avoid obstacles.
Before getting start to operate EBEWC, a template of one shot image is acquired with the forward looking camera at each location (x, y).For instance, if area of a floor in a hospital is 10 m by 10 m, then it takes 100 images.As an initial condition, a floor layout without any obstacles of pedestrians, medical doctors, nurses, patients, etc. is set-up.After acquiring background of template images, a floor layout map with obstacles is updated automatically every unit time.This map is setup manually based on room layout.Example is shown in Fig. 5.
Combination between background images and layout path will obtain main map.EWC will move consider the main map.After obstacle is detected in current path, EWC will switch to another path base on Dijkstra algorithm 7 with the same destination.Obstacle avoidance methods are also useful when user is not confident to pass through the obstacle.EWC will avoid the obstacle.Furthermore, the proposed EBEWC system also is able to update existing map.In this example, a floor is to be 4 by 6 m.Image is acquired at each lattice point location with 1m of intervals and with 10% of overlapping between the neighboring two areas.Then forward looking camera acquires same location of image.After that, the current image is compared to the previously acquired image results in obstacle finding.Namely, the different between the current and the previously acquired images shows location and shape of obstacles.EBEWC may switch back to the original key layout when user looks at the top left corner of key for much longer than 0.7 sec.

D. Obstacle Avoidance
For safety reason, obstacle avoidance system is implemented in the proposed EBEWC system.Obstacle avoidance system is capable to identify the obstacle in front of EWC and avoid it.This system consists of two approaches: (1) Obstacle detection, and (2) Shortest Path Finding.Obstacle detection is consisting of image processing based and ultrasonic sensor based.Image processing based utilizes background subtraction between current image I(x,y) and background image B(x,y).Background subtraction method will obtain black-white image S(x,y) which represent obstacle.On this image, obstacles appear as white pixel.By using searching of outer line from white pixels, we can determine position and size of obstacle.
Weakness of background subtraction method is working only if two images have same position of translation, scale, shear, and rotation.To solve these problems, we utilize Affine transformation 8 .This transformation requires three points that appears on both images.
Translation, scale, share, and rotation parameter can be determined from these points.Affine transformation required at the least three noticeable important points.These points should be appears on both images.It can be detected by several ways: corner, edge, specific object, and etc. Corner and edge have many points and its will create computation problem.In our system, we decide to use specific object.The specific object can be an easily recognized object, text character, chessboard wall, and etc.
The specific object will obtain one coordinate from center of area.Obstacle detection using Affine transformation requires identifying at least three kind of object, so it will obtain three noticeable important points.First step of obstacle detection is convert source image into gray image.By using template matching, system will find specific object position.Using six noticeable points, system calculate translation, share, scale, and www.ijacsa.thesai.orgrotation parameter.These parameters will used for creating Affine transformation matrix.
Affine transformation matrix is applied on background image will obtain Affine transformed image.Normalization is used to eliminate disturbance such shadow, noise, and etc. Next, subtract between output form current image and background image.Smooth filter is used to reduce noise which is caused by subtraction process.Last step is applying threshold on image and it will obtain black and white image.Obstacle will be signed as group of white pixel.After black white image is obtained, we should return center of white pixel area into current coordinate and coordinate of obstacle is founded.Because of so many type of specific object, we got best performance of specific object by using chessboard wall.The advantage of the chessboard (Fig. 6) is easily detected and robust on distance changes.We use three types of chessboard: 3 by 4, 3 by 5, and 3 by 6 as are shown in Fig. 7.The other obstacle detection is use ultrasonic sensor.This sensor has advantage when visual system does not work.In case EWC move into surrounding glass door, smoke condition, and minimum lighting will caused visual system of obstacle detection fail.Ultrasonic sensor consists of transmitter and receiver part.We use ultrasonic sensor with 40 kHz frequency resonance.

E. Shortest Path Findings
EWC will move consider the main map.After obstacle is detected in current path, EWC will switch to another path which have same destination.Obstacle avoidance methods are also useful when user is not confident to pass the obstacle.EWC will take over and pass the obstacle.Best path is chosen based on Dijkstra algorithm.Furthermore, system also able to renew existing map.

A. Obstacle Avoidance Performance
Experiment of obstacle avoidance is conducted by acquiring image in corridor with distance 1m per images.Location which image was acquired, is set with (x,y) coordinate.So, in every (x,y) location will have 1 background image.This image will be used as background reference.Obstacle is detected by subtract background image with current image.To eliminate problem which caused by different position between these images, Affine transformation is used to transform background image to Affine transformed image which have same position with current image.Applying Affine transformation will does work if only if three noticeable important points are appears on both images.These important points are represented by three types of chessboard (Fig. 6).If the chessboard is successful detected, then by using Affine transformation and subtraction between background image and current image, obstacle is founded.Experiments of chessboard detection is conducted by measuring maximum location which still detected by system.Chessboard is put in fix location.After that, EWC is move around the chessboard.Maximum location which detected by system is recorded.Data show that boundary shift will decrease when distance between camera and chessboard is increase.This experiment is equal to obstacle detection.After three types of chessboard are detected, Affine transformation will use these points (three chessboard center of areas) to subtract with current image and obstacle position will be found.
Objective of this experiment is measure accuracy of ultrasonic sensor before used in the system.This experiment is conducted by measure sensor output for measuring distance and comparing with real data.Some object is put on front of sensor with varies distances and measure it.The experiment is conducted on distance 0 cm until 3 m.Ultrasonic sensor use PING type parallax product 9 and microcontroller AT89S51 as processing data (convert from time value to distance output).PING type parallax product and microcontroller AT89S51 as processing data (convert from time value to distance output).Elevation angle is require to know how width of the beam sensor.Ultrasonic sensor with width beam is not benefit to our system.Narrow beam will obtain good result because it will not influence with any disturbance.This experiment is conducted by measure elevation angle from 0 cm until 3 m.

B. EWC Control Performance
This experiment is conducted for measure EWC performance on starting up acceleration, forward and backward breaking deceleration.Also conducted speed measurement when EWC move forward, backward, turn left, and turn right.EWC is drive by user who has weight is 73 kg.We record the duration time and convert it into speed value.Experiment data of speed measurement is shown in Table 1.Graph of EWC acceleration and deceleration when start and stop duration is shown in Fig. 8.

C. Processing Time for Each Process
In order to apply whole method into EWC application, processing time should be measured to identify performance of our real time system.Fig. 9 shows transient time of eye detection and tracking, n the beginning of chart, it seem this method take long time around 300 ms.In this time, system still process face detection, eye detection and others process before running template matching method.www.ijacsa.thesai.org it seem this method take long time around 300 ms.In this time, system still process face detection, eye detection and others process before running template matching method.After eye location is founded, then system bypass previous step and cause process working fast.
After eye location is founded, then system bypass previous step and cause process working fast.Meanwhile, Fig. 10 shows processing time of eye detection and tracking on steady state condition, it looks faster than transient condition.This experiment also was conducted using Optiplex 755 Dell computer with Intel Core 2 Quad CPU 2.66 GHz and 2G of RAM.NET COWBOY DC-NCR131 camera as visual input is also used.Experimental result show average processing time is 32.625 ms.Fig. 12 shows processing time of obstacle detection.
We implemented ultrasonic sensor parallax PING type.This sensor is controlled by using custom microcontroller AT89S51.Data was stored into computer by using USB communication.Result data show average processing time is 568.658ms.Fig. 13 also shows processing time of ultrasonic sensor, it look take longer time than others.
Figure 13 processing time of ultrasonic sensor, it look take longer time than others.

IV. CONCLUSIONS
It is concluded that the proposed key-in system with human eyes only works 100% perfectly for the number of keys is four, start, stop, turn right and left.Also it is concluded that the proposed EWC control system does work in a real basis avoiding obstacles on range bellow 3.4 m using image processing method and bellow 3 m using ultrasonic sensor.By the proposed system, EWC is able to identify obstacle and avoid them.Obstacle avoidance can assist user into interest place although undesired condition such as dark areas, glass wall or door, smoke area, and etc.By implemented this system, it is found that EWC can move more safely and comfortable.
We may conclude the followings, 1) EWC will stop after 70cm when load below than 73 Kg 2) EWC will stops after 30 cm when 85 Kg 3) Increasing load will make EWC stop shorter 4) Proposed system is almost perfect for detection of functional keys which selected by human eye only.

5) System is capable to detect the obstacle on range bellow 3.4 m.
Also it is confirmed that all the required processing can be done in a real time basis.

Figure 1 Figure 2 .
Figure 1 Hardware configuration of the proposed EBEWC system Switch Stop Stop Turn left Forward Turn right Stop Stop StopFigure 2. 3 by 3 of key layout for EWC control

Figure 6 Figure 7
Figure 6 Chessboard as specific point

Figure 8 Figure 9
Figure 9 Transient time of eye detection and tracking, n the beginning of chart,it seem this method take long time around 300 ms.In this time, system still process face detection, eye detection and others process before running template matching method.After eye location is founded, then system bypass previous step and cause process working fast.

Figure 10
Figure 10 Processing time of eye detection and tracking on steady state condition, it looks faster than transient condition.

Figure 11
Figure 11 Processing time of Eye gaze method

Figure 12
Figure 12 Processing time of obstacle detection This experiment is conducted using Optiplex 755 Dell computer with Intel Core 2 Quad CPU 2.66 GHz and 2G of RAM.We use NET COWBOY DC-NCR131 camera as visual input.Experimental result show average steady state processing time is 32.625 ms. it also shows difference processing time between transient and steady state condition.Transient time require more time than steady state time.Objective of this experiment is measure processing time on Eye gaze identification.It is conducted by using ACER computer ASPIRE 5572 Series Core Duo T2050 1.6 GHz CPU and 1G of RAM.Result data show average processing time of this method is 342.379ms.Fig.11 shows processing time of Eye gaze method.