SRAVIP: Smart Robot Assistant for Visually Impaired Persons

Vision is one of the most important human senses, as visually impaired people encounter various difficulties due to their inability to move safely in different environments. This research aimed to facilitate integrating such persons into society by proposing a robotic solution (Robot Assistance) to assist them in navigating within indoor environments, such as schools, universities, hospitals, airports, etc. according to a prescheduled task. The proposed system is called the smart robot assistant for visually impaired persons (SRAVIP). It includes two subsystems: 1) an initialization system aimed to initialize the robot, create an environment map, and register a visual impaired person as a target object; 2) a real-time operation system implemented to navigate the mobile robot and communicate with the target object using a speech-processing engine and an optical character recognition (OCR) module. An important contribution of the proposed SRAVIP is the user-independent, i.e. it does not depend on the user, and one robot can serve unlimited users. SRAVIP utilized Turtlebot3 robot to realize SRAVIP and then tested it in the College of Computer and Information Sciences, King Saud University, AlMuzahmiyah Campus. The experimental results confirmed that the proposed system could function successfully. Keywords—Mobile robot; robotics; robot assistance; and visually impaired persons


I. INTRODUCTION
At present, a large number of persons have various physical impairments. The World Health Organization (WHO) has evaluated that according to examination in 2019 approximately 2.2 billion people worldwide live with a form of vision impairment, among which 36 million people are blind [1]. Visual impairment is a noteworthy issue that prevents from moving freely. Such people usually require the assistance of unimpaired individuals to identify and follow a target direction or to provide guidance while passing stairs and turns. Most visually impaired individuals cannot determine their path freely in an unfamiliar zone.
In general, there are many tools and ways to help the blind people: 1) Traditional ways, e.g. Help by other people, White Stick, and Guide dogs; 2) Technical-based such as Wearable device and Navigation Robot.
The recently introduced technologies, such as Blind Stick, have facilitated them to move freely; however, the implemented procedures are deemed insufficiently reasonable and practical. At present, the rapid technical development has enabled introducing many types of robots, such as ones intended for the commercial use and delivery. In addition, there are several applications in hospitals and hotels, for example, a DoorDash robot employed to deliver food to customers via an application [2]. The other type of robots includes those that are similar to humans, such as Sofia designed by the company Hanson Robotics 1 . These robots have been developed to learn and adapt to human behavior. Other types of existing robots inhibit various features: they can behave independently, identify their way to specific locations in the buildings, and so on.
This study aimed to assist the blind people to reach their target (classroom in schools/universities, and gates in airports) using a robot-based solution. Therefore, a solution that utilizes robotics to assist visually impaired individuals is developed. Robot-based solutions can be applied to assist visually impaired and blind people to walk and move in various environments, such as schools, universities, hospitals, airports, shopping malls, etc. For example, when a visually impaired person enters an airport, a mobile robot can be utilized to enable navigation to a required gate according to the boarding ticket information. As the other example, we can consider the situation in which a robot can assist a student in navigating to a target classroom according to the timetable. The concept of robotic assistance should be grounded on understanding the needs of visually impaired individuals. In the present study, we focus on a task of navigating a visually impaired person to a predefined location within a particular indoor environment.
In the future, we are planning to enhance the proposed SRAVIP system by adding more features such as fingerprint or face recognition, supporting Arabic languages, and enabling remote maintenance.
The remainder of this paper is organized as follows. In Section 2, the related literature review is presented. The proposed system structure is outlined in Section 3. In Section 4, we describe the experimental setup and discuss the obtained results. The discussion is provided in Section 5. And the conclusion provided in section6.

II. LITERATURE REVIEW
Several systems and techniques have been proposed to assist visually impaired persons by means of a wearable device, a smart cane, or a robotic assistant.
One more system that employs a similar vest differently has been proposed in [4] that is implemented as a stereo vision system for visually impaired systems (Fig. 2). Using a visual odometer and feature-based metric-topological and simultaneous localization and mapping (SLAM), the system generates a 3D map representation corresponding to the vicinity of a user that can be utilized for obstacle detection and traversability map development.  In [16], the researchers have constructed a model aimed to control movements, perceive protests, and maintain a distance from obstacles in an indoor environment for visually impaired individuals. The framework comprises a module with a headset, a camera, a marker, an inertial estimation unit, and laser sensors mounted on a user's chest (Fig. 3). Communication between the system and users is established through speech recognition and synthesis modules. The laser sensors registered the data required to maintain a safe distance. The camera is utilized to capture scenes and send it to the system for navigation or recognition.
Bai and Wang [5] have developed a low-cost wearable device aimed to assist visually impaired persons in navigating and avoiding obstacles based on the dynamic subgoal strategy. The device is composed of an ultrasonic rangefinder, a depth camera, a fisheye, an earphone, a pair of see-through glasses, and an embedded CPU board. The visual SLAM is utilized to define a virtual trajectory based on the depth and RBG images. The device identifies a short path by applying the A* algorithm to a point-of-interest. However, the problem associated with this device is that ultrasonic sensor data may vary depending on weather, which may lead to misdirection (Fig. 4).  (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 7, 2021 347 | P a g e www.ijacsa.thesai.org

B. Smart Cane
Visually impaired individuals may encounter numerous dangers that may threaten their security, as there are multiple regular things in their surroundings that may act as obstructions. Therefore, an inside framework presented in [7] is considered as a powerful arrangement between a several segments used at the stage of capturing and handling the data that are then utilized to define the path and transmit route messages to a user while moving in an inner zone. This framework contains key apparatuses including a white stick equipped with different lights, two infrared cameras, a portable computer with a product application to manage the whole framework, and a cell phone to convey the route data to a user through voice messages (Fig. 5).
Navigating indoor environments for visually impaired persons is usually difficult; therefore, in [8], the authors have proposed a system that can assist blind people in identifying the best path to arrive to a target destination. This system implies embedding RFID tags within paths and capturing them by a RFID reader using a cane antenna (Fig. 6).
In [9], a system called AKSHI has been proposed to assist visually impaired people in walking. The system has been implemented using Raspberry Pi, a GSM module, ultrasonic sensor, RFID tags, and a reader. The RFID reader is attached to the bottom of the cane. In the middle of the cane, an ultrasonic sensor has been mounted; the box containing Raspberry Pi 2, GPS, and GSM modules has been mounted above the sensor. Moreover, a mobile application to track the user location has been developed. However, the problem associated with this system is that the utilized sensor cannot function accurately in dirty or muddy environments.
Muñoz and other authors [10] have proposed a system using Tango devices within a mobile computing platform and have implemented a method for obstacle detection based on TSM-KF using RGB-D camera. The automated map generation process has been established by developing an indoor map editor to parse a CAD model of an architectural floor map, as well as implementing a smart cane as a humanmachine interface and communication tool.

C. Robotics
The authors in [11] have proposed a suitcase-shaped navigation robot called Cabot to assist visually impaired persons in navigating to a target destination and avoiding obstacles. It comprises a stereo camera, a LIDAR sensor, a tactile handle, a portable computer, batteries, a motorized wheel, and a clutch. It can be used to receive feedback from a handle and to provide the information about the required direction. However, it does not consider dynamic elements. To enable a robot to move and recognize a target environment, many techniques and algorithms have been used [17][18][19][20][21][22][23]. A specific framework has been proposed in [12]. Here, the RGB and depth images are employed to achieve the purpose. The scheme represented in Fig. 7 can be used to understand the general structure of this proposed framework, which can be divided into two sequential stages: the offline work stage including the establishment of the structure and the electronic stage in which the model is applied to identify a target item according to the predefined map.   Jalobeanu [13] has developed a framework based on a Kinect sensor to map indoor environments. This framework combines a grid map based on the truncated signed distance function with a particle filter used in a real-time mode. This www.ijacsa.thesai.org framework exploits the noisy depth data and can be used to increase the field of view by establishing virtual views from local maps. In [14], the researchers have combined the LIDAR and sonar sensors. Such multi-sensor combination techniques have been applied to identify objects while performing SLAM. In the case when various sensors are used to analyze a similar obstacle, they can confirm the obtained information and improve the estimation precision of each other. Kumar [15] has proposed a robotic navigation system including two controllers. The first one is intended to compute the path from the current position to a target point without considering potential obstacles. The other controller is based on fuzzy logic and is utilized to deal with obstacles by processing the input from a laser sensor and then sending the output to the robot aiming to avoid the identified obstacle.

III. PROPOSED SYSTEM
In this study, we developed a system called SRAVIP aimed at integrating visually impaired persons into society. To do this, we attempted to construct a mobile robotic solution intended to assist such persons in walking and navigating in an indoor environment. SRAVIP is aimed at assisting the blind people to reach the target location (such as classrooms in schools/universities or gates in airports) using a robotic solution, as presented in Fig. 8. The proposed SRAVIP has the following structure: b) User Interface: communicating with a user (through a speech-processing engine or an OCR system).
An important contribution of the proposed SRAVIP is the user-independent, i.e. it does not depend on the user, one robot can serve unlimited users. In this section, we provide an overview of the proposed system and discuss the interaction between its components, explaining each component in detail.

A. System Initialization
This phase is implemented to initialize the system. It includes two steps: constructing the map of a target environment and registering users in the system.

 Map Construction
First, we execute system/robot initialization by determining where a mobile robot is located with respect to the surrounding environment (using simultaneous localization and mapping). Enabling the robot to move from the current location to the destination one according to the predefined map, we utilize an encoder, an IMU sensor, and a distance sensor to avoid obstacles. We will employ a 2D occupancy grid map (OGM) to construct the map based on the distance information obtained using the sensors and the pose information of the robot itself (Fig. 9). Finally, the system saves the map.

 User Registration
In this step, the robot operator registers the information about a user manually in the database of the target organization (school, university, airport, etc.). When the registration is completed, the robot can identify a user when he or she requires assistance in reaching the target destination by using ID scan or speech commands.

B. Real Time Operation
This phase is online and is intended to perform robot navigation, path planning, and user interaction (Fig. 10).
After a user interacts with the robot through telling the ID number or presenting the ID card, the robot can identify the user and retrieve the information about this user from the database (a student timetable or a flight information). Then, it identifies the target location and specifies the current one according to the predefined map of the building. The target and current locations and the map serve as the inputs into the pathplanning algorithm. The robot defines a path aiming to avoid obstacles and to identify the shortest route. Then, the robot initiates moving along the defined path. At this time, even if an unexpected obstacle is suddenly detected, the robot will be able www.ijacsa.thesai.org to avoid it and to proceed moving to the target destination. During the movement, the current location of the robot is identified with respect to its environment using simultaneous localization and mapping. If the robot reaches the destination, it will go back to its initial location; otherwise, it continues executing the localization steps to proceed with the operation. When the robot reaches the initial location, it stops; otherwise, it continues executing the robot localization step.

IV. SRAVIP IMPLEMENTATION
In this section, we describe the implementation of the proposed SRAVIP, which consists of two parts: the first part is the initialization of the system (offline), and the second part is real-time operation (online). In SRAVIP, we have utilized Turtlebot3 and then, we have tested the proposed system within the College of Computer and Information Sciences, King Saud University, AlMuzahmiyah Campus.
Turtlebot3 has the following features: Raspberry Pi 3 Model B and B+; a laser distance sensor (LDS) is a 360 LDS-01. We have modified it mounting a camera to facilitate recognizing a university card and a microphone for speech recognition. We have utilized a robot operation system (ROS) as a robotic platform. ROS is an open-source robotics software platform. In addition, we have employed Gazebo and RViz simulations.

A. Offile Implementation System Initialization Phase 1) Map construction:
We construct the map using the gmapping package, implementing the laser-based simultaneous localization and mapping (SLAM) technique to create a 2D occupancy grid map (Fig. 11). 2) User registration: In the user registration step, we employ MYSQL as relation database management system to register users in the system. The entity relationship (ER) diagram is illustrated in Fig. 12.

B. Real Time Operation (Online)
In this part, we implement the interaction between a user and the robot either by scanning a university card through the OCR system or using the speech recognition system to identify the university ID number. The real time operation modules include two main steps: user interaction and robot navigation/path planning. www.ijacsa.thesai.org 1) User interface implementation: In this section, we explain the details of how a user interacts with the robot through the developed user interface (UI). To enable this option, we have implemented the code in Python. In UI (Fig. 13), a user has two options: to provide a user ID card so as to identify the ID number by means of the optical character recognition (OCR) system from the database, or to say the ID number so that the robot could detect it through the speech recognition module. If the user is registered in the system correctly, the id number appears on the screen.
We implement UI using the tkinter package, the standard Python UI available in the tk graphical user interface toolkit. 2) OCR implementation: Optical character recognition (OCR) is a technology that can be used to extract the ID number based on a student card. To do this, we have implemented several stages, as illustrated in Fig. 14, using various libraries. The first library is NUMBY that allows generating arrays and data-followed processes. Then, we have employed TensorFlow library using a neural network trained on 500 images. Lastly, we utilized the OpenCV library intended for executing clustering operations.

3) Speech recognition implementation:
Speech recognition includes several stages described in Fig. 15, the most important step is to represent the relationship between the language units of speech and voice signals. The other module is used to match linguistic modeling sounds with number sequences to allow distinguishing between numbers that look similar. Using the speech recognition module, the system allows recognizing the ID number pronounced by a user. If the user is registered in the system, the ID number is processed using the database and is represented on the system screen. According to the ID number, the user data and schedule are extracted from the database, and the robot navigates the user to the target location.

4) Robot navigation and path planning:
Using the constructed map and after the student loaded from the database, SRAVIP can define the class coordinates and automatically save them in a text file. Thereafter, the system can process them to set up a coordinate target for navigation, and the robot automatically executes path planning to move from the initial location to the requested one. After the robot reaches the target location, it returns to the initial point.

V. SRAVIP EXPERIMENT AND TESTING
We tested the SRAVIP system within the simulation environment established in College of Computer and Information Sciences, King Saud University, AlMuzahmiyah Campus. In the experiment, we considered the two possible scenarios of the interaction between the proposed system and users. In the first scenario, experiment participants utilized the OCR system, and in the second scenario, the speech recognition system was employed. The experiment procedure was specified as follows. When the participant entered the college building from the main gate, the robot could be found on the right side of the door, and this location was fixed for a robot as "Home Adder." The first scenario implied utilizing the OCR system (Fig. 16). In the beginning, participant A had to demonstrate the university card (Fig. 17) to the robot so as to extract the student ID number. Recognition. www.ijacsa.thesai.org Fig. 17. Extraction of a University ID Number using the OCR System. Thereafter, the system was able to gain the information about the student schedule and define the coordinates of a target class for this user. Then, the coordinates were processed by the robot navigation system (Fig. 18). Fig. 18(a) shows that the robot at the Home Adders location and waiting for the path to the target class ( Fig. 18(b)) to be generated by the robot navigation system. Then, the student goes to his class with assist from the robot (Fig. 18(c)), finally, the robot automatically returns to home address to be able to serve other students. a) Robot at the "Home Adders" Location. b) Generated Path to the Class 98. c) Navigation and Path Planning to the Class 98. The second scenario considered employing the speech recognition system (Fig. 16). In this case, participant B told the university number to the robot for extraction and identification of the user information from the database using the speech recognition technology. After that, the system was able to load the information about the student schedule and coordinates of a target class for this user. Then, the coordinates were employed by the robot navigation system to construct a trajectory (Fig. 19). Fig. 19(a) shows that the robot at the Home Adders location and waiting for the path to the target class ( Fig. 19(b)) to be generated by the robot navigation system. Then, the student goes to his class with assist from the robot (Fig. 19(c)), finally, the robot automatically returns to home address to be able to serve other students. a) Robot at the "Home Adders" Location.
b) Generated Path to the Class 98.
c) Navigation and Path Planning to the Class 97. Fig. 19. Robot Navigation System used in the Second Scenario.

VI. DISCUSSION
As we have seen in the literature review, several systems and techniques have been proposed to assist visually impaired persons by means of a wearable device, a smart cane, or a robotic assistant. All those solutions are user-dependent solution, and work as one to one solution. It means that one device only can be used by one blind person. An important contribution of the proposed SRAVIP is the user-independent property, i.e. it does not depend on the user. It is one to many solutions, i.e. one robot can serve unlimited number of users. www.ijacsa.thesai.org

VII. CONCLUSION
In the present paper, we proposed a robotic solution to assist visually impaired people to walk and navigate within various indoor environments, such as schools, universities, hospitals, airports, etc. according to a prescheduling task. An important contribution of the proposed system is the userindependent property, i.e. it does not depend on the user, it is one to many solutions, i.e. one robot can serve unlimited number of users. While the previous solutions work as one to one solutions. It means that one device only used by one blind person.
The proposed system called a smart robot assistant for visually impaired persons (SRAVIP) was composed of two subsystems: the initialization module and the real-time operation one. We employed Turtlebot3 to implement the proposed system. Then, we tested it in many scenarios within the College of Computer and Information Sciences, King Saud University, AlMuzahmiyah Campus. The experimental result indicated that the proposed system could be utilized successfully. In the future, we are planning to enhance the proposed SRAVIP system by enabling remote maintenance, adding more features such as fingerprint or face recognition, speech recognition, and supporting different languages such as Arabic.