Interventional Teleoperation Protocol that Considers Stair Climbing or Descending of Crawler Robots in Low Bit-rate Communication

—In teleoperation of a crawler robot in a disaster-stricken enclosed space, distress of the crawler robot due to communication breakdown is a problem. We present a robot teleoperation system using LoRaWAN as a subcommunication infrastructure to solve this problem. In this system, the crawler robot is operated by teleoperation using a subcommunication infrastructure in a place where a wireless local area network (LAN) communication is possible. In this study, we assume an environment in which the crawler robot must ascend and descend stairs to evacuate to a place where wireless LAN communication is possible. In addition, the disaster-stricken environment is considered as an environment where obstacles are expected to suddenly occur, and the crawler robot has difficulty avoiding obstacles on the stairs. In this paper, we propose a teleoperation communication protocol that considers the risk of sudden appearance of obstacles and confirm its effectiveness using evaluation experiments in a real environment.


I. INTRODUCTION
When a large-scale disaster occurs, information gathering is required to assess the damage in the affected area [1]. The methods of information gathering in affected areas include aerial photography using drones, monitoring of existing infrastructures, and onsite activities by rescue workers [2][3][4]. However, collecting information from the sky in an enclosed space is difficult. Existing infrastructure may become inaccessible due to malfunction or lack of power supply. When rescue workers search a disaster-stricken enclosed space, human casualties due to collapsed buildings and debris may be expected. Therefore, information gathering using mobile robots has been considered in enclosed spaces after disasters [5][6][7].
Two types of communication methods that use mobile robots are available: wired and wireless. The wired system offers the advantages of stable power supply to the mobile robot via cable and maintains communication quality between the operator and mobile robot [8]. However, teleoperation of the mobile robot using a wired system suffers from problems such as cable tangling and snagging as well as cable handling at the time of return. On the other hand, the wireless method does not reduce the operability due to cables, and mobile robots can maintain high operability. However, as the mobile robot moves, the distance from the base station increases, and the electric-field-strength decreases. As a result, the mobile robot faces the risk of going into locations where radio waves cannot reach and communication is interrupted. When teleoperating a mobile robot, considering the merits and demerits of wired and wireless communication methods and deciding on a communication method that suits a disasterstricken environment are important [9][10]. In the present study, we present a wireless teleoperation method for a mobile robot in environments where the mobile robot has difficulty exploring using wired communication.
Related research on information gathering using mobile robots in enclosed spaces includes the robot wireless sensor networks (RWSNs) (Fig. 1) [11][12]. RWSN is a method of extending the communication range between the operator and mobile robot by installing a sensor node (SN), which consists of wireless repeaters, on the mobile robot and deploying them along its movement path. Whereas teleoperation using RWSN enables a wider range of information gathering, the mobile robot may be isolated due to communication breakdown caused by SN failure, battery failure, and radio interference. Distress of the mobile robot can not only interrupt information gathering but also cause secondary disasters because a mobile robot in distress becomes an obstacle or may ignite from its onboard battery. Therefore, in the present study, we present a teleoperation method using a subcommunication infrastructure as a distress-prevention method when the main communication infrastructure [wireless local area network (LAN)] is disconnected. In the disaster-stricken enclosed space assumed in this study, sudden appearance of obstacles such as collapsed debris is expected. Therefore, when an obstacle is found during the evacuation to the place where wireless LAN communication is available, the mobile robot is required to bypass or climb over the obstacles. The slope of the staircase changes when an obstacle appears in the stair environment. Therefore, if the mobile robot runs on the stairs, it faces the risk of falling down due to insufficient hill-climbing ability. In addition, if the mobile robot climbs over an unstable obstacle, a risk exists in which the robot may fall over or the obstacle collapses. Therefore, shortening the teleoperation time and completing the stair climbing-process before sudden obstacles appear during the stair climbing are necessary by teleoperation using LoRaWAN.
In this paper, we propose an interventional teleoperation protocol. The proposed protocol eliminates the factors that increase the teleoperation time in existing teleoperation protocols. In the experiment, we performed teleoperation using the proposed and existing methods in a real environment and confirmed that the proposed method could reduce the teleoperation time and demonstrated its effectiveness.

II. LORAWAN OVERVIEW
LoRaWAN is one of the communication standards categorized as low-power wide-area standard, which is a generic term for wireless-communication technologies that save power and can communicate over long distances. LoRaWAN is designed for Internet of Things and is suitable for acquiring information from a large number of geographically dispersed sensors [13]. In the agricultural field, it is used to manage sensors that measure cultivationenvironment information such as temperature, humidity, and soil moisture and to transmit the sensor information, which contributes to smart agriculture [14]. Smart meters are expected to enhance power resilience and optimization of power transmission and distribution networks, and LoRaWAN is being considered as a network-connection method for these smart meters [15]. An early-detection system for forest fires using LoRaWAN sensor networks has been proposed, which can contribute to the mitigation of forest-fire damages [16]. Thus, LoRaWAN is being implemented or considered for deployment in many fields.
In the field of teleoperation of mobile robots in enclosed spaces, videos are mainly used to investigate the environment. However, LoRaWAN, which has a maximum bit rate of 50 kbps, suffers from the difficulty of transmitting videos. Hence, few discussions are available on the use of LoRaWAN in the field of teleoperation of mobile robots. On the other hand, LoRaWAN is capable of long-distance communication (2-5 km in urban areas and 15 km in suburban areas) [17]. Therefore, LoRaWAN offers the potential for continuous communication even in an environment where wireless LAN (IEEE802.11.g), which has a communication distance of approximately 100 m, is disconnected. Moreover, because LoRaWAN has a frequency bandwidth of 920 MHz, it has high diffusivity and the potential to continue communication even in an enclosed space with many obstacles. Therefore, LoRaWAN is considered to be a suitable wireless-communication standard for subcommunication infrastructures. Thus, in this study, we adopt LoRaWAN as subcommunication infrastructure and explain the teleoperation method that considers the features of LoRaWAN.

A. Existing Teleoperation Protocol using LoRaWAN
In the teleoperation using wireless LAN, the mobile-robot environment is mainly monitored via videos, and the mobile robot is operated using a controller. Because the video is updated at any time, the operator always knows the difference between the current position of the mobile robot and target position and operates the mobile robot to the target position. However, the low bit rate of LoRaWAN makes employing such method of simultaneously learning the environment and moving the mobile robot in teleoperation using LoRaWAN www.ijacsa.thesai.org difficult. Therefore, teleoperation using LoRaWAN requires a separate operation for learning the environment and moving.
In the LoRaWAN communication, a communication path is constructed by designating two communication terminals as a coordinator and an end device. Packets are transmitted from the end device to the coordinator. In the existing teleoperation protocol, the mobile robot transmits information or performs actions in response to commands sent by the operator by considering this feature into account (Fig. 3) [18]. The mobile robot can be operated using such protocol after acquiring the necessary information for teleoperation even in low-bit-rate LoRaWAN. In the next section, we describe the environment learning and operation methods for teleoperation using LoRaWAN.

B. Related Work
When teleoperating a mobile robot, learning the surrounding is necessary to confirm the obstacles around the mobile robot and determine its movement path. Therefore, accurately learning the environment in teleoperation using LoRaWAN is necessary. In LoRaWAN, a method for learning the environment using images has been proposed [19]. In this method, the obtained image is transmitted in segments, and the receiver reconstructs the received image elements to obtain the original image.
A teleoperation method using a graphical user interface (GUI) has been proposed to move a mobile robot to a target position ( Fig. 4) [20]. In this method, the environment is learned based on the obstacle information obtained by LRF, in addition to the images. The black squares at the interface indicate the presence of obstacles, whereas the ochre squares indicate the absence of obstacles and possibility of movement. Then, when the mobile robot is operated using GUI, its movement path is determined by sequentially selecting the adjacent ochre-colored cells from the cells where the mobile robot is located. However, this method assumes movement on a level ground with obstacles and cannot handle movement in a 3D space such as a staircase. Therefore, if a distressed mobile robot is required to ascend or descend stairs to evacuate to a place where wireless LAN communication is possible, it cannot achieve its purpose. Hence, we need to consider the process of ascending and descending stairs by the mobile robot using LoRaWAN teleoperation. Section C describes the mobile robot used for stair climbing and descending, and Section D describes the teleoperation method in the considered stair environment.

C. Mobile Robots Used in Staircase Environment
Mobile robots with a flipper arm, which is a crawler mechanism (crawler robots), show higher operating performance on uneven terrain such as stairs than wheeled mobile robots. Therefore, in dealing with the Fukushima Daiichi Nuclear Power Plant accident, crawler robots such as the PackBot, Quince, and Survey Runner, which were introduced to collect information in the buildings, were required to climb and descend stairs [21][22]. On the basis of this information, the present study investigates the process of ascending and descending stairs using crawler robots.

D. Command-Input Operation Method
The crawler robot moves and turns by rotating the main crawler and adjusting its posture using its flipper arm. Therefore, in order that the crawler robot can be teleoperated, the operator must be able to send an arbitrary operation command to the main crawler and flipper arm. In addition, for the LoRaWAN teleoperation, moving the crawler robot to the target position based on the acquired information is necessary. The low bit rate of LoRaWAN makes video transmission difficult. Therefore, the current position of the crawler robot as it moves cannot be known in real time. In such an environment, the operator does not know how far the crawler robot has moved if the controller is used in the same manner as that in operation using wireless LAN. Therefore, teleoperating the crawler robot to its target position using a controller is difficult.
Hence, we propose a command-input operation method as a teleoperation method for crawler robots in a staircase environment. In this method, the operator and crawler robot share a correspondence table of actions, as shown in Fig. 5, and the operator operates the crawler robot by transmitting the numbers in the table. The amount of performed selected operations is determined by specifying the amount of movements such as distance or angle. Thus, the operator can teleoperate the crawler robot to the target position and posture using this method even in an environment where the operator does not learn the change in the current position of the crawler robot in real time. In this study, we select the information required by the operator as follows, i.e., factors (1)-(5), to use the command-input operation method.

1) Multiple camera images
2) IMU sensor information 3) Staircase information (slope, kick-up width, and tread) 4) Distance information between the crawler robot and stairs 5) Obstacles and staircase-shape information Item (1) is needed to confirm a safe place where the crawler robot can move forward by learning the stair environment such as the presence or absence of obstacles on the staircase and missing staircase parts. Item (2) is necessary to obtain the posture of the crawler robot in the staircase environment and determine the degree of movement to the target posture. Items (3) and (4) are necessary for determining the amount of movement to move the crawler robot to the target position. Item (5) denotes a factor that determines whether the staircase contains obstacles or part of the staircase is missing. If so, it determines whether these obstacles affect the crawler-robot operation.

A. Problem of Teleoperation using LoRaWAN in a Staircase Environment
In a disaster-stricken enclosed space, obstacles exist, which are assumed to block the path. If an obstacle exists in the crawler-robot direction of movement during the evacuation action, it needs to run through the obstacle by climbing over it or by avoiding it using an obstacle-free path.
If obstacles are present in the stair environment, the slope of the stair changes. If the crawler robot tries to avoid these obstacles by climbing over it, it may fall down due to its insufficient climbing ability and very steep gradient when descending (Fig. 6). In addition, if the crawler robot rides on an unstable obstacle, a risk exists in that the crawler robot may fall over because of collapse of the obstacle. Moreover, if the crawler robot tries to avoid the staircase by diverting to an unobstructed area, it may lose its balance and fall down when it turns on the staircase. Thus, a risk exists in which the crawler robot may fail to ascend or descend stairs if obstacles are present in the stair environment.
The disaster-stricken environment assumed in this study is subject to rapid environmental changes and sudden occurrence of obstacles. In addition, the transmission of various sensor information required to send a single operation command takes several minutes. These factors indicate that a risk of failure in the evacuation actions exist because of the occurrence of sudden obstacles in the staircase environment during climbing and descending stairs by teleoperating using LoRaWAN. Therefore, for successful evacuation in staircases with risk of sudden occurrence of obstacles, teleoperation using LoRaWAN is required to shorten the time needed for climbing or descending stairs and completing the stair-climbing process before obstacles appear.
If the amount of transmitted information is limited, e.g., by limiting the number of transmitted images, the teleoperation time can be shortened, but a risk exist in not being able to accurately learn the environment. Therefore, when the teleoperation time is reduced, we must not reduce the information-transmission time during teleoperation but increase the ratio of the time when information is transmitted by reducing the time when no information is transmitted.
As presented in Section III, in the existing teleoperation protocol that uses LoRaWAN, the crawler robot transmits information or performs action in response to a command transmitted by the operator. Therefore, the operator that teleoperates the crawler robot repeatedly decides the command, inputs the command, and receives data. In the existing protocol, no communication occurs when the operator decides which command to send next and when the command is input (bandwidth is not used). Under the same transmission-capacity environment, the amount of data transmission depends on the bandwidth utilization. Therefore, the time required to obtain the information necessary for operation increases with decreasing bandwidth utilization and decreases with increasing bandwidth utilization. On the basis of this discussion, the proposed protocol needs to reduce the teleoperation time by improving the bandwidth utilization.
In the existing protocol, when the operator transmits an incorrect command, the operator cannot interfere and must wait until the crawler robot finishes executing the issued command. Therefore, the teleoperation time is increased by the execution of the wrong command. Hence, the proposed protocol must be able to deal with wrong command execution.
In this paper, we propose a teleoperation protocol that reduces the teleoperation time by taking into account what has been described in this section.

B. Interventional Teleoperation Protocol
In this paper, we propose an interventional teleoperation protocol as a teleoperation protocol that satisfies the required specifications. The proposed protocol reduces the command input time and improves bandwidth utilization by sequentially transmitting information necessary for operation according to a predetermined order, instead of transmitting information according to the command received from the operator. However, in this communication flow, even if sufficient information is collected to determine the operation command, transmission is not interrupted, and we need to wait until all the information transmission is completed. Therefore, this communication flow may increase the bandwidth utilization but may not reduce the teleoperation time. Hence, in the proposed protocol, information transmission can be interrupted at arbitrary times using an interventional command, and operation commands can be executed. In addition, the operator can also interrupt the execution of a wrong command using an interventional command.
In the interventional teleoperation protocol, the system must be able to transmit packets from the operator to the crawler robot while it transmits packets to the operator. Therefore, when this method is employed, we assume that two communication paths exist: "operator→crawler robot" and "crawler robot→operator." The proposed interventional teleoperation protocol is shown in Fig. 7, and its flow and processing are described hereunder.
1) The operator transmits a synchronized flag (SYN) packet to the crawler robot.
2) The crawler robot sends back an acknowledgement flag (ACK) packet as an acknowledgement of the SYN packet to the address from which the SYN packet was sent. The crawler robot also transmits SYN packets to verify that packets from the crawler robot reach the operator.
3) The operator confirms the communication connection by receiving an ACK packet and returns the ACK packet in response to a SYN packet sent by the crawler robot.
4) The crawler robot sequentially transmits the sensor information acquired by it after receiving ACK in (3). The order of transmission is concrete numerical information (shape information of obstacles and stairs, IMU sensor information, and distance information between the crawler robot and stairs) and images. In the process of ascending or descending stairs by the crawler robot, we assume that multiple images such as subjective and bird's-eye-view images are used, and the order in which these multiple images are transmitted is set in advance. 5) When a task with a higher priority occurs during the image transmission, an interventional command is sent, the current task is interrupted, and the contents of the command are executed. For example, during image transmission, changing the transmission order of predefined images may be necessary, some images may be lost due to packet loss, or learning of the environment may be completed without waiting for all images to be acquired. Then, when the transmission order of the images is changed, the order is changed by transmitting a new image-transmission order as an interventional command. If the priority of retransmission of a packet-lost image is high, the interventional command to execute the retransmission process is transmitted, and the current image transmission is interrupted to execute the command contents. When information gathering is completed, the operation command is transmitted as an interventional command to operate the crawler robot. When the crawler robot has completed executing the interventional commands other than the operation command, it resumes the task it performed before the intervention command was sent.
When a wrong command is executed, its execution can be interrupted by transmitting the interventional command. Thus, the interventional command can be used to reduce the time used in the bandwidth and to deal with the execution of wrong command contents.
6) When the crawler robot receives an operation command as an interventional command in (5), it interrupts the image transmission and executes the command. When the crawler robot moves, highly real-time information such as IMU sensor or encoder information is always transmitted to the operator.
7) The operator checks whether the crawler robot has moved more than the specified distance based on the information obtained in (6) and transmits an interventional command to stop the crawler robot if necessary.
www.ijacsa.thesai.org The packet format used in the proposed protocol is shown in Fig. 8. The interventional command that is part of the packet used in (5) contains commands to "change the order of image transmission," "retransmission processing," "image-quality conversion," and "cancellation of the last sent command".
The packet format used in (4)-(6) contains a packet number at the end. This packet number is a number expressed in ascending order starting from zero, and the operator can understand the order of packets that the crawler robot have sent. When the order of packets transmitted by the crawler robot and that received by the operator are different, the operator can learn the difference using these packet numbers. In addition, if the same command is delivered to the crawler robot more than once, the packet number can be used to cancel multiple executions of the same command.

A. Purpose and Content of the Experiment
The purpose of this experiment is to confirm whether the proposed method can reduce the teleoperation time of the crawler robot. In our experiments, we measure the bandwidth utilization and teleoperation time of the crawler robot by conducting stair climbing and descending using the proposed and existing teleoperation protocols. Bandwidth utilization is defined as the ratio of the time the crawler robot transmits packets to the operator during the teleoperation time. Teleoperation time is defined as the time from that when the crawler robot transmits the first packet to the operator to the time when the crawler robot completes ascending or descending the stairs and confirms that it is at the floor or landing level.

B. Experimental Environment and Equipment
The experimental environment is shown in Fig. 9. In the experiment, LRF, which was mounted in front of the crawler robot, was used to acquire information on the shape of the obstacles and stairs as well as the distance between the crawler robot and stairs. Three cameras were used, which were positioned to acquire subjective, overhead, and rearward images. The control system used in this experiment was Raspberry Pi 4 Model B, and the LoRaWAN module was ES920LR manufactured by EASEL Corporation.

C. Experimental Result
In the experiment, the crawler robot was able to ascend and descend stairs without tipping over once in both cases. Fig. 10 shows the experimental results that used the existing method, and Fig. 11 shows those that used the proposed method. The vertical axis indicates whether the bandwidth was used or not, where one denotes that the bandwidth was used and zero indicates that the bandwidth was not used. In the ascent process using the existing method, concrete numerical information (IMU sensor information, distance between the crawler robot and stairs, and shape information of the obstacles and stairs) was transmitted seven times, images were transmitted eight times, and the crawler robot moved six times. In the case where the existing method was used, concrete numerical information was transmitted six times, images were transmitted nine times, and the crawler robot was moved seven times. In the case of the ascent process using the proposed method, concrete numerical information was transmitted seven times, images were transmitted ten times, and the crawler robot was moved six times. In the case of the descent process using the proposed method, concrete numerical information was transmitted nine times, images were transmitted ten times, and the crawler robot moved eight times.
In the ascent process, the concrete numerical information and number of times the crawler robot moved were the same in both methods, but the number of images acquired by the proposed method increased by two. The bandwidth utilization improved by 38.3% from 55.8% to 94.1%, and the teleoperation time decreased by 2 min and 9 s from 14 min and 1 s to 11 min and 52 s. In the descent process, the amount of concrete numerical information acquired using the proposed method was increased by three, the number of images was increased by one, and the number of crawler-robot movements was increased by one. The bandwidth utilization improved by 32.5% from 61.1% to 93.6%, and the teleoperation time decreased by 1 min 58 s from 14 min 10 s to 12 min 12 s. The average time required to determine and input the next command using the existing method was 19.6 s for ascending and 15.7 s for descending. In the proposed method, the average time from the completion of obtaining the necessary information to inputting of the operation command was 21.8 s for ascending and 17.0 s for descending.

VI. DISCUSSION
We confirmed that the proposed method could reduce the teleoperation time, although two more images were obtained in addition to the information acquired by the existing method when the crawler robot was ascending. We confirmed that the proposed method could shorten the teleoperation time, although three times of concrete numerical information and one more image were obtained in addition to the information acquired by the existing method when the crawler robot was descending. Furthermore, the average time of the proposed method from acquisition of the necessary information to inputting of the operation command was longer than the average time required for the decision and inputting of the next command in the existing method, both in the stair climbing and descending processes. Therefore, the reduction in the teleoperation time in the experiment did not depend on the time to decide and input the commands. Hence, the teleoperation time could be reduced by reducing the number of command inputs and improving the bandwidth utilization using interventional commands. Therefore, this method is considered to be effective for teleoperation in staircases with a large number of command inputs because of the need for concrete numerical information and multiple images.
The teleoperation that uses the proposed method could interrupt the execution of a command sent by mistake because we confirmed that the interventional process could be executed. Therefore, this method is considered to be effective because it could deal with increased teleoperation time when wrong command contents were executed.

VII. CONCLUSION
In the teleoperation of crawler robots in a disaster-stricken enclosed space, the distress of the crawler robot due to communication breakdown is a problem. In this study, we have presented a teleoperation method using LoRaWAN as a subcommunication infrastructure to move a distressed crawler robot to the place where wireless LAN communication is possible. In this study, we have also assumed an environment in which the crawler robot was required to ascend and descend stairs when evacuating to the place where wireless LAN communication was possible. We have presented the operation method of the crawler robot on stairs.
In the disaster-stricken environment assumed in this study, sudden appearance of obstacles is expected. When obstacles appear on the staircase, the slope of the staircase changes and the crawler robot may fail to ascend or descend the stairs because of its insufficient climbing ability. In addition, because of the low bit rate of LoRaWAN, several minutes is required to obtain the necessary information to transmit a single operation command. Hence, a risk exists in terms of failure in evacuation actions due to the occurrence of obstacles during stair climbing or descending teleoperation using LoRaWAN. Therefore, shortening the time required for stair climbing or descending and completing the stair climbing or descending process are important before obstacles appear in the teleoperation using LoRaWAN. Hence, we propose an interventional teleoperation protocol to reduce the teleoperation time.
The proposed method reduces the teleoperation time by reducing the number of command decisions and inputs and employing an interventional command. In addition, the proposed method can interrupt the execution of erroneous commands using interventional commands. Through experiments in real environments, we confirmed that the proposed method could improve the bandwidth utilization and shorten the teleoperation time.
In this study, we developed a multipath communication method that increased the bandwidth by adding more communication paths. Therefore, we plan to increase the transmission capacity of the bandwidth by adding a communication path from the crawler robot to the operator to further shorten the teleoperation time.