Model-based Pedestrian Trajectory Prediction using Environmental Sensor for Mobile Robots Navigation

Safety is the most important to the mobile robots that coexist with human. There are many studies that investigate obstacle detection and collision avoidance by predicting obstacles’ trajectories several seconds into the future using mounted sensors such as cameras and laser range finder (LRF) for the safe behavior control of robots. In environments such as crossing roads where blind areas occur because of visual barriers like walls, obstacle detection might be delayed and collisions might be difficult to avoid. Using environmental sensors to detect obstacles is effective in such environments. When crossing roads, there are several passages pedestrian might move and it is difficult to depict going each passage in the same movement model. Therefore, we hypothesize that a more effective way to predict pedestrian movement is by predicting passages pedestrian might move and estimating the trajectories to the passages. We acquire pedestrian trajectory data using an environmental LRF with an extended Kalman filter (EKF) and construct pedestrian movement models using vector auto regressive (VAR) models, which pedestrian state is consisting of the position, speed and direction. Then, we test the validity of the constructed pedestrian movement models using experimental data. We narrow down the selection of a pedestrian movement model by comparing the prediction error for each path between the estimated pedestrian state using an EKF, and the predicted state using each movement model. We predict the trajectory using the selected movement model. Finally, we confirm that an appropriate path model that a pedestrian can actually move through is selected before the crossing area and that only the appropriate model is selected near the crossing area. Keywords—Prediction of Human Movement; Service Robots; Vector Auto Regressive Models; Kalman Filter; Collision Avoidance


INTRODUCTION
Various service robots are expected to coexist with humans in real environments.Examples include guidance, communication, and assistant robots.These robots must approach a service user and avoid other humans according to the situation.Especially, for the safe behavior control of autonomous robots that coexist with humans, there are many studies that investigate obstacle detection and collision avoidance using mounted sensors such as cameras and laser range finder.For safety and collision avoidance, several methods have previously been proposed to allow autonomous robots to avoid local collisions reactively: potential field methods [1,2], social force methods [3], dynamic window approaches [4][5][6], and vector field approaches [7,8].Furthermore, collision avoidance methods for dynamic obstacles such as pedestrian have been proposed which function by predicting obstacles' trajectories several seconds into the future and making decisions based on these predicted trajectories.At present, trajectory prediction methods are important because of the risk of a collision between obstacles and the robot when the trajectory prediction is not sufficiently accurate.With this in mind, this study focused on predicting the dynamic trajectories of pedestrians.
Several methods for predicting pedestrian trajectories assume that pedestrians move with constant speed [9][10][11][12][13].This assumption may only be effective for short-term predictions because pedestrian trajectories can also change under the influence of the environment.Therefore, some pedestrian trajectory prediction methods considering pedestrian movement tendencies using pedestrian trajectory data that are observed in advance have been proposed.
Those methods predict the trajectory using the current state (pedestrian position and velocity) or the current and previous states.However, a pedestrian's trajectory changes with each step near crossing areas, for example when crossing roads at a crossing point.It may be more effective to consider the pedestrian's state several steps in the past.In this study, we constructed pedestrian movement models based on vector auto regressive (VAR) models.We approximate a pedestrian's position, speed, and direction of movement and predict their trajectory using their states several steps in the past.
Moreover, obstacles may be detected too late to avoid collisions in environments with blind areas caused by visual barriers like walls.In such environments, pedestrian movement prediction methods using environmental sensors are effective [14].In this study, we constructed a model and predicted the pedestrian's trajectory using an environmental sensor.
It is thought that pedestrians change their direction step by step near environments where multiple passages cross (e.g., when crossing roads).There are many paths to the destination, far from the crossing area.To realize safe mobile robot navigation in such environments, we must construct each path model and predict the pedestrian trajectory, and also evaluate each predicted trajectory and select the appropriate path model for the pedestrian.
This study proposes methods that predict a pedestrian's trajectory, evaluates each predicted trajectory, and selects the This study was supported by JSPS KAKENHI Grant Number 15H02890.www.ijacsa.thesai.orgpedestrian's approaching path using an environmental sensor.We expect that a robot can more effectively avoid pedestrians using this method than existing methods, because it reduces the number of candidate paths near the crossing area.
In concrete terms, we construct pedestrian movement models as follows.First, we acquire pedestrian trajectory data using an environmental LRF with an extended Kalman filter (EKF).Second, we construct VAR models of degree ranging from 2 to 30 for each path.Third, we compare the prediction accuracy for each degree.Then, we decide the pedestrian movement models' degree and verify the constructed models' accuracy.We narrow down the selection of a pedestrian movement model by comparing the prediction error for each path between the estimated pedestrian state using an EKF, and the predicted state using each movement model.Then, we predict the trajectory using the selected movement model.In this study, we verify the validity of the constructed pedestrian movement models using experimental data.Furthermore, we confirm that an appropriate path model that a pedestrian can actually move through is selected before the crossing area and that only the appropriate model is selected near the crossing area.

II. RELATED WORK
Many existing pedestrian trajectory prediction methods use the current state (e.g., pedestrian position and velocity) or the current and previous step states.Shiomi et al. proposed a method that predicts a pedestrian trajectory using the social force model [15].Similarly, Ratsamee et al. proposed a method that predicts pedestrian trajectories using social force models, considering pedestrian's body pose, face orientation, and personal space [16].Tamura et al. proposed a method that predicts pedestrian trajectories by storing state transition data in each 1 m 2 and predicting state transitions using the current pedestrian state and the stored state transition data [17].Tadokoro et al. proposed a method that predicts pedestrian movement by estimating movement tendencies via trial and error when a pedestrian moves in an environmental cell [18].Noguchi et al. proposed a method that predicts pedestrian movement paths by modeling pedestrian movement between cells based on a variable length Markov model [19].Other researchers have proposed methods that build pedestrian models using machine learning.Chung et al. used Markov decision processes [20,21], and Ziebart et al. used a softmaximum version of Markov decision processes [22].Callaghan et al. proposed using a Gaussian process [23] and Ellis et al. used Gaussian process regression [24].These methods do not consider pedestrians' destinations when predicting their movement.However, several methods have been proposed that estimate destination and predict the trajectory toward that destination.Thompson et al. proposed a method that derives the transfer probability of each destination, estimates the destination using random sample consensus, and then predicts pedestrian movement using the derived transfer probability [25].Bennewiz et al. proposed a method that estimates the destination using an expectation-maximization algorithm, and predicts the trajectory using hidden Markov models [26].Foka et al. proposed a method that predicts a pedestrian's position at the next step using the current and previous step based on a polynomial neural network.They estimated the destination using the tangent vector of the obstacle's positions at times 1, tt  and the predicted position at time 1 t  [27, 28].These methods predict indoor trajectories toward destinations such as the TV and the refrigerator.However, when crossing roads where a blind area occurs because of walls, it is difficult to depict taking a right turn, going straight and then taking a left turn in the same movement model.Therefore, we hypothesize that a more effective way to predict pedestrian movement is by predicting pedestrian destination passages and estimating the paths to the passages.

III. ACQUIREMENT OF PEDESTRIAN TRAJECTORY DATA
We conducted an experiment to acquire pedestrian trajectory data when crossing roads using LRF (UTM-30LX, Hokuyo Automatic Co, Ltd., Japan) at the height of 0.22 m that is pedestrian thigh.Figure 1 shows the experimental environment and pedestrian movement direction.We observed the distance to the obstacles at each 0.050 s using LRF in advance.Then, we acquired the position of pedestrian on each time step and collected 157 trajectory data points in total using an EKF with position as the observation value.Table 1 shows the number of trajectory data points that we acquired.We used 142 data points for constructing models and another 15 data point for verifying the models.The process of acquiring the trajectory data is as follows.
To acquire the position data, we compared current LRF data and environmental LRF data that we acquired in advance without the presence of obstacles.Then, we acquired leg data that was different from the environmental LRF data by more than 0.10 m.The   li-th observed position data point is and we cluster by thus a is 0.10 m. Figure 2 shows examples of data from one pair of legs, and the derived position at that time.b is the width of the cluster, and the b of the data from each pair of legs is less than 0.20 m. Figure 2 (a) shows the pattern with the legs apart.In this case, we acquire the data as one pair of legs when the adjacent cluster distance c is less than 1.0 m. Figure 2 (b) shows the pattern with the two legs together.In this case, we acquire the data as one pair of legs when b is more than 0.20 m.The position of the pedestrian is at the center of the pair of legs.www.ijacsa.thesai.orgNext, we acquire the trajectory data, which consists of the position data at each time step.Acquired position data includes sensor noise stemming from the LRF accuracy, and the system noise stemming from the position acquisition process.So, we estimate the trajectory data by considering this noise using an EKF [29].
We define the pedestrian state vector k x at current time step k as: and k v is movement speed.The state equation and observation equation are as follows.: thus the observation value k z is: where k w is the system noise and k v is the observation noise.
The prediction and correction steps of the EKF are given by: where k


x is the a priori estimation value and k  P is the error covariance; and where k K is the Kalman gain that needs to be calculated, x is the a posteriori estimate value, and k P is the error covariance.We define k F , Q R as follows: where   and v  are the variance values of the system noise, and x  and y  are the variance values of the where we acquire the LRF data every 0.050 s.
Figure 3 shows the estimated pedestrian trajectory data using the EKF.In the following, the estimated pedestrian state vector k X is:

IV. CONSTRUCTION OF PEDESTRIAN MOVEMENT MODELS
We construct pedestrian movement models using VAR models.To predict pedestrian trajectory accurately, it is necessary to use high degree models.However, more time is needed for these models to predict trajectories than for lower degree models, because they have to use more time step data.Therefore, it is necessary to construct models in which prediction error is small but degree is low.The construction of the pedestrian movement is as follows.
First, we construct each 2-30 degree VAR model.VAR models (p) enable us to predict the ( 1 k  )th step in the state given the state at the k th step: where d is a direction parameter that can be concretely right ( r ), straight ( s ), or left ( l ).
Second, we derive the coefficient d β using the maximum likelihood method for each degree to compare accuracy.The multidimensional normal distribution of y with mean μ , covariance matrix Σ and degree D is: X is of the 4 th degree, the likelihood function and log-likelihood function of k X that have mean ˆd X and covariance matrix d Σ are: where d n is the number of data steps that are used to construct the VAR models, and the number for each direction is: We estimate VAR models' (p) coefficients 01 , , , β β β , which maximize the log-likelihood function, using the maximum likelihood method.
Third, we compare the models of each degree and decide the degree of the pedestrian movement models.The center is approximately 1.1 m from the edge of the passage, considering the general width of the passage is 2.3 m.Therefore, it takes approximately 2.2 s until the end of the avoidance procedure when the speed of the robot is assumed to be 0.50 m/s.We consider that the robot can avoid a pedestrian with enough margins by predicting the pedestrian trajectory up to 3.0 s in the future.Therefore, we predict the pedestrian trajectory up to 3.0 s in the future.Accordingly, we compare position prediction error d k E up to 3.0 s in the future and decide the pedestrian movement models' degree.
where k P and ˆd k P is the position vector of k X and ˆd k X .The state at the ( kj  ) th step using the data before the k th step is as follows: ˆˆ( 2) ˆˆ() shows prediction error d k E until 3.0 s in the future for each degree.From Fig. 4, we decide that the pedestrian movement models' degree 8 p  because the prediction error decrease after this point is very small.Next, we verify the appropriateness of the constructed models.The pedestrian trajectory needs to be predicted with about 0.50 m accuracy considering the relative sizes of pedestrians in the environment.From Fig. 4, the constructed models satisfy this prediction accuracy and enable the robot to avoid obstacles safely.

V. PREDICTION OF PEDESTRIAN TRAJECTORY
It is advisable to predict all trajectories that pedestrian might move for safety when crossing roads.It is difficult to predict destination passage pedestrian might move when pedestrian walks far from the crossing area.So, we assume all passages as pedestrian destinations and predict trajectories toward each passage as shown in Fig. 5.However, we can pare down the candidates near the crossing area because pedestrians change their moving direction to the destination.Therefore, we predict toward most likely passage near crossing area as shown in Fig. 6.To validate the model against the actual pedestrian trajectory, we used the verification data as shown in Table 1. Figure 7 shows the predicted and actual trajectories.Tables 2  and 3 show the y coordinates at which the appropriate model was chosen.VI.DISCUSSION Figure 7 confirms that the number of selected models decreased and that only one appropriate model was selected near the crossing area.The selected models narrowed to only one appropriate model at 2.0 y  when turning right (Fig. 7  (a)).However, the narrowing of the selection of models is late when heading straight and turning left (Fig. 7 (b), (c)).Moreover, in Table 2 there is no change in the point at which the appropriate model is selected when going straight or turning right or left, but the point only appreciate model selected is later when heading straight and turning left in Table 3, similar to Fig. 7.

TABLE IV. PREDICTED TRAJECTORY ERROR AT
The point at which the selection of models narrows stems from the environment.The environment that we experimented with has a wide road on the right and a narrow road on the left.Moreover, most of the participants whose trajectory was acquired were students who used this environment often, and whose curvatures when turning are thought to be small when turning right and large when turning left.So, there was likely little difference in the position error when heading straight or turning left, because participants tended to begin turning further before the crossing area when turning right than when heading straight or turning left.Tables 4 and 5 show the means and standard deviations of the prediction error d k E at 1.5 y  and 2.0 y  , that is, the mean points where only one appropriate model was selected when turning right and left.We confirm that the constructed models satisfy the prediction accuracy that is necessary for safe obstacle avoidance in an autonomous robot, because pedestrian trajectories need to be predicted with about 0.50 m accuracy considering the size of a pedestrian in the environment.

VII. CONCLUSION
We proposed methods that predict a pedestrian's trajectory, evaluate each predicted trajectory, and select the pedestrian's approaching path using an environmental sensor, for mobile robot navigation.We believe that a robot can avoid a pedestrian with enough margins using the proposed method.Our technique predicts pedestrian trajectories by selecting likely models for environments where several passages cross, and using only one model in environments with only one passage.This method can predict trajectories of several pedestrians if combined with, for example, the potential field or social force methods, and by considering the influence of other pedestrians.
We demonstrated a method to construct pedestrian movement models based on VAR models that consist of pedestrian position, speed and direction for each passage using trajectory data that was acquired in advance by sensors in an environment where a blind area occurs to a mounted sensor on an autonomous robot when crossing roads.We also demonstrated a method of determining the degree of the model,

Fig. 4 .
Fig. 4. Prediction error d k E until 3.0 s later for each degree (red: right trajectory, green: straight trajectory, blue: left trajectory)

TABLE I .
NUMBER OF EXPERIMENTAL TRAJECTORY DATA