Abandoned Object Detection using Frame Differencing and Background Subtraction

Tracking objects over fixed surveillance cameras are widely used for security purposes in public areas such as train stations, airports, parking areas, and public transportation for the prevention of terrorism. Once the object is accurately detected in the image scene, we can use various visual algorithms to find a number of applications. In this paper, we introduce a model for tracking the multiple objects along with detecting the abandoned luggage in the real time environment. In our model, we used the initial frames to model the background scene. Next, we used the motion model that is background subtraction to detect and track moving objects such as the owner and the luggage. The proposed model also maintains the position history of moving objects followed by the frame differencing technique to find out the luggage history and detect the abandoned luggage by a human. We have used PETS2006 and PETS2007 dataset for the testing of the proposed system in various indoor and outdoor environments with varying lighting conditions. Keywords—Object detection; video surveillance; tracking; background subtraction; frame differencing; motion model


I. INTRODUCTION
Multiple object tracking and the detection of an abandoned object is a common and important visual-based application used in a video surveillance system. Thus, finding an accurate and efficient algorithm for the abandoned object detection is still a challenging task for the researchers. A good object detection model can improve the surveillance system and improve security to save lives.
Global security and terrorism are some of the major problems the world is facing these days. In recent years, most terrorist attacks happened in public places involving some suspicious objects which are left unattended at public places such as train stations, market places, public transport, and airports.
Human life is so precious and mean to be protected, in the observation of protection it can be classified as suspicious behavior of a person like attempting to attack or though the unattended dangerous objects like suspicious baggage detection in public places. These terrorist attacks have highlighted the need for the video surveillance systems at public places and guarded 24 hours round the clock.
In this situation, it is important to introduce and deploy an autonomous visual-based model [1], [2] to detect the unavoidable activity that can recognize suspicious activities in public areas. Mostly the threat is generated from the unattended or left luggage in the public area, See Fig. 1.
Various research studies conducted on this area to make video surveillance [3] more versatile and reliable but the detection of suspicious objects is still a challenging job in video surveillance. We need a system that distinguishes and identify highly hazardous situations and makes alerts to take proper action.
In this research paper, we introduce a general visual-based framework that autonomously detects the unattended baggage in forbidden areas. Our model detects and tracks multiple humans with luggage and for a period of time. As the human leaves the luggage, the proposed model detects the left luggage and mark it unattended luggage.

II. RELATED WORK
In the left luggage scenario, we need to localize the abandoned objects and also classify them. In this section, we will discuss the existing literature about the abandoned object.
Li et al. [4] used the long and short term Gaussian mixture model in the RGB color space to build two binary foreground masks. The radial reach filter (RRF) model used to refine the mask and control the illumination changes. For identifying the left luggage, they used the linear SVM classifier, the histogram of oriented gradient (HoG) for the feature extraction, and the width and height ratio of the object.
Filonenko et al. [5] proposed a sequence of dual background difference (SOBD) model which is achieved by differencing the intensities of the current and the referenced background model to find out the static pixels in the image scene. An object detector is later integrated with the clustering method to identify the unattended luggage.
Hassan et al. [6] generated an illumination free template and proposed a tracking method to detected and track the left luggage in the public area. The intensity values were used to generate a binary mask containing all the moving objects of the scene. They tracked the binary blobs and extracted the static pixels with the help of the centroid-range. Laplacian of Gaussian filter is later applied to the current and background frame including the static regions of the frame to get the highfrequency components. The total energy is computed for both the current frame and the background frame to make sure the illumination change does not participate in the segmentation. Finally, they recorded the edge map and perform tracking with the use of a correlation matching method.
Jadhav et al. [7] used a monocular camera to detect the unattended objects in a video surveillance environment. They modeled two backgrounds transient and permanent backgrounds. Both backgrounds were defined as a Gaussian mixture model (GMM) and subtracted from the current frame to receive two binary backgrounds. They also applied the shadow removal algorithm to get the real shape of the foreground object. Later they used size, height, width, and color features to classify the extracted object. The output is the abandoned object.
Sirisha et al. [8] used a background subtraction algorithm and mixture of Gaussian (MOG) to produce the long and short term foreground models. A backtracking mechanism used to identify the owner of the luggage. The hardware ARM 7 microcontroller, buzzer, and GSM used to generate alerts. The PETS 2006 and AVSS 2007 dataset used for the experiments.
Foggia et al. [9] proposed a technique to detect the motionless objects in the video sequence. The method encoded the spatio-temporal information into the heat map. The heat map carries all the information about the tracking object in a video instead of updating the background by evaluating the movement of the objects. The experiments performed over two well-known datasets and the acquired results were compared with the state of the art methods to confirm the robustness and efficiency of the proposed system.
Lin et al. [10] used a double Gaussian Mixture Model in the RGB color space, the space was able to receive the two binary foreground masks. They refer to the two Gaussian Mixture Models as long and short term models. The proposed Radial Reach Filter (RRF) method refined the foreground masks and reduced the illumination changes. Later, they used a descriptor Histogram of oriented gradient (HoG) with a linear support vector machine (SVM) in order to recognize the left luggage. The authors generated their own datasets PETS2006 and PETS2007.
Bhargava et al. [11] brought a technique to avoid terrorism and to strengthen the global security issues with the significance of unattended baggage in mass transit areas. They described the algorithms to recognize four events that kept track of the activity. The algorithm identifies the owner of the unattended luggage based on the position history of the objects. The long absence of the owner from the image identified the status of the left baggage.
Grzegorz Szwoch [12] proposed a method to identify the stationary objects in the images. The method separates those objects from the static background that remained motionless for a long time. The image's stable pixel values in a time of the foreground regions are used to a build relationship with the movable objects. Firstly, the model tests the pixels stability belong to the moving target object based on vectors.
In the next stage, the model extracts the image regions with the cluster of stable color and brightness and related contours of the detected objects. The false contours belong to the object removed from the background. The Author presented a complete framework for the unattended baggage detection to use it in the event detection. The experiments showed the validation of the methodology.
Lin et al. [13] brought improvements in a method to detect suspicious stationary objects in images. The proposed machine model addresses the abandoned object in an image through the extraction of foregrounds and Stationary foregrounds for the real-time monitoring systems.

A. Frame Differencing
The frame differencing is the technique, we use to detect motion and find out the moving objects and stationary objects in the image scene.
Nishu Singla [14] proposed a frame differencing technique is an algorithm to observe the motion in the image scene and detect the moving objects using a fixed surveillance camera. In this technique, the model captured the image from the static camera and sequence of image from the camera stream. In the second phase, the absolute difference is calculated between the consecutive frames and the record the difference. Finally, the image processing techniques are applied to remove the noise.
Gupta et al. [15] in his work suggested three methods to find motion in the image scene. The background subtraction, frame difference, and self-organizing background subtraction (SOBS) to detect the moving object in the video frames.
Thapa et al. [16] used a background subtraction technique to detect moving objects and perform segmentation. They achieved the detection and segmentation using the differencing and summing algorithm. The method has low computational complexity to work in the real environment.
In this paper, we proposed a reliable algorithm to detect motion and find humans in the surveillance video stream and identify the left luggage. We used the background subtraction technique to detect motion and find the new objects appearing in the image scene. Additionally, we used the frame differencing technique to find out the status of the detected object by calculating the tracking motion history that helps to mark them as left luggage.

III. PROPOSED METHODOLOGY
In the proposed algorithm, we use the initial frames to build the background model to use it later for finding the history of the position of the new object and identify the abandoned objects.
After learning the background, the model detects the new objects in the observed scene using the reference background model against the new frames. We use the frame differencing technique to detect the new objects in the scene. Once we have the new objects, we extract them from the image frame and record their central position. Next, we need to classify the detected new objects into moving or stationary objects. For the classification, we record the new object position and keep them tracked. If the tracked object position remains stationary (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 7, 2020 or unchanged for more than 100 frames we classify the object as an abandoned object otherwise moving object. The block diagram gives a review of the proposed method with various steps in consecutive order, See Fig. 2. We describe the details of each step in the succeeding sub-sections.

A. Pre-processing
The preprocessing is a necessary and standard step used generally in every machine vision algorithms. This will help to remove the noise in video frames to avoid false detection and improve the performance of the algorithm.
In the proposed method, we first resize and crop the input image to discard the unwanted pixels. Next, we convert the image into grayscale followed by the Gaussian filtering to smooth the image and suppress the noise.

B. Background Model
After preprocessing the frames, we apply the background learning technique to build the background model to enable the proposed technique to detect the moving objects when it appears in the image scene and keep them tracked. The proposed algorithm detects and tracks multiple objects simultaneously.
The background learning module takes the initial input frames and uses them to build a background model, we use the background model as a reference model to detect motion. After this step of learning the background model, we feed the frame to the next phase of the algorithm to detect and extract the new objects in the image scene.

C. New Object Detection and Pre-processing
Detecting the new objects in the image scene is an important step in the proposed technique. First, the proposed method learns the background from the initial frames later it checks the presence of the new objects using the frame differencing technique.
The detection of the newly appearing objects in the image scene works on frame differencing method. We use the background model as a reference model and difference with the new frame to find the motion and new objects. And if the object is found it status would be either static condition or moving.
This method works on the pixel-based difference of the frames to find moving or static objects based on the computed position history explained later in the sub sections.

D. Object Classification
Next the proposed method classifies the newly detected object into static and moving objects based on the computed position history of the objects. With the tracking and detection of the new objects, the proposed method also computes the position of the detected objects, the method also stores or save the object position.
The frame differencing method help to find the motion in the image scene that ultimately detects objects after that, we calculate the centroid of the detected objects and store the centroid position. We repeat the process of every upcoming new frame and check the position of the centroid. If the centroid position of the detected objects remains stationary or unchanged for the number of frames, we classify them as static objects otherwise we reset the position counter and declare as the moving object.

E. Identifying the Abandoned Baggage
After calculating and saving the position history of the objects, the object is classified as a moving or static object based on the position history. The next step is to declare the static object as the abandoned baggage. If the object remained in static condition for the threshold of 100 frames, it would be declared as abandoned baggage or abandoned object detection. The static object is marked as drawing the red bounding box around it and marked as left luggage.

IV. EXPERIMENTS AND RESULTS
In this section of experiments and results, we detail the experimental setup we carried to testify the proposed methodology. We used various datasets to test the robustness of the proposed methodology. The first experiment is carried on different video datasets and showed the method successfully detected the abandoned baggage.
The dataset of [10], we used for the detection of moving objects and Abandoned objects. Videos containing different objects like people, bikes, bags, or suitcases in different environments.

A. Experiment I
In experiment I, we used the dataset [10] where the video was recorded in the street and an outdoor environment. In the video, the person appears in the scene and keep on walking in the field of view, the person also moves off the image scene and reappears in the scene after some time to make the dataset more challenging and fairly tested the robustness of the proposed methodology. The method detects a person as a moving object and draw a blue bounding box around him and keeps him tracking.
In the same video dataset, another person appears in the scene with carrying baggage and after walking a little, he left his baggage in the scene and walks away. After some time the bag was detected successfully as a left luggage and shows a red bounding box around it. When the person came again into the scene it detects him again and draws a bounding rectangle around him, see Fig. 3.
The first column shows the moving object with the baggage, the algorithm draws a blue rectangle around the moving object. In column two the moving object human left the baggage and do not move away, our system still considers them as one object. In the third column, the left baggage is detected as a new object and draw a blue rectangle. Our proposed method also records the location of the detected objects in this case the location of both human and left baggage is stored. After some time in the fourth column, the baggage is declared the left luggage and rectangle in red based on the location history of the baggage. In the last column, the object reappears in the image scene and the proposed system detects it back, see Fig. 3.
Each row shows the various steps of the proposed algorithm applied to each frame. The first row shows the original input frame in RGB (red, green, blue) color space. In the second row of the result, we applied mask and frame differencing to detect the new objects including moving and abandoned objects. In the third row, we binarize the image to segment the image and extract the desired objects such as human and left luggage. The fourth row shows both the objects in binary form while they are separated from each other, see Fig. 3.

B. Experiment II
In experiment II, we used the video from the dataset which is captured in an indoor environment. The scene shows a front After some time one person came with the bag on his shoulder and meet another person. When the bag was on the shoulder of a person, it is part of that moving object or person. While talking with the other person the method detects it as two objects in the scene and draws blue bounding boxes, then the person puts off the bag on the floor and moves away from the bag.
As he left the baggage on the floor and walks away, the proposed method detected the left baggage as a new object in the scene. When the same person came again into the scene it was detected as a new object and the baggage left on the floor was detected successfully as left luggage with red bounding box, see Fig. 4.
We show the various frames consist of moving objects and left baggage column-wise. In the first column, frame 1050 shows the two persons coming from two different sides of the scene and greetings each other, the algorithm draws a blue rectangle around both moving objects. The algorithm works efficiently on the multiple objects detection and tracking. In the second column frame 1279, both objects came closer and had some talk while the talking the person carrying a bag on shoulder, putting it off from the shoulder, and put aside on the floor. In column third frame 1428, both persons leaving the scene together having ones hand on the others shoulder, it is detected as one object because the there is small space between them to recognize as two separate objects by the system. The bag on the floor is detected as a new object and successfully detected by algorithm having a blue bounding box around it. In the fourth column frame 1797 the baggage remains in the static condition and detected as an object having blue bounding box on it. After some time the baggage is declared the left luggage and draws a red rectangle on the object based on the position history of the baggage, see columns in Fig. 4.
The key steps for each frame are shown row-wise. The first row shows the input frame in RGB (red, green, blue) color space. In the second row, we applied mask and frame differencing to detect the new objects including moving and abandoned objects. In the third row, we binarize the image to segment the image and extract the desired objects such as human and left luggage. The fourth row shows both the objects in binary form while they are separated from each other, see Fig. 4. From the past few years, global security and terrorism are one of the major problems in the world. Terrorist attacks caused human death and most probably happened in public areas like airports, transports, and market places. In order to overcome the security issues, there is the need for automated surveillance systems probably in the public places.
We proposed a framework to detect the abandoned luggage in the public area. The proposed system is efficient to work in the presence of occlusion, noise, and affine distortion. The algorithm learns the background and then detects the moving object in the foreground through a contour-based method and draw a bounding box. The proposed method detects the left object using the frame differencing to find motion and new objects in the image scene. The method additionally calculates the detected object position, check the background history for the period of time. If the detected object centroid position remains unchanged for some time, it is detected as left luggage.