The Research on the Motion Control of the Sorting Manipulator based on Machine Vision

—With the development of production technology, manipulators are gradually introduced in advanced production manufacturing industries to complete some tasks such as picking and sorting. However, the traditional manipulator has a complicated sorting process and low production efficiency. In order to improve the accuracy of sorting and reduce the labor intensity of workers, this paper studied the motion control of the sorting manipulator with machine vision. After placing four kinds of objects of different shapes on the conveyor belt, experiments were conducted on the catching and sorting process of the manipulator under different experimental environments, different conveyor belt speeds, and with or without machine vision. It was found that the overall success rate of the sorting robotic arm using machine vision for catching objects of different shapes was as high as 96%, and the sorting accuracy was as high as 97.91%. Therefore, it is concluded that the manipulator can achieve high accuracy in catching and sorting objects with the guidance of machine vision, and the adoption of machine vision has a positive impact on the motion control of the sorting manipulator.


INTRODUCTION
In today's rapid development of the intelligent technology industry, industrial automation has become the development direction of the production manufacturing industry, and manipulators have been widely used in industrial operations [1], for example, manipulator sorting technology. Traditional robot sorting is mainly carried out by means of demonstration and can only work in a fixed environment, and the manipulator does not work efficiently because it cannot recognize and process objects without a vision system. The manipulator using machine vision system can take camera shots of the work site to collect images and obtain information such as the location and size of the object to realize catching and sorting. Therefore, the combination of machine vision and manipulators allows the arms to identify objects independently, which has important practical significance for improving efficiency and reducing labor costs. Radcliffe et al. found that the use of machine vision allowed small vehicle platform systems to navigate autonomously and reduce errors through laboratory field tests [2]. Min et al. found that a portable machine-based visual inspection system for track defects could replace manual labor to some extent after field experiments [3]. Liu proposed that robots equipped with machine vision-based manipulators could efficiently perform tasks such as logistics courier sorting and fruit picking in orchards [4]. Abad et al. found through field experiments that machine vision was able to achieve at least 95% accuracy in color discrimination of stacked colored objects [5]. Nair et al. found that the application of machine vision for image analysis helped in classifying flood zones and had an accuracy of 83.1% [6]. Mohamed et al. applied a realtime machine vision manipulator through Python software and found that the arm was very accurate in detecting external defects in agricultural products [7]. He et al. verified that a machine vision-based sorting method was feasible by obtaining the contours, features and dimensions of mechanical parts with a camera and filtering out defective products through image analysis [8]. The article first described the machine vision of the sorting manipulator through camera calibration and image processing methods; then, the inverse kinematic equation in the D-H parameters was used to solve different joint angles of the manipulator to achieve motion control. Finally, after placing four objects of different shapes onto the conveyor belt in turn, the experiments were conducted to investigate whether the application of machine vision is helpful for the motion control of the sorting manipulator by adjusting the experimental environment and the speed of the conveyor belt and using machine vision or not. This work provides a theoretical basis for future research on the motion control of the sorting manipulator using machine vision.

II. MACHINE VISION FOR SORTING MANIPULATORS
Machine vision technology involves several disciplines, including image processing [9], artificial intelligence [10], pattern recognition [11], etc. Computers are used to simulate human visual ability, and their combinations with manipulator catching technology allow arms discriminate and grasp like humans. This paper established a manipulator vision system to realize the camera calibration of the machine vision part and carried out image pre-processing [12] and feature extraction to match and recognize objects. The system determined the catching coordinate position by the image edge contour information of objects, and then positioned, grasped, and sorted objects.

A. Camera Calibration
The purpose of camera calibration [13] is to correct the distorted image and construct a three-dimensional scene based on the obtained image. It is assumed that there is a point named P in the space [14]. Point P is converted into a camera image plane to obtain the image pixel coordinates to realize manipulator catching. The specific conversion process is as follows.
(IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 14, No. 3, 2023 207 | P a g e www.ijacsa.thesai.org where S is a scale factor, R is a 3*3 rotation matrix, T is a 3*1 translational transformation vector, , , and are the homogeneous coordinates of a point in the space under the world coordinate system and camera coordinates, and are the focal lengths of the camera in the X/Y directions, respectively, and and are the position of the image coordinate system origin O in the pixel coordinate system.
where and represent the internal and external parameter matrices of the camera, respectively. The internal parameters of the camera include the internal information such as the focal length, optical axis, and focus position of the camera, while the external parameter of the camera is the conversion relationship between the world coordinate system and the camera coordinate system. The overall conversion relationship is as follows: B. Image Pre-processing 1) Image filtering process: Median filtering [15] can eliminate isolated noise points in an image and effectively protect the boundary information of images, which will not cause severe blur to images like mean filter. The specific method is described below. First, sequence , , , and is defined, and the length of the window corresponding to the median filter is odd number L, L = 2n+1, where n is a positive integer. It is assumed that at a time point, is the value of the signal sample located in the center of the window, and the signal sample inside the window is , , , , . These L signal sample values are ranked from large to small, and the i value in the middle is taken as the output value of median filtering: To obtain better image processing result, the twodimensional median filtering is used: where f(x,y) is the original image and g(x,y) is the image after median processing. W is a two-dimensional template, usually 3*3 or 5*5, and in this paper, a 3*3 two-dimensional region is chosen.
2) Image grayscale: Regarding image grayscale [16] in the RGB model, the value of R=G=B is called the grayscale value, and the grayscale range is , -. When the gray scale is 255, it is the brightest (pure white); when the gray scale is 0, it is the darkest (pure black). There are two main implications of converting the captured sorter image to grayscale image, one is that the grayscale image takes up less memory and has faster computing speed compared to the color image; the second is that the contrast is more obvious visually after conversion to grayscale image, highlighting the object location. After grayscale processing of the image of the sorter captured by the camera as in Fig. 1, the next step of feature extraction operation can be performed. 3) Image segmentation: Image segmentation [17] is a key step from processing an image to analyzing it, and is an indispensable pre-processing for recognition images and computer vision. In the acquired image of the sorter, the identified object is found to be only a small part of the overall image, and the excess makes the system slow in processing the image, so the image is segmented to highlight the object. In this paper, Matlab code [18] will be used to segment the sorter images, using the functions shown in Table I.

III. MANIPULATOR MOTION CONTROL
The manipulator is an artificial intelligence device, and its motion control is the basis for ensuring the stability of the arm's posture [19]. The kinematic analysis of the manipulator can be divided into forward kinematics and inverse kinematics. Forward kinematics means deducing the pose of the endeffector of the manipulator relative to the reference coordinate system when the geometric parameters of the connecting rod joint and the angular value of the joint angle have been known. Inverse kinematics means calculating the joint angle of the www.ijacsa.thesai.org manipulator when the geometric parameters of the connecting rod joint have been known and the pose of the end-effector of the manipulator relative to the base coordinate system has been given. D-H parameters [20] determine the coordinate change relationship between adjacent joints by assigning a coordinate system to every joint of the manipulator (Fig. 2), then the joint transformation relationship are combined through a mathematical formula to determine the overall change relationship between the end-effector and the base to obtain the related kinematic equation. D-H parameters contain four basic parameters, as shown in Table II.
where P is the translation vector used for determining the coordinates of the end of the manipulator in the space, n, o, and a denote the rotation vectors used for determining the attitude information of the end of the manipulator. The equation is solved by isolating the joint variables. Joint variable can be obtained by left multiplication of the unknown inverse transformation of connecting rod and the two sides of (7).

A. Experimental Principle
The working principle of the sorting manipulator system used in this experiment is shown in Fig. 3. The image of an object is captured by an industrial-grade 3D smart camera [22]. The image data captured by the camera is transmitted to the input port of the image capture card. The image capture card transforms the analog video signal into a digital image. The computer program calculates the specific position of the object through the image and sends the position coordinates to the control system of the manipulator. Finally, the manipulator grasps and sorts the object.

B. Experimental Design
The machine vision-based manipulator first scans the object with bilateral cameras, then pre-processes and recognizes the collected object image, and finally catches and sorts the object. Since the objects on the conveyor belt are dynamic and the background of the images captured by the cameras is variable, it is also important to study the manipulator's catching and sorting for dynamic objects under different experimental environments. The objects were placed under three different conditions, including different experimental environments, different object movement speeds, and with or without machine vision, 100 times each condition. Four objects of different shapes [23] were placed onto the conveyor belt in turn, and the conveyor belt and manipulator were turned on. The original environment of the experiment was an empty laboratory without any shade and with no shadow on the conveyor belt under the light. The main change of the experimental environment was to add large potted plants to the original environment and place them under the light to make their shadows cast the conveyor belt. The specific change before and after the experimental environment is shown in Fig.  4. The number of times the manipulator successfully caught and the number of times it correctly sorted under different experimental environments and movement speeds of the four objects were recorded, and the success rate was calculated. Finally, the catching and sorting results of the manipulator with and without machine vision were presented in the form of a table.

C. Experimental Results
Sorting accuracy formula: sorting accuracy = number of correct sporting/number of successful catch.
According to the experimental data displayed in Table III, it was seen that the manipulator completed the catching and sorting of objects of four shapes in the 100 times of catching and sorting under every condition. After calculation, the overall catching and sorting rates of the manipulator with machine vision were 96% and 97.91%, respectively, while the overall rates of the manipulator without machine vision were 84% and 57.99%, respectively. It was also found that the catching rates for the objects of four shapes showed decreasing trends, but the decreasing trend in the sorting rate was more significant than the catching rate. Thus, it was concluded that the manipulator with machine vision had a huge improvement for both object catching and sorting, i.e., machine vision had a positive influence on the motion control of the manipulator.
It was seen from Table IV that when the speed of the conveyor belt rose to 150 mm/s, the success rates of catching and sorting clothes, ball sports equipment and ring-pull cans gradually decreased, but the sorting success rate of the cardboard box remained at 100%. Overall it was seen that the sorting rate using the machine vision-based manipulator remained above 90%. This showed that although the number of successful catching and correct sorting decreased as the speed of the object moved faster, but the manipulator with machine vision could still effectively catch and sort objects. The results fully demonstrated that machine vision was beneficial to the motion control of the manipulator.
In order to compare the effect of the machine vision-based manipulator for object catching and sorting under different experimental environments, large potted plants were placed next to the conveyor belt to create a shadow influence. It was seen from the data in Table V that the catching and sorting rates of objects of four shapes in the complex environment decreased, and the catching rate of cardboard box decreased from 96% to 80%, which was the most significant. However, from an overall perspective, the manipulator with machine vision performed better than the manipulator without machine vision. This showed that the application of machine vision had a very significant impact on the sorting service of the manipulator and machine vision helped the manipulator to sort objects.

V. DISCUSSION
The manipulator plays an important role in the field of automation engineering as an important component of industrial systems [24]. A study has developed an improved machine vision system that is capable of identifying and classifying items with different geometric shapes and colors and manipulating and separating them using a computercontrolled manipulator [25]. The experimental parameters in this paper differed from those in existing studies in the following aspects: (1) existing experiments use three cameras, while this paper used two cameras; (2) the sorted objects in existing research are in a condition of rest, while the sorted objects in this paper are in a state of motion; (3) existing research on manipulators under machine vision is oriented to grasp and sort items according to different geometric shapes and colors, but this paper took into account not only the different geometric shapes of the items, but also the different running speeds of the items on the conveyor belt and the complexity of the environment in which the manipulator performs sorting. In this paper, by reviewing a large number of literature on machine vision and robotic arm, the authors further deepened the research on the use of machine visionbased sorting manipulator on the basis of existing research and achieve good experimental results. However, there is still room for further research. www.ijacsa.thesai.org 1) This paper showed that the correct sorting rate of manipulators using machine vision decreased to a certain extent in the case of high speed motion of sorted items and in the case of complex sorting environment, and subsequent studies can be conducted to analyze these problems.
2) To improve the sorting speed of the manipulator using machine vision, speeding up the image acquisition or transmission process can be considered.
3) Whether classification algorithms can be added for sorting items, and whether classification algorithms can play a role in improving the correct rate of robotic arm sorting should be studied.

VI. CONCLUSION
This paper mainly introduced the machine vision and the motion control of the sorting manipulator. After putting four kinds of objects of different shapes on the conveyor belt in turn, the cameras on both sides scanned the objects, the collected images were fed back to the manipulator control system after processing, and the catching and sorting behaviors of the manipulator were tested by adjusting the experimental environment and the speed of the conveyor belt and applying machine vision or not. The experiment found that among the 300 catching and sorting tests, the overall catching and sorting success rates of the manipulator were 96% and 97.91%, respectively, while the manipulator without machine vision had an overall catching rate of 84% and an overall sorting success rate of 57.99%. The experiment verified that the manipulator guided by machine vision could improve the ability to perceive and recognize the external world and achieve high accuracy in catching and sorting objects. Therefore, the application of machine vision has a positive impact on the motion control of the sorting manipulator, laying a foundation for the future application of machine vision in sorting service in practice.