Video Processing for Animation at Key Points of Movement in the Mimosa Pudica

The processing of an image of a moving plant is inadequate, for this reason, digital video processing must be incorporated, which allows the behavior of an algorithm to be analyzed over time. A method is presented that takes images of a plant with autonomous movement filmed on video; the frames are digitally processed and the information is used to generate animations. Our representation of the structure is derived from an analysis of the image where the plant is deformed; the projections of the movement of the plant are recovered from the video frames and are used as a basis to generate videograms in an animation based on key points taken from an image; Harris and Brisk algorithms are applied. The main plant used is the Mimosa Pudica. Once the frames have been obtained, correlation is proposed as a mechanism to find movement. The techniques are equally useful for any other moving plant such as carnivores or sunflowers. Keywords—Harris; Brisk; correlation; ROI (Region of Interest); Canny. Sobel; Mimosa Pudica; movement


I. INTRODUCTION
Most people think of plants as inactive [1]. Plants move in different ways and for different reasons [2]. Numerous investigations have led to a deeper understanding of the physiology and biomechanics of these living beings; however, they are not yet fully understood [3]. Because their movements occur in multiple ways [4].
The leaves of Mimosa pudica and many other legumes are characterized by their motor organs that allow the leaves to carry out sleep movements [5] [6]. These actions appear to be regulated by electrical signals and chemical properties [7]. The movements go through four stages: open state, closed state, locked state, semi-open state [8] [9]. The investigations try to discover if these plants have an electrical component [10]; based on the characteristic of cyclic voltage current, where the possibility of memory must manifest itself [11]; in this way, mathematical and electrical-chemical models have been proposed [12].
The study using video shots when the plant is impacted by rain or some other substance can reveal how the system is activated [13]. However, Mimosa pudica rapidly closes its leaves in response to mechanical stimulation [14]; in such a way that the properties of the electrical signals generated can be studied [7], but a video turns out to be a better option because, although they lack muscle, plants have developed a remarkable variety of mechanisms to create movement. Video can show how the plant uses mechanical instability to accelerate its movements [15]. In the video you do not necessarily see what the eyes observe; Frames can reflect the effect of applying techniques such as the based on classifiers, neural networks, corner detectors, etc. [16] [17].
Many structural and functional properties that plants possess have great potential to stimulate new concepts and innovative ideas in the field of biomimetic engineering. Key from biology can be used to create efficient and optimized structures [5]. The study of plant movement has generated applications such as robots [18] [19], energy harvesting systems [20], development of simple teaching laboratories to illustrate the dynamic qualities of plant movement using smartphones [21]; the realization of tactile sensors inspired by the schismatic movement of plants [22]. The use of animation techniques and their algorithms can lead to a greater number of applications, especially in the area of artificial intelligence [23]. Therefore, the impact of the system with animation has potential aspects. Since the development of a system that animates the movement of the Mimosa pudica allows the analysis in time intervals.
The referenced articles (1 to 23) perform an analysis considering a mechanism that isolates the plant considering an image, such a situation avoids a method that allows reviewing the behavior of the plant in a sequence of time, and with the disadvantage of not being under a natural environment, this a problem is solved in this article through image processing techniques and the use of key points generated from methods such as Harris and Brisk presented in animation.
Another obvious problem is that the processing of the referenced works is applied to isolated plants, which affects the natural behavior of the plant. To avoid the problem, a way to isolate the problem is by using ROI (Region of Interest).
The main objective is to develop a procedure that allows generating animations from keypoints detected in videos of the mimosa pudica plant; The system first performs motion detection and subsequently performs a statistical analysis of the sequence of frames in an animation, without modifying the natural environment in which the Mimosa pudica is found.
The present work aims to carry out video processing applicable to plants with autonomous movement, through the digital treatment of frames of the Mimosa Pudica and to generate animations of the collected data, which allow its evolution to be visualized. In addition to producing a technique based on key points for the analysis of movements. In this way, 588 | P a g e www.ijacsa.thesai.org the hypothesis that the movement can be detected utilizing the correlation coefficient of frames with key points is supported.
The main motivation is the development of the system that will allow the intelligent behavior of plants to be analyzed in their natural environment, since it will permit the generation of systems based on the movements of the plants, starting with sensors, reaching the generation of artificial intelligence paradigms. based on the results delivered by the system or methodology presented.

A. Incremental Life Cycle
The research methodology used is based on the Incremental Life Cycle of a system. The model is built by increasing functionalities. It was carried out by modules that fulfill different functions within the investigation. The increments allow the capacities of a system to be gradually improved [24].
In the diagram of Fig. 1, the blocks of the procedure are observed. The pre-processing adapts the video frames through filters, enhancement of certain parameters, conversion to gray levels, etc; the part of the interest of the Mimosa Púdica plant is located through ROI (Region of Interest). Following the incremental model, key points are located, by the Harris or Brisk method and an analysis of key points or Landmarks is performed [25]. Once finished, the animation is generated on which the amount of movement is determined. Animations can be generated in various stages; which improves the behavior analysis of the techniques used.

B. Region of Interest
A region of interest (ROI) is a part of an image that you want to filter or operate on in some way. To determine the area you can use many shapes, such as circles, ellipses, polygons, rectangles; hand-drawn shapes were chosen. In the present project, it was used to create a binary over-face image [26].

C. Sobel
The Sobel operator handles a matrix where the central row or column of the filter is given greater weight [27]. Said matrix is defined in equation (1).

D. Canny
The method applies several filters in different directions and resolutions, which are combined. The method has three objectives: Minimize the number of false, Improve edge localization, deliver a border of one-pixel width. The technique is based on the gradient and the second derivative of Laplacian [27].

E. BRISK Corner Detector (Binary Robust Invariable Scalable Key Points)
The detector can be used to locate corners with multiple scales. The goal is to decompose an image into regions of local interest; which reduces complexity while exploiting appearance properties. The hotspot detector finds regions in the image that stands out even when the viewer of point of observation changes. See Fig. 2. BRISK has three phases, the detection of the characteristic, the composition of the descriptor, and finally the pairing of the key points [28].
The technique is applied during the sampling of the intensity of the image at the standard pi point, and a Gaussian smoothing with standard deviation σi proportional to the distance between the points of the respective circle [22]. Also, the pattern is positioned and scaled according to the key point k, in one of the (N (N-1)) / 2 pairs of sampled points pi, pj. The smoothed intensity values at these points are I (pi, σi) and I (pj, σj) respectively, and are used to estimate the gradient g (pi, pj). See equation 2.
The distance thresholds are δmax = 9.75t and δmin = 13.67t with a t(k) scale. Iterating through the even points in L, the direction over all the characteristics of the pattern of key point k is estimated, using equation 3.

F. Harris Detector
A corner can be interpreted as the junction of two edges, where one edge is a sudden change in the brightness of the image. The Harris algorithm detects corner points, regardless of rotation and change in gray level [29]. The response is used to select isolated corner pixels, and reduce the pixels at the edge. The Harris matrix is a symmetric matrix similar to a covariance matrix. The main diagonal is made up of the two averages of the square gradients "y". See equation 4. The offdiagonal elements are the averages of the cross product of the gradient 〈 〉.
First, the measure of the response of the corner R is considered. The contours of the constant R are shown by thin lines. R is positive in the corner region, negative in the border regions, and small in the flat region. The R values increase the contrast and magnitude of the response. The flat region is specified by Tr, which falls below some selected threshold. R can be determined by equation 5.
where k is a constant typically with a value of 0.04; R can also be expressed with gradients: So when the response is greater than a predefined threshold, a corner is detected. See equation (7).
In this way, a pixel in the corner region is selected (positive response) if its response is a local maximum of 8 axes. Pixels in the edge region are considered boundaries if their responses are local minima and negative in the x or y directions; depending on, if the magnitude of the first gradient is greater in the x or y-direction. The result is continuous thin edges that generally end in corner regions.

G. Animations
An environmental animation algorithm is a laboratory for the dynamic investigation of programs. Multiple graphs of an algorithm in action can be presented, exposing program properties that in other cases would be difficult to understand or notice.
To animate an algorithm, the frame of reference must be prepared. The framework works as a test, connecting the algorithm to an input generator, and monitoring both input and internal events, displaying the action in one or more views. A videogram is an image of an abstract state of the algorithm or its data, handled by events [30]. Frames show relevant parameters and variables, of the current state. An algorithm in the video becomes clearer. The idea is to help humans quickly get a clear picture of what is happening. Properties can be intuited that can be verified with more formal methods [31].
Steps to create a frame animation: 1) Run a simulation or generate data.

H. Correlation
The covariance of two random variables X and γ, with a joint probability density function f (x, y), is defined as: Where μ is the mean and E is the mathematical expectation. The correlation coefficient is given by: Where > 0 y > 0 Correlation is a measure of the linear relationship between two random variables. If the joint distribution of two variables has a correlation coefficient, then −1 ≤ , ≤ 1 . When , = 1, then X and γ are perfectly related positively. The conclusion is that the possible values of X and γ lie on a line with a positive slope. On the other hand, when , = −1 then the situation is opposite: X and γ are perfectly negatively correlated. If X and γ are independent, then , = 0 [32].

A. Active Contours vs. Region of Interest
One of the main problems for the analysis of the mimosa pudica is that is immersed within an environment whose image segmentation makes a problem is not solved; since if the segmentation is done by color, the algorithm is confused when finds similarities between the other sheets and backgrounds with a similar color. Thus, the first option that was considered was a semiautomatic segmentation. The original snake model of the active contour proposed by Kass [33], is represented as a parameterized curve v [s] = [x (s), y (s)], s ∈ [0, 1] that moves through a spatial domain and that seeks to minimize the following energy functional. * = ∫ ( ( )) 1 0 Active Contours seems like a good option, but the results show the opposite, as made known in Fig. 3 and 4. The situation is difficult to analyze in the image in Fig. 5 because www.ijacsa.thesai.org even for one person it is difficult to separate the branch to observe. In Fig. 4, the cut leaf can be detected since the selected area (in yellow line) is modified due to the algorithm of the snake. This fact impairs the analysis since part of the movement of the plant is truncated. The intrusion of other objects is again due to the similarity of patterns with the rest of the leaves and the surrounding environment, together with the fact that the leaf to be analyzed is moving. That is, the area of energy covered by the integral of equation (10) is modified, which is easily observable in animation with frames. For this reason is better to use a region of interest by freehand limiting.
The binary mask created allows us to isolate the region to be analyzed using ROI. The area remains throughout the cycle of movements of the plant. Therefore, the edges should be chosen with excess, but without going too far from the extremes to be analyzed. Trying to cover the entire movement. As seen in Fig. 5. In this way, the problem of analyzing the plant in its natural environment is solved without affecting the conditions that we wish to review. In Fig. 5 can be seen that the frame covers only the area to be analyzed without invading other regions, so this method gives better results than the snake algorithm. Fig. 5 shows the intermediate Frame of the animation sequence.
If the key point detection algorithms are applied to the pudic Mimosa without using ROI, the results shown in Tables I  and II are obtained. Several videos were processed with shots of the Mimosa; the results were similar to those presented in Tables I and II. Table I shows the Mimosa with the leaves practically horizontally. It is not possible to observe the details that happen over time in a single image of the added value of the animation. Using the Brisk algorithm, motion detection is easy, as more circles appear as the sheets are folded. Finally, the Harris algorithm detects corner points that change position, but they are not enough. The case presented in Table I is the worst-case evaluated. Table II shows a case where the Mimosa leaf is facing the camera. For the Brisk algorithm, although letter-shaped noise is detected, such fact is easy to observe the accumulation of detections at points where the sheet is folded. Finally, when applying the Harris algorithm, the stem and green points are detected at the tips of the leaves, reducing the distance between points as the movement evolves; Tables I and II show the utility of these methods used; however, a single image does not allow us to observe the behavior of the plant concerning the passage of time; therefore, an animation is essential. However, the video generation consumes several computing resources; therefore, only the key points should be used rather than the entire image.
The primary results would indicate that is feasible to analyze the movement of the plant just by using the key point detectors. However, is not the case, since the problem arises when the branch to be analyzed is found in clusters of mimosa; Separating the plant with a mechanism modifies the normal conditions and therefore changes the behavior of the plant. ROI is used to solve the problem. The results are shown in Table III.
Before the detection of key points, digital filters are applied and we have Fig. 4. The Sobel and Canny techniques improve the detection of key points, which is best observed with animation and that would be difficult to notice with a single image or with a set of them, especially if is considered that the conditions are changing over time. This fact is observed in Table III for the cases presented in the second 1, 2.5, and 4 when applying Brisk to the video.
Results are better when applying Canny Edge Detector. Table IV shows the results when applying the Harris detector with Canny. The second 1 is the case when the movement starts, the second 2.5 is the middle of the movement and the second 4 is the end of the movement. You can see the similarities and differences in the detected points.   (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 9, 2020 591 | P a g e www.ijacsa.thesai.org

B. Statistical and Correlation Analysis Brisk
The detection of key points by the Brisk technique was applied to videos of Mimosa Púdica. The first video frame to be analyzed is shown in Fig. 6. The Region of Interest exposed in Fig. 7 will be processed.
The area shown by Fig. 7 is chosen, and from this selection, the frames that form the animation are generated. Thanks to the generation of the animation are possible to perform the statistical analysis during the time interval in which the movement of the plant lasts.
Animation with Brisk key points was obtained; the final frame is shown in Fig. 8. A single 3-second video can generate more than 300 images. In the same way, as with the Harris method is observed that the amount of information to be processed allows the saving of computing resources since the whole image or its contours do not have to be processed, the key points are enough.     Table V presents the analysis of the graph. A value equal to 1 or -1 is never observed, so there is a linear dependence between images, but obviously, the movements are not sudden. The maximum value is 0.62 for frame 90. The minimum has a value of 0.31 because there is no movement.

Harris
The video processing was also performed by applying the Harris corner detector to a square area of the leaf frame of the Mimosa Púdica. As shown in Fig. 10. Only the contours generated by Canny are shown in Fig. 10. Fig. 11 shows the key points of the final frame. With frames of 400 x 400 pixels. The amount of information processed is much less than if the original image were processed, so computing resources are saved; Since we have the position of the key points, the statistics of the time interval that the video lasts are taken.  (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 9, 2020 593 | P a g e www.ijacsa.thesai.org The graph in Fig. 12 shows the basic statistic points between frames. As there is a greater number of Mimosa pudica plants, more chaotic movements are shown where the correlation coefficient can range from -0.14 to 0.28 as shown in Table VI.  IV. CONCLUSIONS The processing of an image of a moving plant is inadequate, for this reason, digital video processing must be incorporated, and the generation of animations facilitate the review of the behavior of an algorithm over time, solving the problem of having a large magnitude of time-consuming images compared to animation. An analysis was proposed employing a non-autonomous segmentation (Region of interest) that avoids that when the conditions change over time, the segmentation of the information is out of control. The referenced articles isolate the plant from its natural environment, with the proposal it is feasible to carry out the analysis within the natural environment of the plant without using mechanisms or systems that affect the behavior or the analysis.
We comply to develop a procedure that allows generating animations from Harris or Brisk key points of the pudic mimosa plant, where motion detection was first achieved, and later a statistical analysis of the sequence of frames in animation is performed. The use of key points as data to be analyzed is another contribution, since it greatly reduces the computation used, unlike the referenced articles.
As for the key point detection algorithms, we can take the information and perform simulations and/or animations based on the movement of the plants. The ROI application manages to reduce the amount of information to be analyzed and improves the processing speed of landmarks.
Regarding the analysis carried out as a research tool, the application of the detection of key points can yield interesting results that can lead to applications of biomimetic impact, the development of sensors, and the use of movement mechanics for the generation of small energy sources.
The correlation coefficient presents us with an alternative for the detection and analysis of the movement of the Mimosa Púdica. As the correlation coefficient increases, the amount of movement decreases.
Regarding animations, the relative execution of different algorithms becomes clearer and improves their potential. The idea is to help humans quickly acquire a clear picture of what is happening. Due to the exploration of the algorithm in a visual way, you can intuit properties that are verified with methods more formal.
As future work, using the implemented algorithms, the behavior of the Mimosa pudica plants will be observed for the generation of paradigms in artificial intelligence based on the behavior of the plants.