Overlapped Apple Fruit Yield Estimation using Pixel Classification and Hough Transform

Researchers proposed various visual based methods for estimating the fruit quantity and performing qualitative analysis, they used ariel and ground vehicles to capture the fruit images in orchards. Fruit yield estimation is a challenging task with environmental noise such as illumination changes, color variation, overlapped fruits, cluttered environment, and branches or leaves shading. In this paper, we proposed a learning free fast visual based method to correctly count the apple fruits tightly overlapped in a complex outdoor orchard environment. We first carefully build the color based HS model to perform the color based segmentation. This step extracts the apple fruits from the complex orchard background and produces the blobs representing apples along with the additional noisy regions. We used the fine tuned morphological operators to refine the blobs received from the previous step and remove the noisy regions followed by the Gaussian smoothing. Finally we treated the circular shaped blobs with Hough Transform algorithm to calculate the center coordinates of each apple edge and the method correctly locates the apples in the images. The results ensures the proposed algorithm successfully detects and count apple fruits in the images captured from apple orchard and outperforms the standard state of the art contoured based method. Keywords—Apple detection; pixel classification; curvature estimation; Hough circle transform; visual tracking; color segmentation


I. INTRODUCTION
The frequently used typical fruit yield estimation methods are usually based on historical data, weather conditions and manually sampling statistics.However these methods are all time-consuming, requires huge human resource and their prediction results are not accurate enough.The autonomous and accurate visual based fruit yield estimation can help farmers to improve fruit quality through reasonable pruning, designing planting and harvest plan.
Visual data such as images are the good source to analyze and monitor the growth of apple fruit in the orchard.Images are the supporting technology to study the apple and apple tree growth rhythm quantitatively but it lacks the automation process such as fruit count and size analysis.
Authors used various computer vision and machine learning techniques to estimate the yield estimation of different fruits.In this paper our concern is on quantitative analysis of the tightly overlapped apple fruit in the complex cluttered orchard environment.We propose a machine learning free novel visual based method to count the overlapped apple fruits.Next in this section, we discuss the existing methods that estimates the fruit yield estimation.

A. Color based Segmentation
Wang et al. [1] used two cameras stereo rigs for image acquisition, and showed the system worked smoothly with red and green apples both.The proposed system used with controlled artificial lighting at night time, and the software has a restriction during dealing with fruit clusters composed of more than two apples.
Payne et al. [2] proposed an approach to calculate the mango fruits from daytime images on individual tree.The method segmented the pixels into the fruit area and background by applying color segmentation using RGB and YCbCr color spaces with threshold methods.The specific connectivity regions used to count the number of fruits.However the method did not consider the circumstances of overlapping and covering.
Zhou et al. [3] proposed an apple fruit recognition algorithm based on color features computed on the difference of R-B (red minus blue) and the G-R (green minus red) to estimate the fruits quantity.The method did not well adapt to the changes of illumination and shading problems among fruits, branches and leaves.Nuske et al. [4] presented an autonomous method to detect and count grape berries.They used both shape and visual texture features to detect berry and demonstrated segmentation of green berries from green leaves.The method used radial symmetry transform and led to a large amount of arithmetic operation.
Hung et al. [5] proposed the multi-class image segmentation method to automate fruit segmentation without pre-defined features descriptor.A feature learning algorithm joined with a conditional random field and was used to process multispectral image data.
Various authors worked on crop yield assessment by using digital camera acquired images and practiced for crops such as wheat [6], wild blueberry [7] and rice [8].
The discussed visual methods did not consider the circumstances of overlapping and cluttered environment and did not incorporate the concept of objects so that it could not provide the actual fruit count.

B. Feature Classification
Lak et al. [9] developed an algorithm based on color-shape and edge detection to segments the red apple from images obtained under the different natural lighting.The method filtered the images, converted them to binary and reduced the noise.Color-shape supported algorithm detects the apple fruits in the image while edge detection supported algorithm was not successful to segment the red apples.
Moonrinta et al. [10] build a framework procedure that is based on image processing methods for detection and tracking of the pineapple fruit along with 3D reconstruction.They employed scale invariant SIFT and SURF descriptor with SVM learning and carried series of experiments to receive the pineapple feature classification, fruit blob tracking, 3D reconstruction, structure from motion and ellipse estimation.They obtained that the SURF feature points and descriptors give the finest trade-off among classification accuracy and processing time and the technique adequately effective for fruit region detection.

C. Learning based Methods
Seng et al. [11] used k-nearest neighbors (k-NN) to detect apple fruits.In first step, the authors detected apple pixels using fruit texture and color.The k-NN classifier produced pixels association towards "apple" along with "non-apple" items.In the second step they detected apples and estimated the growth of apple surface by its seed area.Additionally, they used (blobs) linked sets of apple pixels to detect the areas of apples and wrap up region of an apple to the concern.In third step they carried apple detection and contour segmentation by analysing the contour of every blobs.
Unay et al. [12] introduced a method for apple fruit detection and its quality classification with multilayer perceptron (MLP) neural networks.The primary examination of the quality classification system used for "Golden Delicious" and "Jonagold" apples.Next they selected the texture, colour and wavelet features from the apple images.Principal components analysis (PCA) was utilized on a selected features and carried some introductory performance tests with single and multilayer perceptions (MLP).
Tabb et al. [13] proposed a technique for the segmentation of apple fruit from video using background modeling.They used global mixture of Gaussian (GMOG) that worked on the principles of mixture of Gaussian (MOG) for motion detection.Gaussian mixture models (GMMs) are the significantly developed techniques for clustering.A Gaussian mixture model is probabilistic model which assume all the data points that are initiated from a mixture of finite number of Gaussian distributions with unspecified parameters.Dubey et al. [14] divided color images using soft computing techniques.The soft computing techniques they used are possibilistic c-means (PCM) algorithm and competitive neural network.Fuzzy logic and Fuzzy set techniques also used by researchers and investigators for resolving segmentation issues.
The learning methods never discussed, how to segregate the overlapped fruit regions and count them correctly, the methods may be good in detecting the fruit regions.

D. Overlapped Fruit Estimation
Xu et al. [15] proposed a technique for strawberry detection using image sensor mounted on strawberry harvesting robot.The technique laid on the histogram of oriented gradients (HOG) descriptor, that combined with support vector machine (SVM) classifier.The detection method concerns with two stages.First, the strawberry-similar regions detected from HSV (hue, saturation, value) color information.They calculated the HOG descriptor by dividing the image into five region of interest (ROI), later the descriptor is an input to the SVM classifier that detects the strawberries.The vector sizes efficiently reduced and higher detection speed attained without effecting the accuracy (relative to conventional approaches).The technique also appropriately handle slightly overlapping strawberries.
The state of the art the contour based [16] appearance analysis and object categorization is also unable to detect the overlapped region of the fruits accurately.
All the above methods either require a complex training or lacking to analyze the curvature of fruit to accurately count the tightly overlapped fruits.
In this paper we took a step towards curvature analysis and pixel based segmentation and propose a fast, novel and learning free model to segment the tightly overlapped apple fruits from the background and count them in real time.We combined the pixel classification for fruit segmentation with curvature analysis using Hough circle transform to segregate the overlapped fruits and count them correctly.
Our method is robust in illumination changes, occluded by leaves and branches, overlapped by other apples, complex background and other issues under natural light and field conditions, see Fig.

II. PROPOSED METHODOLOGY
The proposed methodology first apply image preprocessing steps to the image inclusive of noise reduction and image enhancement.Later, we employ the pixel segmentation based on color to subtract the background and detect the region of interest with the apple fruits, later the image is reprocessed by applying morphological operators and Gaussian smoothing to refine the image.Finally the method estimates the centroid of the circular regions using Hough circle transform technique and counts the apple.
The block diagram gives a review of the proposed method with various steps in a consecutive order, see Fig. 2. We describe the details of each step in the succeeding sub-sections.

A. Pre-processing and Color Space Generation
The preprocessing of image aims at selectively removing the redundancy present in captured images without affecting the details that play a key role in the overall process.
In proposed method, we first read the image in RGB (red, green, blue) format.Reading the image in color format is important to perform the pixel based segmentation in the later phase.Next, we apply the Gaussian method to smooth the image and suppress the noise.
After the preprocessing, we convert the image into HSV (hue, saturation and value) color space to segment the image.HSV describes the color in the hue (color), saturation (vibrancy) and value (brightness).The reason to use HSV for the segmentation because of it robustness towards illumination and shadow removal that separates the color information from intensity and it describes the color as the human eye perceive the color [17].
We remove the (V) element from the HSV color space as it has no effect in the color segmentation and it possesses strong response to light variations.We carry the further processing in later phases on HS image.RGB to HSV conversion formula: Hue, Saturation and Value calculations:

B. Pixel Classification
The apple fruits grow on bunches, overlapped and surrounded by the green leaves of the tree in orchard.Pixel classification is an important phase to segment the image and extract fruits such as apple fruits in our case.
After the conversion to HSV, we define the lower and higher range of the apple color to threshold the image.lower_red = (0, 23, 30) upper_red = (23, 255, 255) Once we generate a desired color range, we apply it to the HS image to extract the apple regions from the image and throw the unwanted pixels.This process returns the binary image where 1 indicates the white color regions and shows apple blobs where 0 values show black color and unprocessed by the algorithm in later phases.
Next we use this binary image as a mask and apply it over original image to segment the image and extract the desired fruit regions.
After the pixel classification, we segment the image and the apple fruits are correctly extracted from the cluttered background, see Fig. 3.The figure concludes that apple color identifying model effectively removes the background pixels shown in black color and segment the apple pixels out.In the later phases, we continue to use the binary image produced by the color classification method.

C. Reprocess the Blobs
After the image segmentation using color information, the output binary image contains the smaller noisy blobs and some open blobs generated by occlusion and the cluttered environment, see Fig. 4. The morphological operators are the best choice to get rid of these noisy blobs and restructure the existing blobs.It deals with modifying the geometric structure in the image and refines the segmented image by smoothing the object boundaries by filling small holes and eliminating small holes.We used the two fundamental morphological operators erode and dilate to better structure the blobs.We first apply the erode to the binary image to remove the smaller noisy blobs followed by the dilation to fill or close the open blobs in the image.After applying the morphological operators the output image is left with bigger and smooth blobs.We expect the output blobs would nearly be circular in nature and overlapped over each other, see Fig. 5.
Finally in this phase, we smooth the image again using the Gaussian filter to better apply the curvature analysis algorithm in the later phase.

D. Curvature Estimation with Hough Circle Transform
The contour of a perfect apple can be well represented as a circle along the radius within a pre-defined range [18].Because of individual differences and the impact of the position, we usually found the variable apple shapes in images.However, in general, the shape of an apple has high similarity with roundness and we receive the apple partial round contour from different view points.
The second genuine challenge is the overlapping and shading that makes the shape of an apple in the image as an incomplete circle.We left with the incomplete overlapped circular objects and determined the circular curve as the standard apple shape.
We found the Hough circles is the best solution for the aforementioned challenges and count the partial overlapped regions of the apple fruit.The better tuned Hough circles can better identify the circular regions and help counting the apples.
After applying the morphological operators, we received the overlapped blobs from the previous step.In this phase, we apply the Hough circles to detect the overlapped and partial round shape objects in the image.The method successfully identifies the circular regions and help counting the apple accurately, see Fig.   1) Hough circle explained: Hough transform is the shape positioning technique in image analysis.The aim of using this technique was to obtain inaccurate instances of objects within certain class of shapes by a voting mechanism.This voting mechanism carried out in a parameter space, from this space the object candidates achieved when the local maxima in this space was found.With respect to the template matching, fewer computing resources were required by the Hough transform [19].
The parameter space and the voting procedure are defined by Equation (3).
In which, the point of (x 0 , y 0 ) represents the center of circle, and r represents the radius of a circle.For a given range of r, the center of each circle can be detected through circle Hough transform (CHT) algorithm.Accordingly the circle number can be calculated by the accumulation of point of (x 0 , y 0 ).

E. Proposed Algorithm Steps
Steps that have been carried out for this study, they are: 1) Obtain an input apple image in RGB color space.
2) Convert the input image into the HSV color space, remove the value (V) element and left with HS color image.3) Apply the pixel classification to remove the unwanted image regions (leaves, branches, stems) or pixels from the image.4) Reprocess the image using morphological operators, erode to remove the smaller blobs and dilate to close the open blobs, and received the true circular blobs.5) We smooth the image again with Gaussian filter.6) We apply the Hough circle transform for the curvature estimation to circle the partial round shapes.7) Finally output the total count of apples in the image.

III. EXPERIMENTS AND RESULTS
In order to examine the robustness of the proposed approach, we have selected 36 images of apple fruits from different orchard with varying illumination, color and cluttered background.The images includes overlapped apples occluding each other.
We divided the experiments into two categories, in experiment I, we applied the proposed method on the various images and showed the successful counting of apple fruits.In Experiment II, we carried the quantitative analysis of the proposed method.

A. Experiment I
In Experiment I, we picked various images with the overlapped apple fruits and test the robustness of the proposed method in different environments.
We divided the experimental images into two sets.For each set of images, we showed the results of the four key phases of the proposed method and excluded the minor steps that includes image processing techniques.The four phases are generating HS image, color based segmentation, blob smoothing and restructuring with morphological operations and counting apples using Hough circle transform.The division of images in two sets of images are only for the sake of clarity and better understanding of the proposed method.The ground truth for the first row image frame contains five apples, we first generate the HS model from the original image and later apply the color based segmentation correctly.The color based segmentation generates the blobs in binary image format along with the noisy blobs.The noise are the blobs that do not contribute in the apple shape, these blobs are either tiny or broken.We use the Morphological operators in the next step to remove the noisy blobs and generate the well refined and structured blobs.We first applied the erode to remove the tiny blobs followed by dilate to fix the broken blobs.The fine tuned morphological operator generates well structured and shaped blobs, see Fig. 7-(c) in first row.Finally the last image in first row shows the true apples circled in green by the Hough circle transform.The proposed method correctly circled and counted the overlapped all five apples in the cluttered environment, see first row of Fig. 7.
The second row of the first set of image consists of ten apples with more complex environment than first image.The HS model generated from the original image followed by application of color based segmentation with accurateness, the step also generated the smaller noisy blobs, see Fig. 7-(b) in second row.The third image in the second row shows the well refined blobs generated after applying the morphological operators erode and dilate.In final phase, we used Hough transform to count and encircle the apples in green.The proposed method correctly circled the nine apples, see Fig. 7-(d) in second row.
The third row of first image frame comprises of six apples, We successfully show the result of all four key phases and the method detects and counts all the genuine apples circled in green by the Hough transform.The proposed method correctly circle the overlapping five apples and missed one apple, the method also detected one wrong apple, see Fig. 7-(d) in third row.
2) Image set II: The second group of images consist of two images taken from various frames of the video stream, see Fig. 8.This image set holds more complex environment and many apples than the image set I. The first image frame comprises of thirteen apples encompassing the generation of HS model from the original image to later administer the color based segmentation accurately with the smaller noisy blobs, see Fig. 7-(b) in first row.The third image in the first row showed the well refined blobs after applying the morphological operators erode and dilate.The fine tuned morphological operators generated well structured and shaped blobs, see Fig. 7-(c) in first row.Finally the last image in first row shows the true apples circled in green by the Hough circle transform.The proposed method correctly estimate the twelve apples and counted the overlapped apples in cluttered environment, see Fig. 7-(d) in first row.There is little chance for the apple detecting algorithm to generate false detecting results as there is a range for the parameter of radius, not all circles can be recognized if the apples stayed too far away and turned too small in the image.
The second image frame includes twelve apples that involves the production of HS image from original image followed by the application of color based segmentation with accurateness, see Fig. 7-(b) in second row.After applying the morphological operators erode and dilate, well refined and fine tuned blobs produced, see Fig. 7-(c) in second row.The proposed method Hough transform correctly circled twelve apples and counted the overlapping and packed apples, see Fig. 7-(d) in second row.

B. Experiment II
We also carried the quantitative analysis of the proposed method and compared with the state of the art contoured based color based classification.
In this experiment, we took a video of 5 various frames from a video stream of an apple orchard.The ground truth consists of total 46 apples.The proposed method detected 43 apples correctly with the true positive (TP) rate of 93.48%, whereas the state of the art contour based method detected only 19 apples with the true positive rate of 41.31%.Our proposed curvature estimation dramatically beat the contour based method with high margin.
The contour based method is only good in drawing the boundaries of the detected object, the method is unable to detect the overlapping regions and count them correctly, especially the overlapping fruits in our case, it detected only 19 apples.The method return only one circle for the four overlapping fruits and counted them as one apple and show the results, see Table I.
We also consider the rate of miss classification that is the detection of the wrong objects instead of apples, we name it false positive (FP).Our method false rate is 4.35% where as the contour based method is 2.17%.It is clear when we apply the curvature estimation we may find some wrong circles whereas the contour does not depend on such estimation which shows a slightly better in the false positive case.Table I shows that our method is capable of detecting and counting the apple fruits with more accuracy as compared to the contour based method.

IV. CONCLUSION
This paper introduced the concept of circular Hough transform to estimate the curvature to find the apple fruit yield estimation.It is typically preferred technique for circular object detection.The proposed method apple fruit yield estimation and counting apples comes to the following conclusions: 1) We proposed a machine learning free pixel classification and curvature analysis algorithm for tightly overlapped apple fruits count.The method is fast, robust and can be used with the small unmanned ground and ariel vehicles in real time.
2) The algorithm proposed in this study showed robustness in detecting and counting apples from apple tree images.Since it could deal with the apple recognition problems such as illumination changes, shaded by leaves and branches, overlaps with other apples, and complex cluttered background.
3) The proposed algorithm made full use of the vision features including targets color and shape to detect and count apples on the trees.The algorithm with strong noise resistance succeeded since the pixel classification could fit complex function and circle Hough transform provided the incomplete shape detection specialty.4) The proposed method had strong generalization capability.The algorithm could be extended to the detection of other types of round fruits by re-training the color identifying model by pixel classification.5) The proposed approach adapted to the natural conditions and could be used to detect fruits and count number in tree images, and it could also be used as a core detection algorithm in orchard yield estimation system to provide guidance for the management of the orchard.
Although unavoidable problems were analyzed in detecting and counting apples from apple tree images.In future work, we plan to address more accurate methods to estimate the curvature and advanced deep learning methods and comparison with them.

Fig. 4 .
Fig. 4. Noisy smaller blobs and open blobs generated by occluded fruit and cluttered environment.

Fig. 5 .
Fig. 5. Reprocessing the blobs.(a).Morphological operator erode applied to eliminate the smaller blobs.(b).Morphological operator dilate applied to fill the open blobs.image.

Fig. 6 .
Fig. 6.Counting apples using Hough circle transform.(a).Apple fruits blob received from previous step.(b).Hough circle applied and correctly estimate the apples.The green circle shows the count of apples.

TABLE I .
APPLE DETECTION AND ESTIMATION RESULTS OF THE PROPOSED SCHEME AND COMPARISON WITH THE STATE OF THE ART CONTOURED BASED METHOD.