Unified Approach for White Blood Cell Segmentation, Feature Extraction, and Counting using Max-Tree Data Structure

Accurate identification and counting of White Blood Cells (WBCs) from microscopy blood cell images are vital for several blood-related disease diagnoses such as leukemia. The inevitability of automated cell image analysis in medical diagnosis results in a plethora of research for the last few decades. Microscopic blood cell image analysis involves three major steps: cell segmentation, classification, and counting. Several techniques have been employed separately to solve these three problems. In this paper, a simple unified model is proposed for White Blood Cell segmentation, feature extraction for classification, and counting with connected mathematical morphological operators implemented using the max-tree data structure. Max-tree creates a hierarchical representation of connected components of all possible gray levels present in an image in such a way that the root holds the connected components comprise of pixels with the lowest intensity value and the connected components comprise of pixels with the highest intensity value are in the leaves. Any associated attributes such as the size or shape of each connected component can be efficiently calculated on the fly and stored in this data structure. Utilizing this knowledge-rich data structure, we obtain a better segmentation of the cells that preserves the morphology of the cells and consequently obtain better accuracy in cell counting. Keywords—Segmentation; feature extraction; White Blood Cell (WBC); mathematical morphology; max-tree


I. INTRODUCTION
Microscopic blood cell image analysis is crucial for the diagnosis of several blood-related diseases. It may require complete blood count (CBC) where a complete count of red blood cells, white blood cells, and platelets is investigated. In some cases, differential blood count (DBC) may be required where five different types of white blood cells: eosinophils, basophils, monocytes, lymphocytes, and neutrophils need to be separated and counted. Blood image analysis is also crucial in the diagnosis of leukemia where lymphoblasts are needed to be separated from the healthy WBCs and counted. Manual analysis by the human experts is time-consuming, the accuracy of the result vastly depends on the expert's capability, and varying results may be obtained even if the procedure is repeated by the same expert. Thus, image-based analysis of blood cells gained much popularity in the past decades.
For classification different features such as morphological and textural features have been used [15]- [17]. Few others used genetic features extracted with the genetic algorithm [18]- [20].
Finally, different types of classified cells need to be counted. For counting, some methods that require prior cell segmentation and detection [21], few others approximated the number of cells from estimated density obtained from user annotation by compromising accuracy over speed [22], [23]. A method of cell counting based on morphological image analysis of blood cell images without requiring user annotation is reported in [24].
From segmentation to the counting of the cells widely varying techniques have been utilized in the literature. There is no unified approach that can facilitate in all three analysis steps of segmentation, feature extraction for classification, and counting. Inspired by the work in [24], this paper tries to use the full potential of Max-tree data structure which is an efficient structure for morphological connected operators. Morphological connected operators work on connected components of a gray level image known as flat zones and preserve only those flat zones that satisfy given criteria removing the rest of the flat zones [25], [26]. The criteria can be based on one or more attributes computed from the flat zones. Max-tree data structure enables the processing steps of these operators efficiently. Max-tree is a structured representation of an image where connected components with the highest intensity are in the leaves of the tree, the connected component with the lowest intensity is in the root, and the rest of the nodes hold the connected components for all threshold levels present in the image. Besides, the nodes of the tree are capable of storing a plethora of knowledge such as size and shape granulometry, texture, moment, or motion-oriented attributes. In this paper, the capability of this knowledge-rich data structure has been utilized for cell segmentation, feature extraction, and counting. The cell counting part of the work is also reported in a conference paper [24]. In [24] only the structured representation of the image by the tree is utilized where the number of leaves in the tree is reported as the number of cells present in the image.
Piuri and Scotti proposed a system for leukocytes detection and classification based on the morphological operators in [27]. However, they used the structuring element based morphological operators for the identification of the cells. They only reported observational performance instead of any quantitative performance measure of their work. They also did not solve the cell counting problem. Besides, the structuring element based filtering is known to distort the original shape of the object. On the other hand, shape of the object is preserved in our method and thus yield better segmentation.
Moshavash et al. proposed a color-based cell segmentation technique where they converted the RGB images into CMYK color space and separate the background and red blood cells using the M-channel [28]. After the background and red blood cells are separated remaining are considered as the candidate for the WBCs.
On the other hand, this paper utilized connected morphological operators that filters without using any structuring element; rather it uses the structure of the input signal itself for filtering. Connected operators do not introduce any distortion or new structures to the resultant image. In this paper, a Max-tree data structure is used where connected components of the image are hierarchically stored thus every connected component is reachable for further processing. The main strengths of the proposed model are • The method obtain a better segmentation of the cells that preserves the morphology of the different types of WBCs and their nuclei.
• The Method achieve better accuracy in counting cells.
• Feature extraction can be done on the fly.
• Segmentation, feature calculation, and counting are done in a unified model where Maxtree data structure plays the central role.
• Maxtree representation can be utilized in other applications such as cell tracking, cell visualization, etc.
• It is conceptually simple.
The rest of the paper is organized as follows: in Section 2, the concept of mathematical morphology and connected attribute filters are described. In Section 3, the basic idea, implementation detail of max-tree data structure, and attribute estimation methods are discussed. Section 4 contains a detailed discussion of the proposed method. A brief description of the data set can be found in Section 5. In Section 6, experiments and results are reported. Section 7 concludes the paper with a few directions for future work.

II. MATHEMATICAL MORPHOLOGY AND CONNECTED ATTRIBUTE FILTERS
Mathematical Morphology [29] is popularly used in digital image analysis consists of several operators based on topological and geometrical concepts such as size, shape, contrast, etc. In mathematical morphology, grayscale images are considered in a form f (x) : Z → R where Z ⊆ E mapping Euclidean space or grid, E into R. Image, f (x) is interacted with a small set called structuring elements, s(x) utilizing the order relation on R. Connected attribute filters are morphological operators that can eliminate or merge the connected components or flat zones of an image where the image signal is constant [30]. Its power of simplifying images without distorting the contours makes it popular for various applications.
The notion of connectivity in digital images can be defined as the local neighborhood of pixels. If a 2D image, I, is mapped into m × n grid and the position of a pixel, p in the grid is defined as row and column pair (i, j). The four pairs of pixels positioned in (i ± 1, j) and (i, j ± 1) are 4-neighbors of p. A common choice of the local neighborhood of a pixel in the case of 2D images is 4 or 8 adjacency; and in the case of 3D is 6, 18, or 26 adjacency.
Connected operators act as a filtering tool on gray-level images that eliminates some of the connected components leaving other components unchanged. If I is the original image and S k is the structuring element of size k, the opening of I can be defined using equation 1 which is erosion ( ) followed by dilation (δ): where c = structuring element that defines the connectivity The operation in equation 1 will mark the connected components that need to be preserved and iterating equation 2 until idempotence will provide the desired result. Different types of connected operators can be obtained by the composition of any family of openings and closings by reconstruction. Connected attribute operators such as attribute openings, closing, thickenings, and thinnings can be utilized to filter connected components based on their attributes such as size, shape, contrast, etc. Simplest size oriented connected operators can be obtained by area opening and closing which is idempotent, anti-extensive, and increasing [31]. A large number of connected operators based on size attributes such as the moment of inertia, diagonal length of smallest enclosing box, etc. can be obtained for image filtering. The binary area opening,Γ of a binary image, I at point x with threshold parameter, λ obtains the connected component with an area greater or equal to λ and to which x belongs: Apart from filtering images using size-based connected operators, shape-based filtering can be implemented using attribute thinning and thickening which is antiextensive, idempotent, and scale-invariant [32]. Being scale-invariant shapebased operators are insensitive to the size of the structures. Several shape-based attributes such as elongation [33], complexity or simplicity, motion estimation, entropy, etc. [34] can be derived using these connected operators. Binary attribute thinning can be defined in terms of binary connected openings. The trivial thinning T C with criterion C of a connected set S is the set that satisfies C, or empty otherwise. Thus T C of set x with criterion C can be obtained by To decompose an image, I, according to size or shape, the image needs to be filtered using Γ λ (I) or T C (I) and after filtering the resulting image, I r will contain structures that meet the criteria λ or C. The difference image, I − I r should contain the structures that fail to meet the desired criteria. Derivation of these size and shape based operators for grayscale images is straightforward and can be obtained from their binary counterpart.

III. IMPLEMENTATION OF CONNECTED OPERATORS
Implementing connected operators using a structuring element for more than one dimension is difficult [35]. There are several algorithms proposed for attribute opening and closing such as the Pixel-Queue algorithm [32], the Union-Find algorithm [36], Max-Tree algorithm [34]. In [37] Meijster and Wilkinson discussed the pros and cons of these algorithms. The method proposed in this paper considered the findings of [37] and uses the Max-Tree algorithm because it requires liner time in both the number of pixels and connectivity for processing and pruning the tree and also for creating output images. Additionally, it also can be used for thinning and thickening.

A. Max-Tree
Max-Tree is a proficient data structure introduced by Salembier et al. in [34] for connected attribute filtering of the images. Max-tree being a rooted tree provides a hierarchy of flat zones with ordering relationships for extracting and filtering of the connected components by the operators. To describe the Max-Tree description of a few related terms is required. A connected component or flat zone, F li at gray level l of an image I constitutes a set of pixels p ∈ E|I(p) = l; a regional maxima, R lj corresponds to a flat zone at level l of which gray values of neighboring pixels are smaller than l; a peak component, P l k is a flat zone of the thresholded image, T l (I) at level l. In these definitions, i, j, k indicate the index of several such components. In the max-tree representation of an image, each node N l k holds only those pixels of the peak component P l k which have a gray value of l. The node N l k also contains attributes such as area of P l k . The node N 00 is the root node of the tree and contains the flat zones with the lowest gray value and the hierarchical structure of the tree ensures that all of the flat zones of the highest gray value can be found in the leaf-nodes. After the max-tree representation of an image is obtained filtering of the image based on any attribute can be done by pruning the tree in an appropriate branch where all nodes have attributes such as area smaller than the threshold. Fig. 1 depicts a pictorial description of the max-tree representation of an image containing six cells. The image has been binarized first then a Gaussian kernel is applied to obtain a smoothed image that will produce flat zones with various gray levels such as figure 1(c)). In Fig. 1(d) max-tree representation of the flat zones is shown where the root A0 represents the background of the image which is the flat zone at gray level G 0 . B1, C1, and D1 are the three children of the root that correspond to the three flat-zones at gray level G 1 . Among them, B1 has three flat zones as its children and D1 has two flat zones as its children. All the flat zones are stored in the tree preserving the parent-child hierarchy and the leaves B04, B14, B24, C04, D04,, and D14 holds the connected components of the highest gray level G 4 .

1) Max Tree Creation:
The process of Maxtree creation can be described using the original work of Salembier et al. [25] where recursive flooding is utilized to build the tree. A hierarchical FIFO (first in first out) queue is implemented for each gray level value for the appropriate scanning and processing order of the pixels. The nodes, N Gi (represents all nodes at gray level, G for all available gray levels, G) and the links between parent and child nodes are established by storing all local background pixels of gray level, G, to the parent node and the child nodes get the pixels at each connected component with a gray level higher than G.
2) Attribute Estimation and Filtering Approaches: Different attributes such as size, shape, the moment of inertia, etc. of each node of max-tree can be calculated on the fly. During the max-tree creation required attributes of each node (for each connected component) is also calculated.
In max-tree implementation whenever a pixel is added to a node, an associated variable is increased to obtain the area of the component in terms of the number of pixels. Similarly, the ratio of the moment of inertia, M I to the square of the area, A i.e., M I /A 2 is easily and accurately calculated as a shape attribute.
Later, during the filtering phase based on the threshold value of a specific attribute the algorithm decides to remove or keep the node. Different decision criteria for filtering are described in [25] such as Min, Max, Viterbi, Direct, etc. Urbac et al. proposed another decision criterion: the subtractive decision rule in [33]. According to Min decision rule, a node, N li is removed if the criterion value such as area, perimeter, or moment of inertia, etc., of that node, is less than the set threshold value or any of the ancestors of N li is removed. Max decision rule removes a node, N li along with its descendants if the criterion value is less than the set threshold value. Viterbi solves the filtering process as an optimization problem. The direct rule removes a node, N li if its criterion value does not meet the threshold value and assigns the pixels of that node to a gray value of its highest ancestor which meets the threshold value while keeping the descendants unchanged. The subtractive decision rule is the same as the direct rule but it changes the gray value of the descendants by the same amount as the node, N li .

IV. PROPOSED METHOD
This paper proposed a method of blood cell image segmentation, feature extraction, and counting using connected morphological operators implemented using the max-tree data structure. The work is based on the principle of mathematical morphology and connected attribute filters described in the previous section. Segmentation of the WBCs in the blood smear images has been done by pruning the max-tree that has been created as a representation of the image, based on area as a size attribute taking advantage of the distinct size of the RBCs and WBCs. Other attributes such as shape, moments, etc. that are computed on the fly during max-tree creation contributed to the feature vector to be used for the classification of the cells.

A. Morphology of the Blood Cells
The morphology of the blood cells plays a vital role in different steps of this work. There are three different major cells present in blood namely Red Blood Cell (RBC) or Erythrocytes, White Blood Cell (WBC) or Leukocytes, and Platelets (Thrombocytes). RBCs are generally smaller in size approximately 7µm − 8µm and has a very thin cell membrane that allows easy oxygen diffusion. The morphology of an RBC is similar to a torus without a whole inside and does not contain any nucleus. On the other hand, WBCs are larger in size of approximately 15µm − 20µm and contain a large nucleus that occupies most of the cell area. The platelets are the smallest in size approximately 2µm − 3µm in diameter with a very irregular shape. In this work, the separation of RBCs and WBCs is discussed where the platelets are considered to the RBC group.

B. Preprocessing
Microscopic blood-stained image datasets are usually with high-quality images because of their exclusive and careful acquisition processes. Due to this assumption, there are no noise removal steps involve in this work. However, this step can be introduced if needed and several filtering approaches can be utilized.
The preprocessing step of our work is greatly influenced by the staining process of peripheral blood smear images. Staining blood cells with different colorants is a common practice in peripheral blood smear images so that various components especially the white and red blood cells can be examined microscopically. Typical components of the stains are oxidized methylene blue, azure B, and eosin Y colorants [38]. The methylene blue and azure B stains the nucleus of cells with different shades of blue to purple color and eosin Y colorants stains the cytoplasm of cells an orange to pink shades [39]. Thus, after staining torus like part of an RBC get a nonuniform shade of pink where intense pink in some areas and pale shade in some other areas. The central part of the cell gets brighter intensity. The nucleus of the WBC is intensely stained with blue color and the cytoplasm is stained with different shades of blue.
To separate the WBCs from other blood cells the intense staining of its nucleus is utilized. Otsu's [40] method for thresholding is applied to the images and the nucleus of the WBC being intensely stained resulted into larger connected components and the RBC turned in to a combination of several small connected components almost the size of the dot because of the non-uniform staining [see Fig. 2(c)]. After binarization, a Gaussian kernel is applied to obtain a smoothed image [ Fig. 2(d)]. This is done to minimize the noise introduced by the binarization process and to obtain a continuous grayscale image.

C. Cell and Nucleus Segmentation through Attribute Filtering
The block diagram in Fig. 3 summarizes the proposed method. The proposed method concentrates on segmenting WBCs, calculating features for the feature vector to be used in the classification of different WBCs, and counting the WBCs.
RBCs and WBCs can be distinctly identified by their size and presence of the nucleus. In the preprocessing step, WBCs due to the presence of a nucleus and its intense staining resulted in connected components with the largest area, the second-largest area is the area between the nucleus and the cell membrane i.e., the cytoplasm, and rest of the blood components resulted in much smaller connected components due to the binarization process. Therefore, filtering based on size (area) attribute results in the removal of those connected components with an area smaller than the threshold value. Filtering based on area attributes separate the candidates of WBCs but there remain several abnormal components that need to be removed. These abnormal components are not WBCs but stained similarly as a WBC. However, these components are irregularly shaped, unlike the WBCs which are mostly circular. Thus, shape-based attributes such as Eccentricity, Solidity, Compactness, etc. can be utilized to remove such abnormal components. However, in this work, the ratio of the moment of inertia to the square of the area (as described in Section III-A2), is used where the value of M I /A 2 increases as the shape deviates from the circular shape. This attribute is used for the filtering of the abnormal components.
In Fig. 4, cell and nucleus segmentation for a sample image from the ALL-IDB dataset is shown. The shape of the segmented region is the shape of the original cell or nucleus unlike structuring element based filtering where the shape of the segmented region is influenced by the shape of the structuring element. The proposed method faces difficulty if the cell boundary becomes hard to identify after the binarization process (see Fig. 5b). In that case, only the nucleus is segmented. However, in the Maxtree implementation, we also preserve the centroids of each nucleus which can be used to obtain the sub-image containing one WBC (Fig. 5d). Later using gray-level thresholding followed by hole filling and erosion resulted in cell segmentation (Fig. 5f).

D. Feature Extraction
Pattern spectra based on size, shape, or any other attribute or a combination can also be computed from the max-tree representation to obtain the feature vector for classification of  the WBCs. Size and/or shape distribution, used for generating pattern spectra, comprises an ordered set of operators each of which removes features smaller than a particular size or shape from the image. Different sizes and shape-based features such as Area, Entropy, Moment of Inertia, Elongation, Mean x-position, Mean y-position, Eccentricity, Solidity, etc. can be calculated incrementally during the Maxtree creation. Besides, the number of lobes in the nucleus which is an important feature for the classification of WBCs can also be calculated from the Max-tree using the same technique used to count the number of cells [24]. Similarly, elongation of the connected components can be used as a shape measure. After obtaining the feature vector various state of the art classifiers can be explored for classification. In this paper, we did not explore the classifiers since a plethora of work achieved the state of the art classification performance with these features but the method of extracting the features was different than our approach. Nevertheless, we will also explore different classifiers with the features extracted using our approach in our future work.

E. Cell Counting
In [24] we have reported the process of counting and annotation of cells using Max-tree representation of the cell images. In segmented cell images with a dark background, cells are represented with bright intensity. Therefore, in Maxtree representation cells being the extremal intensity can be found in the leaves of the tree (see Fig. 1) and thus the number of cells can be approximated by the number of leaves. The proposed method is conceptually easy, does not require any prior training or annotation on the contrary to the other state of the art approaches. Rather the proposed method provides the annotation of the cells for further use. Our Maxtree based cell counting approach is robust in case of partial overlapping of cells. Segmentation of such cells will result in a single connected component. However, filtering the binary image with a Gaussian kernel creates a region in the center of each cell which is brighter than its surrounding (see Fig. 6(c)). In Maxtree representation these regions will be in three leaves and thus counted as three different cells. However, if the cells are fully overlapped, it is not possible to identify them with our method and will be counted as one cell.

V. DATASETS
In this work, we have tested the proposed method with two datasets: Leukocyte Images for Segmentation and Classification (LISC) database [42] and Acute Lymphoblastic Leukemia Image Database I (ALL-IDB I) [41].

A. LISC Database
LISC Database 1 includes peripheral blood samples from healthy people. The Gismo-Right technique is used for smearing and staining the slides for obtaining microscopic images. The microscopic images are then digitized in BMP format with a size of (720×576) pixels. There are 250 images with ground truth provided with the freely distributed database. Ground truth for segmentation along with the classification of the WBCs into five classes of normal WBCs is done by the expert. There are 53 images with basophil, 39 images with eosinophil, 52 images with lymphocyte, 48 images with monocyte, and 50 images with neutrophil WBCs in this dataset.

B. ALL-IDB I Database
The ALL-IDB I Database 2 includes blood samples from both Healthy Non-ALL subjects and probable ALL patients. There are 108 images in JPG format with 24-bit color depth and resolution of (2592 × 1944) pixels. It also includes the ground truth positions of the WBC cells in the images that are labeled by the experts.

VI. EXPERIMENTS AND RESULTS
The performance of the WBC cell segmentation follows the performance of the nucleus segmentation. Segmentation of the nucleus uniquely identifies a WBC from other blood cells since other cells do not have a nucleus. Therefore, we report the performance of the nucleus segmentation and compare the performance with the performance of the method proposed by Moshavash et al. in [28]. In [28] Moshavash et al. used a similarity measure defined as in equation 5 for measuring the performance of nucleus segmentation: LISC database contains five classes of WBCs, each of the classes has nuclei with a distinct morphology. For example, nuclei of Neutrophil are mostly multilobed, nuclei of Basophil and Eosinophil are bilobed, nuclei of Lymphocyte are eccentric, whereas the nucleus of Monocyte is almost kidneyshaped. The performance of WBC Classification is immensely influenced by the proper segmentation of the different types of the nucleus. Moshavash et al. reported that their method obtained 76% of the average similarity measure for the LISC database which is much better than other methods. However, there is no discussion of the segmentation performance of their method in different classes of WBCs. The similarity measure used in performance measurement is based on the assumption that the true segmentation is the manual segmentation done by an expert. However, in our experiments, we have observed that the segmentation made by the expert fails to achieve the complex morphology of nuclei properly in most of the cases. In Fig. 7 it is visible that the proposed method achieved better segmentation of nuclei where lobes of the nucleus are properly segmented. The number of lobes present in the nucleus is  [28] one of the important features in classifying different types of WBCs. The proposed method will able to obtain this information more accurately because of its better segmentation of the lobes. Therefore, similarity measure obtained using equation 5 may not able to perform proper justice to the proposed method and the actual performance of the proposed method would excel.
Even though, the proposed method obtained significant improvement compared to the state-of-the-art method proposed by Moshavash et al. in [28] in segmenting nucleus (as seen in Fig. 8) where the proposed method achieved 89.912% average similarity measure. The segmentation performance of the proposed method varies among different types of the nucleus. This varied performance is mainly due to the quality of the staining of the nucleus. Cells with the properly stained nucleus are segmented more accurately since proper staining resulted in larger connected components.
In the ALL-IDB1 dataset, only the positions of the lymphoblasts are identified by the experts. However, there are lots of other healthy WBCs along with several abnormal components present in the images. Due to the absence of ground truth position of all WBCs and ground truth segmentation of the cells, it is not possible to measure the segmentation performance using the equation 5. Therefore, the performance of this dataset is evaluated visually (Moshavash et al., also do not report any similarity measure for this data set). Few results are shown in Fig. 9.
The performance of the cell counting algorithm proposed in [24] by the author of this paper mostly depends on the performance of the segmentation. If the cells are segmented properly the performance of the proposed counting algorithm also improves. In Fig. 10 the performance of the counting algorithm on the LISC dataset can be observed.

VII. CONCLUSION
The task of automated microscopic cell image analysis to identify the WBCs is very complex. Numerous researches have been done to solve this problem. However, to overcome the challenges of different stages such as segmentation, feature extraction, classification, and counting different techniques are utilized. In this paper, a system based on connected mathematical morphological operators implemented using Maxtree data structure is proposed to solve the problem of segmentation, feature calculation, counting of the WBCs. In the proposed system Maxtree data structure plays the central role that facilitates the analysis of the cell images. It stores the connected components of every gray level present in the image along with their different attributes/features calculated on the fly during max-tree creation. These attributes are used in the segmentation stage and also can be used as a feature vector in the classification stage. Besides, the hierarchical structure of the tree enables the counting of the number of WBCs present in the image. The proposed system is conceptually easy, computationally efficient, and performs better than a state of the art method.

VIII. FUTURE WORK
In the future, the performance of the classifiers will be explored using the features obtained from Maxtree. An interactive platform for ALL cell detection will be developed using the proposed technique.