Semantic Image Retrieval: an Ontology Based Approach

—Images / Videos are major source of content on the internet and the content is increasing rapidly due to the advancement in this area. Image analysis and retrieval is one of the active research field and researchers from the last decade have proposed many efficient approaches for the same. Semantic technologies like ontology offers promising approach to image retrieval as it tries to map the low level image features to high level ontology concepts. In this paper, we have proposed Semantic Image Retrieval: An Ontology based Approach which uses domain specific ontology for image retrieval relevant to the user query. The user can give concept / keyword as text input or can input the image itself. Semantic Image Retrieval is based on hybrid approach and uses shape, color and texture based approaches for classification purpose. Mammals domain is used as a test case and its ontology is developed. The proposed system is trained on Mammals dataset and tested on large number of test cases related to this domain. Experimental results show the efficiency / accuracy of the proposed system and support the implementation of the same.


INTRODUCTION
Images / Videos are major source of content on the internet and the content is increasing rapidly due to the advancement in this area [10,12,13].Digital Image processing / retrieval is one of the hottest research field and researchers from the last decade have proposed many efficient approaches for image analysis such as [6,7,14,15] and retrieval [9,11,16,17].Image retrieval systems are usually based on keywords or text meta-data based [4,18,19] where the retrieval is done based on the textual description of the images.The description about the image is usually provided by the user.Most common search engines such as Google and Bing used keyword based search techniques; this approach is fast and effective; however it still has some disadvantages.In this approach, the image is described by a set of keywords or text-metadata and usually this information is provided by the user.
The keyword based image retrieval system matches user text query to the textual description of the images and return all the images whose description is the possible match.However, it is quite possible that the results returned contain irrelevant images.For example, you may find a dog picture while you are searching for human.This usually happens because the description of the irrelevant image contains that specific keyword.So, the major disadvantage of text-based image retrieval system is that it may return redundant or irrelevant images in the result [13,4].
The accuracy of keyword based image retrieval systems is far from perfect because of the following reasons: 1) If the user made spell mistake while describing the image, this image will never be listed in the result because of this mistake.
2) Sometimes the user has to specify the image description / keywords in natural language which makes it difficult to describe the image as the user has little knowledge about the natural language.
3) It is very difficult to find appropriate keywords for image description (i.e.synonym plays important role in image retrieval).
In conclusion, keyword approach ignores the image features which sometimes results in irrelevant image retrieval [23,24].
Content based Image retrieval (CBIR) has been studied for many years which focuses on extracting and comparing features from the images [20,21,22].Image Features are usually extracted using dominant color, dominant texture, or shape (i.e. this technique focuses on the visual features of the image).Researchers in the last decade have demonstrated the efficiency and accuracy of CBIR based techniques, however, CBIR still lacks to understand the semantic analysis of the image.For example, if the user wants to search -Loin‖ images, CBIR system will not be able to map human concept into image feature (i.e.creating a semantic gap between the lowlevel image features and high-level human understandable concepts).Therefore, semantic analysis needs to be incorporated in content based image retrieval to reduce this gap.
Semantic technologies like ontology offers promising approach to image retrieval as it tries to map the low level image features to high level ontology concepts.Compared to the existing approaches (i.e.text / keyword based and content based image retrieval), Ontology based image retrieval focuses more on capturing semantic content (i.e.mapping image features to concepts), because this can help in satisfying user requirements in much better way.In this paper, we have proposed Semantic Image Retrieval: An Ontology based Approach which uses domain specific ontology for image retrieval relevant to the user query.www.ijarai.thesai.orgThe user can give concept / keyword as text input or can input the image itself.Mammals domain is used as a test case and its ontology is developed.The proposed system is trained on Mammals dataset and tested on large number of test cases related to this domain.Experimental results show the efficiency and accuracy of the proposed system.
The remainder of this paper is organized as follows.In Section 2, we present brief overview of ontology and image analysis, this section is followed by the discussion of literature survey.In Section 4, the proposed Semantic Image Retrieval: An Ontology based Approach architecture and classification mechanism is discussed.In Section 5, the experimental analysis of proposed solution is presented.Finally, the conclusion is drawn in Section 6.

II. ONTOLOGY AND IMAGE ANALYSIS
The word ontology refers to the science of metaphysics which defines the nature with its properties and relations [8].In Computer Science, ontology is a systematic arrangement of concepts, their properties and relations which exist in domain [25].Common components of ontology includes Individuals, Classes, Attributes, Relations, Function terms, Restrictions, Rules, and Axioms; for more details related to these concepts please see [3,5].Ontology can be domain-specific or generic; the former means ontology concepts are defined with reference to the specific domain whereas the later means the concepts are defined in general (i.e. the meaning / relationship of these concepts are already defined by English language) [26].
The implementation of ontology is generally a hierarchal representation defining concepts and their relationships.Three kind of relationships namely is-a, instance-of and part-of are generally used in the ontology; for more information please see [27,28].Ontology are usually develop to share common understanding of information among entities or softwares where each node in the ontology is a concept containing set of attributes and relationships.
In the last decade, Ontologies have been widely used for knowledge representation and sharing.Ontology-based systems have been used in diverse areas such as software maintenance, Business Process Management, Biomedical Informatics, Knowledge Sharing, Knowledge Integration, Semantic Web, Fuzzy Systems, Supply chain management, Healthcare, Text Classification, Medical Domain, Robotics, Autonomic Computing, System Modelling, etc.
The idea of using the ontologies in Image processing for content used retrieval is not new; in the last decade, researchers have proposed many efficient solutions using Ontologies for content based Image processing and retrieval such as [29][30][31][32][33][34].The existing approaches can broadly be categorized into three types namely 1) Color based techniques 2) Shape based technique and 3) Texture based technique.The color based approaches proposed calculate the color histogram of the image and use the same for classification, shape based approaches identify the shape(s) in the image and use it for classification whereas the texture based approaches identify the texture in the image and use it for classification purpose.
Each of the approaches discussed above have some limitation, for example the color based technique will work effectively on the color-dominant image dataset whereas it will be outperformed by other technique on non-color-dominant image dataset.Similarly shape detection in complex images are hard and texture based approaches will be outperformed on non-texture-based image dataset.In this paper, we have proposed a hybrid technique which uses color, shape and texture feature of the image and use these features for classification.www.ijarai.thesai.org[38] had a great impact on the Image Retrieval field as it was very helpful for improving the semantic image retrieval systems accuracy.In this research various image search techniques are described for reduction of semantic gap.Furthermore, based on existing methods and application requirements author have suggested few future assessments.Another important survey was conducted by Liu Ying et al. [39] in 2007 about the recent technical achievements on semantic based Image Retrieval; majority of the recent publications were included as the test data for the survey covering diverse amount of aspects in this area.Similar work has also been conducted on medical images by Xu J et al. in [41], the authors focused on the key features of the image (e.g., shape, texture) in this research.The authors concluded that the performance of most CBIR systems is forced by these features because they cannot efficiently model the expectations of the user.All of existing studies helped in improving the results of content based images retrieval and www.ijarai.thesai.orglowering down the semantic gap between the user requirements and the search results.

IV. SYSTEM ARCHITECTURE
Semantic Image Retrieval (SIR): An Ontology based Approach system architecture describes the working of the various components / modules of the system and their interaction with each other.Figure 2 shows the detail system architecture of SIR and consists of the following modules:  Query Engine

A. Query Engine
Query Engine is responsible to take input from the user using the web interface; the input contains the content which

1) Text Input:
The first method of providing the input to the SIR is text based.In this approach the user is required to enter the text containing the information about the thing that he / she wants to search.This approach is commonly used in the current search engines, e.g.Google, Bing, AltaVista etc.The main focus of incorporating this approach in SIR is to provide ease to the users as they do not have to learn the new way of interacting with the SIR.The user has to simply write down the text query (e.g.Cheetah, Elephant, Horse etc and the same is passed to Text based Query module. 2) Image Input: The second method of providing the input to the SIR is image based.In this approach the user is required to provide the image of the object(s) which he/she wants to search.The input image can contain a single object or multiple objects.The user is also provided some options (optional) to describe the input image.This approach is feasible when the user wants to search related objects / images similar to the one he / she has.Furthermore, this method provides flexibility in the input method, as it gives new dimension to the searching.After taking input from the user, Query Engine built the query for the input.As Ontology based Knowledge base is used, the query is built in SPARQL language.The query building process consists of the following two components.
a) Text based Query: This module is responsible for building the query for the text based input.In Step 1, all standard stop-list / stemmer words like (-is‖, -the‖, -on‖, -and‖…) are removed from the input text.In Step 2, SPARQL query is generated with all possible -AND‖ and -OR‖.The generated query is then passed to the Matching Module for the further processing.
b) Image based Query: This module is responsible for building the query for image based input.In Step 1, object(s) in the image are detected using shape based feature extraction as described in [2].After object detection, two sub-steps are performed: In the first step, the detected objects are passed to www.ijarai.thesai.orgColor based Feature Extraction technique which uses MTH algorithm proposed by Guang-Hai Liu et al in [1] to calculate the color value and pixel color of the objects; In the second step, the detected objects are passed to texture classification technique proposed by Mohsen Zand et al in [35] to identify texture / pattern (if any) in the detected objects.In Step 3, the low level features extracted using the previous two steps are converted into high level ontology concepts; the image description if provided in search by the user are also converted into ontology concepts, after completing this step SPARQL query is generated using these parameters.

B. Matching Module
Matching Module takes SPARQL query as input from the Query Engine and executes the same on the Ontology Knowledge Base to retrieve the most related images.If the query results in successful search, the output images are passed to ranking module for result ranking.If the search is unsuccessful (i.e.relevant images are not found in our knowledge base), matching module performs the following three steps: Image Search: Matching Module searches the internet for relevant images by querying existing search engine (i.e.

1) Image Processing:
The images returned by search engine may not be relevant to the user query; therefore the content of each image needs to be verified.This module is responsible to check the images for the compliance with the input query.The objects in each image are detected using shape based feature extraction and these objects are passed to 1) Color based Feature Extraction technique which uses MTH algorithm proposed by Guang-Hai Liu et al in [1] to calculate the pixel color and color value of the objects and 2) texture classification technique proposed by Mohsen Zand et al in [35] to identify texture / pattern (if any) in the objects.In the next step, the low level features extracted in the previous step are converted into high level ontology concepts; afterwards SPARQL query is generated using these concepts and executed on the ontology knowledgebase.If the result class(es) matches user search query, the image is included in the resultant set otherwise it is discarded.As a result only the related images remains and the non-relevant images are discarded in this step.
2) Ontology Manager: Ontology Manager is responsible to insert the new relevant images features and concepts (gathered from the web and filtered in the previous step) in the ontology knowledge base.

C. Ranking Module
Ranking module is responsible to rank the images according to relevance with the user query.The resultant image set passed by Query Matching Module contains image and matching value (which is calculated as a sum of matched ontology concepts with reference to user query); the result set is sorted in descending order according to the matching value.After sorting, top ten images are displayed to the user (i.e.most matched images are showed first) and the remaining are displayed on user request in the decreasing order.

V. SIMULATION
Initially for the experimentation, we trained Semantic Image Retrieval (SIR) and built the ontology concepts using 900 images which contain pictures of 20 different mammals.Partial training dataset is shown in figure 5. We have evaluated SIR on large number of test cases; results were promising and showed the efficiency of the proposed system.In this section, few of the test cases are presented and discussed in detail.www.ijarai.thesai.org Figure 6 shows the accuracy comparison of color based shape based, texture based and our proposed approach with reference to four different test cases.As depicted by figure 6, our proposed hybrid approach outperforms these approaches with reference to accuracy.
Figure 7 shows the percentage improvement of proposed hybrid technique over number of test cases; as shown in figure 7 the proposed solution improvement percentage varies over number of test cases; this is because the content of images present in each test case plays an important role.In figure 9, the user used cheetah image as input; Query Engine generates the query for the same and executes it on ontology knowledge base.The resultant images are found in the knowledge base, therefore web image search, image filtration and ontology updation steps are skipped in this test case.www.ijarai.thesai.orgThe results are passed to Ranking Module which ranks the results and displayed it to the user as shown in Figure 9.In test case 4, the user used text input feature of the SIR system and provided the input as text.SIR generates the corresponding query for the same and executes it on the knowledge base.The related images are displayed to the user as shown in figure 12.

Fig. 1 .
Fig. 1.Zero Level Architecture of Semantic Image Retrieval: An Ontology based Approach

Fig. 2 .
Fig. 2. System Architecture of Semantic Image Retrieval: An Ontology based Approach

Fig. 3 .
Fig. 3. Partial Ontology Knowledgebase the user wants to search.The input can be provided in two ways by the user.

Fig. 4 .
Fig. 4. Query Matching Google or Bing).The results returned by search engine are passed to Image Processing Module for content verification.

Fig. 8 .
Fig. 8. False Positive Percentage vs Test CasesFigure 8 shows false positive percentage over number of test cases, the proposed solution false positive percentage ranges from 0.60 to 2 percent in the test cases which shows the result accuracy of the proposed solution.

Fig. 10 .
Fig. 10.Test Case 2 Figure 10 and 11 are similar to the first test case (figure9) where the user enters an image and relevant images are returned to the user.