Behavior of Learning Rules in Hopfield Neural Network for Odia Script

Automatic character recognition is one of the challenging fields in pattern recognition especially for handwritten Odia characters as many of these characters are similar and rounded in shape. In this paper, a comparative performance analysis of Hopfield neural network for storing and recalling of handwritten and printed Odia characters with three different learning rules such as Hebbian, Pseudo-inverse and Storkey learning rule has been presented. An experimental exploration of these three learning rules in Hopfield network has been performed in two different ways to measure the performance of the network to corrupted patterns. In the first experimental work, an attempt has been proposed to demonstrate the performance of storing and recalling of Odia characters (vowels and consonants) in image form of size 30 X 30 on Hopfield network with different noise percentages. At the same time, the performance of recognition accuracy has been observed by partitioning the dataset into training and a different testing dataset with k-fold cross-validation method in the second experimental attempt. The simulation results obtained in this study express the comparative performance of the network for recalling of stored patterns and recognizing a new set of testing patterns with various noise percentages for different learning rules. Keywords—Hopfield network; Odia script; Hebbian; pseudoinverse; Storkey; NIT dataset


I. INTRODUCTION
Odia language is one the oldest and official language of Odisha state in the Indian subcontinent used by more than 50 million people. Modern Odia language consists of 47 different characters out of which 11 are vowels and rest are consonants. Automatic recognition of these Odia characters is particularly difficult because many of these characters are similar looking rounded in shape. Many works have been done so far in the field of OCR (Optical Character Recognition) for various Indian languages but very little research has been done for Odia language. In this study we focused on 11 Odia vowels and 36 basic consonants for storing into and recalling from Hopfield network with different learning rules especially when the patterns gets distorted. We add explicit noise into the input patterns from 10% to 50% while recalling them and compare their results for analysis. The main idea for choosing Hopfield network is that it can be a multiple point attractors for high dimensional space and due to the dynamics of network that guaranteed to convergence to local minima. Hopfield network is an auto associative memory network that reproduces its input pattern as an output even if the input pattern is distorted or partially feed into the network. It a fully connected special type of recurrent network excluding self connection with binary or bipolar inputs. The network returns a stored best matching pattern when an input pattern is presented to the network. The experimental simulation have been carried out in MATLAB2014a environment and the performance parameters such as recalling stored patterns and recognition accuracy of new patterns with noisy version is validate and compared with other techniques in the literature.

II. RELATED WORK
Many articles have been published for various Indian languages like Telgu [1], Tamil [2], Gujarati [3], Kannada [4], Bangla [7] and Gurmukhi [5]. Whereas for Odia character recognition, there are few articles available for study. Development of a novel OCR system for Odia character is difficult and challenging task because most of these characters are identical and roundish shape. In 2002 Chaudhuri et al [6] suggested a system for printed odia script with 96.3% accuracy. Mohanti [8] proposed a system by using Kohonen network to recognize Odia alphabets. Roy et al. [9] proposed a system for Odia handwritten numerals with chain code histogram with a accuracy of 94.8%. Bhowmik et al. [10] suggested a system for Odia handwritten digits with approximately 93% accuracy by using hidden markov as a classifier. Sarangi et al. [11] proposed a classifier by using Hopfield neural network for Odia numerals. Panda et al. [12] proposed a single layer perception for Odia handwritten digits recognition by using gradient and curvature feature extraction methods with an accuracy of 85%. Das et al. [13] presented a hybrid method for Odia digit classification using Kirsch operator. Meher et al. [14] proposed a method for Odia character recognition in the presence of vowel modifiers of complex Odia characters with an accuracy of 91.24%. Kumar et al. [15] proposed an Ant-Miner algorithm for Odia character recognition with an accuracy of 90%. Mishra et al. [16] proposed a fuzzy based algorithm using HMM classifier for handwritten characters with an accuracy of 96.3%. Pujari et al. [17] compares different algorithms proposed so far for Odia character recognition. Chen L -C et al. [18] proposed a modified Hopfield network for character recognition system by using a single error correction-single error detection method to improve the learning in the network. M.B. Sukhaswami et al. [19] proposed a method for Telgu character recognition using Hopfield network and then modified it to Multiple Neural Network Associative Memory (MNNAM) to overcome the storage capacity limitation of Hopfield network.
Prateek Malik et al. [20] proposed handwritten character recognition using Hopfield network and wavelet transforms www.ijacsa.thesai.org with various noise percentage. Kalyan S Dash et al. [25], Proposed a hybrid feature based model for Odia numerals with ISI Kolkata database. With hybrid feature and DLQDF classifier they claimed a 98.50% of recognition accuracy and with MQDF classifier they achieved 98.40% of accuracy. Kalyan S Dash et al. [26] presented a review report on various preprocessing, segmentation and feature extraction techniques with several classifiers for Odia character recognition. They also created a standard alphanumeric Odia handwritten character database. The comparisons presented there are being taken into consideration for this work. Indugu Rushiraj et al. [27] Achieved 87.60% accuracy for Odia consonants classification by using shadow, centroid and distance based features with weighted Euclidean distance method for classification. Smruti Rekha Panda and Jogeswar Tripathy [28], proposed a template matching with Unicode mapping for Odia offline character recognition with a accuracy of 97% for basic characters and numerals. Kalyan S Dash et al. [29], proposed a novel feature extraction method called BESAC: binary external symmetry axis constellation with a Boolean matching character recognition technique for Odia and Bangla Numerals and characters.  Table I, a detail study on Odia characters and numerals recognition is shown with their claimed accuracy.

III. ODIA CHARACTER SET AND CHANLLEGES
Odia language is the mother tongue of the Indian state of Odisha and the script originally came from Brahmi script. The way of writing Odia language is unique and different from other regional languages in India. Modern Odia script consists of 11 vowels, 36 consonants and ten digits. Apart from these basic Odia character set, the Odia ligatures may be formed either merging a vowel diacritic with the consonant or by clustering two or more consonants. There are nearly 116 composite characters also known as Juktakhyaras. A printed version of Odia characters and numerals are shown in Fig. 1, 2 and 3 for vowels, consonants and numerals respectively. Most of the Odia characters are roundish in shape and very similar to each other. Recognition of Odia characters are more challenging pattern recognition task as i) many roundish and similar shape, ii) there is no reference line as in other Indian regional languages like Hindi and Bangla, iii) unavailability of more number of standard databases for both printed and handwritten Odia characters. Few samples of Odia characters with almost similar in shape are shown in Fig. 4.

IV. HOPFIELD NEURAL NETWORK
Hopfield network [21] is merely the best known autoassociator neural network that acts as content addressable memory. It is a fully connected network with symmetric weight where no neuron is connected to itself. The neurons of this Hopfield network are updated asynchronously and in parallel and this type of networks guaranteed to converge a closest learnt pattern. The weight matrix of N neurons Hopfield network is given by a matrix with symmetric weights and the diagonal is set to zero. The state vector S at any point of time of the Hopfield network is given as: Where is the current output state of unit and the local net input to unit is given by Where is the weight of connection to unit and the next state of the of a unit is given by Where is the threshold at unit .

V. LEARNING RULES
Learning rules in Hopfield network is basically finds the set of connection weights which allow the network to produce desired response when a pattern is submitted to the network. To compare the performance of Hopfield network with Odia vowel characters, three learning rules are used in this study. These learning rules are presented in the following section.
A. Hebbian Learning Rule [22] Hebbian rule states that when one neuron stimulates another neuron from two connected neurons in the network for firing then the connection weight between these two cells is strengthened otherwise it is weakened. In other words we can say when a weight contributes to firing a neuron, the weight is increased. The Hebbian rule for updating the weights is presented below:  Initialize all weights to zero * +  For each input training(I)-target output(O) vector pair {I:O}, set the following activation as  Adjust the weight for * + as follows (4)  Now set the activation of output unit as follows To store(train) patterns * + with a nodes Hopfield network performs the following equation by Hebbian learning rule. www.ijacsa.thesai.org For training the two associated patterns in each pair are the same which means with both have same dimension i.e.
. Then the weight matrix is obtained for patterns as the sum of their outer products as: And in the matrix form it is as follows: And the weight between connecting node i to j is ∑ and (7) Hebbian learning rule is incremental and local which means the previous matrix can reused when we add a new pattern to the memory that to be learn.

B. Pseudo-Inverse Learning Rule [23]
Pseudo-inverse learning rule uses the pseudoinverse of the pattern matrix while Hebbian learning rule uses the pattern correlation pattern matrix. When pattern vectors are not orthogonal this learning method is more efficient and it is neither local nor incremental which means a new pattern cannot incrementally added to the network and update does not depend on either side of the connection as it calculate the inverse of the weight matrix. The procedure to apply pseudoinverse learning rule to obtain the weight matrix of Hopfield network is explained as follows.
 Let input patterns of vectors * + and target pattern of vector * + then we can write: (8)  If the matrix has an inverse, then we can rewrite: (9)  Again if is not a square matrix then no exact inverse will exist, but it has been shown that it will minimize to ( ) ∑ by using the pseudoinverse matrix and is as follows: Where is the Moore Penrose pseudoinverse, then the pseudoinverse of real matrix is the unique matrix that satisfies:  So the pseudo-inverse weight matrix can be calculated as :  Hence the Pseudo-inverse learning rule is given by:  Hence the Pseudo-inverse learning rule is given by: ∑ Pseudo-inverse rule works well with linearly independent patterns and stores up to N patterns in an N unit network. This rule finds a set of orthogonal vectors and calculate output weight matrix by pseudoinverse solution.
C. Storkey Learning Rule [24] Amos Storkey in 1997 proposed a new learning rule named as Storkey learning rule is also both local and incremental and provides greater storage capacity ( ( ) ) than Hebbian learning rule ( ). It is local because it only considers neurons at either side and incremental as new pattern can be added to the network without knowledge of old pattern that have been also used for training. The learning rule mathematically defined as: Where w ij p is the weight between i and j after p th pattern has been learnt and y p is the new learning pattern.

A. Dataset
The dataset used for this paper is a combination of printed and handwritten Odia vowel characters which are collected from NIT, Rourkela database for handwritten characters and some downloaded printed vowel characters. The NIT database consists of 47 folders of different Odia characters written by 160 individuals in a specific format with A4 sheet. Each folder contains 320 handwritten samples of dimension . In this paper we consider all images of vowels contains in the first 11 folders of downloaded database. For printed Odia vowel characters we downloaded 20 different samples of each character type, so a total of 3740 samples with 340 samples per each class are collected. For consonants we consider 100 images from each of 36 consonants folders with a total of 3600 patterns for training and testing in our first implementation and divide these patterns with k-fold cross validation method for second part of implementation.

B. Experimental Setup
The experimental environment used in this paper is Window8.1 as OS with 16GB memory and Intel runs in MATLAB2014a. Two experimental scenarios are presented in this paper. First experiment is to recall all the patterns with www.ijacsa.thesai.org different noise percentages from the network. These noises are explicitly introduced with five variations such as 10%, 20%, 30%, 40% and 50% while recall the stored patterns. Two sample images of Odia vowel characters are shown in Fig. 6 with different noise percentages from 10 to 50 percentage of noise .In our second experiment; we partitioned the data with k-fold cross validation method for different training and testing dataset and then recognize test patterns with various noise percentages. To reduce the processing time of the network, image sizes are resize to .

C. Implementation and Discussion
Before training the network with three mentioned learning rules, the dataset is divided into training and testing and preprocessed. Then we applied histogram equalization, binarization and finally resize them to to speed up the training process. For the first implementation we trained the full dataset and tested all patterns for recalling whereas for the second implementation the full dataset is shuffled and then partitioned in training-testing dataset with ( ) method. Both the experiments were implemented for three mentioned learning rules with various noise percentages. The training (storage) algorithm is discussed in algorithm-1 for three learning rules to get the weight matrix. The final weight matrix obtained from the algorithm-1 is used to predict the class labels of test data for each learning rule and the recalling algorithm is given in algorithm-2. To predict a test class label hamming distance metric has been considered in this study that finds the minimum hamming distance difference between the test pattern and one of the trained patterns. perform matrix multiplication of first pattern P 1 with its transpose and obtained the new weight matrix by equation (6). Finally get the weight matrix of all patterns by equation (7). 8. else if rule is -Pseudo-Inverse‖ then 9.

Input
Obtained the weight matrix by equation (12) 10. else if rule is -Storkey‖ then 11. Obtained the weight matrix by equation (14) 12. end if 13. end step-5 14. Assign the diagonal of the final weight matrix to zero and normalize the weight matrix by dividing with N.

D. Implementation-I
In this paper, the first experiment is performed to demonstrate the recall efficiency of the full dataset (3740 Odia vowel patterns) in Hopfield network with three different learning rules such as, Hebbian learning, pseudo-inverse learning and Storkey learning for various noise percentages and a comparative result is shown in Table II. Similarly for consonants, 100 samples of each class (i.e. 3600 patterns) are considered to store into Hopfield network by using three mentioned learning rules and the recalling efficiency with various noise percentages are shown in Table VI. It has been observed that the recalling efficiency of vowels are more than consonants with different noise percentages as many of these consonants are similar rounded shapes in their upper part.

E. Implementation-II
Second experiment is presented to demonstrate the performance of recognition accuracy by splitting the dataset in train and test data with k-fold cross validation method for three different learning rules. In k-fold cross validation, the dataset is partitioned into k randomly chosen equal (or nearly equally) sized subsets (or folds). One fold is used to validate (testing) and other k-1 folds are used for training in the network. This process of training and testing is repeated for k times such that each subset (fold) of data is used for testing exactly once. In this paper we portioned Odia vowels dataset into 5-fold (k=5) with 748 patterns are in each fold and 720 patterns in each fold for Odia consonants. The k-fold cross validation with k=5 is shown in Fig. 5 and for each learning www.ijacsa.thesai.org rules we performed validation for five times named model-1, model-2, model-3, model-4 and model-5 so that each segment of data must be tested once. The recognition accuracy of each model and the average accuracy of each learning rules for Odia vowels are presented in Table III, Table IV and Table V  with different noise percentages respectively whereas  Table VII, Table VIII and Table IX presented here demonstrates the recognition rate for Odia consonants with Hebbian, Pseudo-inverse and Storkey learning rules respectively. Fig. 7, 8 and 9 are presented here to demonstrate the comparison of different models of k-fold validation method for Odia vowels with Hebbian, Pseudo-inverse and Storkey learning rules respectively whereas Fig. 10 demonstrates the comparison graph of all three learning rules. At the same time Fig. 11, Fig. 12 and Fig. 13 presents the comparison for Odia consonants with three learning rules respectively and Fig. 14 represents the comparison graph of three learning rules for Odia consonants.

VII. CONCLUSION
The present study demonstrated the comparison of three learning rules such as Hebbian, Pseudo-inverse and Storkey rule in Hopfield neural network for recalling and recognizing Odia handwritten and printed basic characters (vowel & consonants) in image form. Our study mainly focused on the recalling performance (stored patterns) and recognizing efficiency (new patterns) for distorted Odia characters. It is hard to compare with other models as very little work has been done so far and no such standardized test set available for Odia character recognition, hence the results we obtained are compared with claimed outputs of different methods in literature survey. The simulation results shows a better accuracy of 95.20% with Storkey learning rule than Hebbian (93.80%) and Pseudo-inverse (94.67%) learning rules without noise for Odia vowels and in case of Odia consonants Storkey rule (94.74%) also out performed than other rules. In this simulation work for recognising a pattern, we used a machine learning validation method named k-fold cross validation to partitioning the dataset in five sub parts and in every iteration one sub part is for testing and other four sub parts are used for training by which each pattern must be tested with various noise percentages. It has been seen that Storkey learning rule provides better accuracy in terms of recognising new test sets with noise as compared to other two learning rules.