Audio Augmentation for Traffic Signs: A Case Study of Pakistani Traffic Signs

Augmented Reality (AR) extend the appearance of real-world by adding digital information to the scene using computer graphics and image processing techniques. Various approaches have been used to detect, identify and track objects in real environment depending upon the application, shape of the tracking object and environment type. The marker-based tracking technique is the most commonly used method in augmented reality applications in which fiducial markers are put in the real-world for tracking. In this work we proposed a model to detect and identify the traffic signs through marker based technique to improve the usability of marker-based detection in augmented reality applications. We developed an AR application that can detect and recognize the markers designed for Pakistani traffic signs and augment them with voice alert to the driver so that he does prepared for the upcoming hurdle on the road. As identified by literature no work has been performed on augmentation of voice for traffic signs. From experiments the model outperforms than baseline techniques. Keywords—Augmented reality; traffic sign detection; traffic sign recognition


INTRODUCTION
Road safety is becoming increasingly critical day by day world over, particularly in developing countries like Pakistan with the growing rate of ownership of vehicles.In fact, a large number of individuals lose their lives every year due to road accidents and the yearly rate of such individuals seems to be increasing.According to statistics, every year 1.25 million individuals lose their lives in road accidents.A huge number of others sustain wounds, and some experience long lasting disabilities.No nation is saved this toll in lives and suffering that especially influences youngsters.Enormous human potential is crushed, with severe social and financial effects.Road safety is therefore a noteworthy general medical issue around the world [1].
Advanced Intelligent Driver Assistant System (AIDAS) is a critical part of the intelligent transport system which surely warns potential hazards on the road, helps the driver in routing and direction.Traffic signs are landmarks that direct the driver on hazardous road conditions and therefore enhance road safety [2].Generally, the traffic signs have distinctive color, shape, size and characters or symbols.Most commonly triangular, circular, rectangular and octagonal shapes are used to symbolize warning, prohibitions, recommendations and stop signs respectively [3].
The traffic signs are put beside the road so that they can be seen by the drivers easily.Inability to see traffic signs due to various conditions can bring about lethal car accidents.Currently, traffic risks are increasing due to the increase in number of vehicles.The adoption of an automated system for detecting and recognizing traffic signs thus improves road safety.
In general, global positioning system (GPS), radio frequency transmitter and vision-based techniques are used to improve driver alertness [4].The first two methods have weaknesses such as failure in GPS connection and installation of RF transmitter at every single traffic sign is very difficult and costly.As a result, the vision based technique is more reliable and inexpensive.From two decades, traffic sign recognition (TSR)has become very attractive research topic among many computer vision communities due to its more practical applications [2].
In this paper we present a model to detect and recognize the traffic signs through marker-based technique.When the car's camera sees the marker, it will generate a sound, corresponding to that traffic sign which will alert the driver.
The rest of the paper is ordered as follows: literature review is described in section-II.Section-III elaborates the proposed model.Section-IV provides the results and discussion of detection and recognition of markers in outdoor environment.Finally, the conclusion is given in section-V.

II. LITERATURE REVIEW
Road signs are defined with a particular shape and color that makes them easy to spot before other common object.In general, three primary approaches are used as a part of traffic sign recognition i.e. color-based, shape-based and neural network based.Due to unique color of traffic signs, the colorbased technique has turned out to be exceptionally prevalent.

A. Color Segmentation
Color is a predominant visual component and without a doubt represents essential data that the driver must oversee.The most specific colors used for traffic signs are red, blue, yellow, black and white.Color-based detection technique is considered to be the most widely used strategy for initial search space reduction.
The RGB color space is followed by [5].In light of their investigation they worked just on 3 colors: red, blue and yellow to make 3 binary images containing just those pixels www.ijacsa.thesai.orgonly that belong to these colors.All associated pixels to the same connected area are marked together.Beneath a predefined threshold regions are simply rejected in light of the fact that they are generally not traffic signs or otherwise, the target is such that detection due to its smaller size could not be successfully classified.They accomplished good results for their classifications of selected signs.For example, signs of obligation and prohibition signs.
Numerous researchers believe that the RGB color space is extremely delicate with respect to lighting variations.These strategies are led by HSI or HSL color space.The direct threshold on (RGB) space is rarely used due its sensitivity to variations in lighting.The HSI color models are the most popular as it depends on the human color observation.In addition, it is generally considered invariant to changes in lighting and brightness [6][7][8].HSI is used by many researchers such as [9][10][11][12]with the reason that the HSI space models is superior to RGB for human vision and permits some variety in the intensity of light.
However, some researchers are not happy with the efficiency of HSI as it does not exhibit any adjustment in color temperature under various climate conditions such sunny, cloudy, foggy or rainy, yet just changes the intensity of light.A group of researchers [13,14] used CIECAM color space which can foresee color appearance as precisely as a normal observer.Compared with other color models, the data demonstrate that CIECAM surpasses the RGB and HSI color spaces for sunny, cloudy and rainy conditions with a correct fraction of 94%, 90% and 85% respectively.Variations in climate conditions, when there are various artificial lights, the color of traffic signs and lighting sources fluctuate, leading to the situation when the adoption of most techniques in colorbased traffic sign segmentation may not work accurately all the time.In fact, till date no generally acknowledged technique is available in color-based segmentation of traffic signs [15].

B. Shape based Detection
Another basic strategy is to exploit shape information in a way got from edge properties.The primary advantage is its strength in connection to the distinctive lighting conditions and the deterioration of the signboard.A few authors favor a colorless approach since they don't consider color segmentation to be absolutely reliable because of their sensitivity to different factors, for example, changing weather situations, reflection of sign board surfaces, time, or extreme camera exposure.On the other hand, shape of the traffic sign is more powerful attribute with respect to the varying lighting conditions and the degradation of signs boards.
A corner detection technique with a specific color was applied by [16]to discover traffic signs.They just used a few points on the boundary of circle rather than every pixel.Subsequently, this technique is susceptible to occlusion issues.The same authors have accomplished the task of traffic sign identification by merging color information and shape information using boundaries [15].
A new approach of shape-based identification is the Haar cascade classifier which needs training a classifier for each traffic sign, prompting a tedious identification process that was tea large amount of the processor time as applied in [6].
Hough transformation and its derivatives dominate the detection phase of traffic signs as used by [17,18].
The groups of researchers [19,20]utilized the fast radial symmetry transformation to discover normal signs in the form of a polygon such as triangle, square and octagon.The general polygon detector algorithm is utilized in these attempts to detect rectangular speed signs in the Unified State of America.Barnes and Zelinsky [4] applied radial symmetry detector in their system that uses gradient image to recognize Australian circular speed signs in real time.

C. Classification of Signs
The meaning of the traffic sign lies inside the candidate in the form of symbols.To recognize the meaning of a traffic sign, the classifier generally require either an inside part of the target sign or some specific properties as its inputs, that recognize the candidate sign from other signs.Most general classifiers used as a part of traffic sign detection are neural networks [5,21] and support vector machine [22,23].Previously, some work has merged the color with support vector machines (SVMs) for shape-based detection and in addition SVM-based classification [24].

D. Augmented Reality Applications to Navigation
A tourist application for an Android system called i-Street was introduced by [25], which intends to detect, recognize and read street plates in a video stream and after that measure the relative location to augment them precisely with virtual overlays.Another application developed by [26] augments surrounding geographic information such as street name, virtual roads and current location on the screen as users walk on a street to give them an overview of the surrounding environment.The Smart Multimedia Guide developed by [27] is a location-based application that manipulates various services and technologies.The system can detect the artwork closest to the user, and make these works able to automatically tell their stories using multimedia facilities while users visit.
A device for a finger called EyeRing allows the user to point to an object, take a picture, and listen to comments about what he has just focused on.EyeRing can also act as an assistant for navigation or translation and even help children learn to read [28].
Through the study of relative work done in recent years in traffic sign recognition domain we can say that this is a vast domain where a range of techniques available for classification of signs and still there are a lot of ongoing research.In the literature as identified, no work has been performed on augmentation of voice for traffic signs.To overcome this problem, this research work focuses on the development of an application that will guide driver about road signs through aural instructions.

E. Challenges in Traffic Sign Recognition
The recognition of the road sign at first look seems to be extremely basic and simple, since it is characterized by a few directives, such as color, shape, size and pictogram.However, in reality it is exceptionally difficult job as there are additional www.ijacsa.thesai.orgfactors that influence the task and performance of traffic sign recognition.Automatic detection and recognition of a traffic sign may include same challenges as object identification in natural environments.
In spite of the fact that traffic signs are very much characterized through laws and intended to be seen easily, there are still a lot of difficulties for TSR systems.For example: a) Traffic signs are similar inside or across the categories b) Traffic signs may have blurred due to long exposure to sunlight or are grimy so they are not any more in their specified colour c) The sign board shaft may have twisted and therefore, the sign is no longer symmetrical to the road.
d) Lighting conditions may make colour identification unreliable.
e) Low contrast of environment may make shape identification hard.
g) In urban regions different objects may look fundamentally the same as traffic signs.
h) The occurrence of other items (e.g., moving autos, plants, walkers, shop signs, and so forth.)can occlude the visibility of a sign.
i) Video or image gained from a moving auto frequently experiences motion blurs.So the detection and recognition of traffic signs from such scenes turn out to be very challenging.

III. THE PROPOSED MODEL
This research work is mainly focuses is on developing an augmented reality application that can detect and recognizes the markers created for Pakistani traffic signs.When the car's camera see the marker it identify it and then generate a voice command associated with that particular traffic sign to the driver to be alert.
The design of the proposed model is given in fig. 1 while the algorithm of the proposed model is given below:

A. The Marker Database
For the recognition of traffic signs through marker, Pakistani traffic signs were collected.A total of 130 traffic sign images were collected for analysis.The images were then converted into black and white for creating markers.The markers were created in Adobe Photo Shop CS4.Images database were classified into four categories i.e.Warning, Mandatory, Prohibitory and Stop as shown in Table 1.Input: Video frames from dashboard camera of the car 2.
Output: Audio augmented scene of the road on the camera

3.
Initialization(Camera, Variables)  Marker-based methods utilize pattern files for identifying a specific marker.The pattern file is recorded in stable environment where the lighting and brightness level are constant.The marker tracking failure happens in uncontrolled lighting and brightness as well as varied contrast levels in an environment.To avoid this problem, multiple pattern files are recorded for a single marker of traffic sign.In this technique, single marker is recorded under different lighting conditions according to daytime such as in the morning, noon, afternoon and evening time.This strategy enhances the marker detection performance in conditions where lighting and brightness levels are changing.

B. Effect of Lighting and Brightness on Marker
Markers are affected constantly with the change of lighting and brightness conditions of environment which may lead to identification failure or false identification of markers.For the solution of this problem a technique of multiple pattern files at different timing of the day with different lighting and brightness levels of environment is proposed.
When the car's camera detects a marker on the road, these multiple pattern files will be checked for matching with the detected marker.This approach increases the rate of marker identification during different lighting, brightness and contrast levels in environment.Applying this technique may increase the processing time but since our research is only related to the detection and recognition of the markers rather than with continuously tracking the markers, so this processing time does not overload the system.For accurate and fast identification of traffic sign markers, total of 4 pattern files were captured for a single marker at different lighting conditions depending on the day time i.e. morning, noon, afternoon and evening as shown in Figure 2.

C. Experimental Setup for Checking the System
Experiments were performed with a car prepared for the evaluation of the system.A webcam of 5.0 Mega pixels is used with a Laptop having Core i5 (quad core) 2.5GHz processor and 4 GB RAM.The webcam was attached with the rearview mirror of the car and connected to the laptop.The laptop was also attached to the speeker of the car through a 3.5mm cable for the voice alert.So that when the camera detect and recognize a marker then it can generate voice command to the driver through the car's speeker loudly.The experimental setup of the car is given in Fig. 3.For experiment purpose the markers were place at different location on the road side.The experiments were conducted on different kinds of road such as rough and smooth, straight and having turns.A sample marker for STOP sign is place on the road as shown in Fig. 4.

IV. SYSTEM EVALUATION AND RESULTS
To prove the effectiveness of the framework, the proposed system was tested on 20 markers.These 20 markers are selected manually from the database such that some of them are very similar to each other to check the accuracy/precision of the system.The resolution of the video is 640×480.Our main focus is only on detection and recognition of traffic signs when they appear in front of camera.We do not consider tracking the signs continuously.

A. Detection Rate Results
In this section the quantitative data is presented for the experiments performed on the proposed system.The experiments were conducted at different timing of the day i.e. www.ijacsa.thesai.org at morning, noon, afternoon and evening and with different car speeds i.e. 0-60 km/h, 65-80 km/h and 90-110 km/h.Similarly in the second part the recognition results of the system are given in the similar way.

1) Detection Rate at Morning
The experiment conducted at morning gave good results at all speeds i.e. 100%, 96%, and 86% for speed limits 0-60km/h, 65-80km/h and 90-110km/h respectively as shown in Table II.

3) Detection Rate in Afternoon
The afternoon experiments gave similar results as in the morning.The results are shown in table IV.

4) Detection Rate in the Evening
The experiment conducted in the evening gave us good result but low than morning and afternoon as shown in table V.The overall results of the percentage detection rate recorded at morning, noon, afternoon and evening, with different car speeds i.e. 0-60 km/h, 65-80 km/h and 90-110 km/h are shown in figure 5.

B. Recognition Rate Results
In this part the results obtained for the recognition of markers from the experiments are elaborated.

1) Recognition Rate at Morning
The experiment conducted at morning gave good recognition results at all speeds i.e. 100%, 96%, and 86% for speed limits 0-60km/h, 65-80km/h and 90-110km/h respectively as shown in table VI.    Figure 6 shows the overall results of the percentage recognition rate recorded at morning, noon, afternoon and evening with different car speeds i.e. 0-60 km/h, 65-80 km/h and 90-110 km/h.The comparison of both the detaction and recognitation rates on the spacified speed is given in figure 7:

C. Subjective Evaluation of the System
In this section the proposed system is evaluated subjectively in the form of graph of a survey performed on Pakistani drivers.The system is evaluated on different perameters such as, knowledge, training tool, information/learning, ease of use, interesting and usability.In this survay 70 drivers were given the questionnaire after giving them demo of the proposed system.There was variety of drivers for example experienced drivers & Nonexperienced (New) drivers and young drivers & old-age drivers.The graph is given in Fig. 8:

D. Limitations of this work
The first challenge in marker-based approach is its tracking process which is extremely sensitive to light and brightness levels.The tracking failure ordinarily happens in outdoor environment where the light and brightness of environment constantly varies depending upon the weather The marker tracking distance is the second challenge that should be tackled.In existing methodology, every marker has its own constrained challenge range.Due the short tracking range, the, marker-based approach is infrequently used in the development of large indoor or outdoor AR applications.There is a need to extend the tracking distance of markers to improve its usability.These issues confine its uses in uncontrolled environment and long range AR applications.Hence, marker based approach of tracking needs change to probe these issues.

V. CONCLUSION AND FUTURE WORK
This work analyzed the current marker-based techniques and pointed out some important issues associated to it.The proposed system enhanced the functionality of marker-based tracking techniques in outdoor environment by developing an application capable of detecting the markers designed for Pakistani traffic signs.This work is primarily concern with Pakistani road traffic signs.Focus on developing an augmented reality application that can detect and recognizes the markers created for Pakistani traffic signs.When the car's camera see the marker it identify it and then generate a voice command associated with that particular traffic sign to the driver so that he may prepare for the hurdle on the road.
During our experiments, we learnt that the speed of car on a rough road produces motion blur in the environment that can affect the marker detection.Therefore, the future work will focus on the design of algorithms that provides a robust and accurate detection and recognition solution for blurred markers.While performing out experiments, we noticed that this our application produces false detection rate (false positive rate, false negative rate and inter-marker confusion rate).Therefore a procedure is needed to overcome this problem of false detection

Fig. 2 .
Fig. 2. Multiple Pattern Files for a Single Traffic Sign.

Fig. 4 .
Fig. 4. STOP Sign Marker place in the Road.
ijacsa.thesai.orgconditions.Fiducial markers with varying lighting and brightness background cause tracking failure.Similarly, the varying sunlight on the camera or fiducial marker also causes tacking failure.

TABLE I .
SAMPLE MARKER DATABASE

TABLE II
The experiment conducted at 12 O'clock gave us low detection rate comparatively to the morning results due to high brightness in environment.The results are shown in table III.

TABLE III .
DETECTION RATE IN THE EVENING

TABLE VI .
RECOGNITION RATE IN MORNING

TABLE VII .
RECOGNITION RATE AT NOON

TABLE VIII .
RECOGNITION RATE IN AFTERNOON

TABLE IX .
RECOGNITION RATE IN THE EVENING