Differential Evolution-based Approach for Tone-Mapping of High Dynamic Range Images

—Recently, high dynamic range (HDR) imaging has received significant attention from research community as well as the industrial companies due to valuable applications of HDR images in better visualization and analysis. However, HDR images need to be converted to low dynamic range (LDR) images for viewing on standard LDR display screens. Several tone-mapping operators have been proposed for the conversion, however, so far, no significant works have been reported employing artificial intelligence to achieve better enhancement of the output images. In this paper, we present an optimization-based approach, to enhance the quality of the tone-mapped LDR images using metaheuristics. More specifically, the optimization process is based on the differential evolution (DE) algorithm which takes tone-mapping function of an existing histogram-based method as initial guess and refines the histogram bins iteratively leading to progressive enhancement of the quality of LDR image. The final results produced by the proposed optimized histogram-based approach (OHbA) showed better performance compared to the existing state-of-the-art tone-mapping algorithms.


I. INTRODUCTION
The human visual system (HVS) can adapt to the high dynamic range (HDR) of natural and synthetic scenes for viewing details of the dark and bright regions.However, the capturing of devices, such as commonly used cameras, suffer from sensor limitations and loss of precision in the quantization process for storing on digital media [1][2].This is to say that the images captured with standard cameras and displayed on standard screens have lower quality than the actual scene viewed by the human eye.This gap in quality inspired research works on development of software and hardware technologies for capturing and displaying high quality images using HDR image and video technologies.HDR images have valuable advantages in many fields of life, such as realistic visualization experience, better scene analysis, gaming, and enhancement of medical images for correct and accurate diagnoses among others [3][4].
In image processing, the dynamic range refers to the ratio between the intensities of the brightest and darkest pixels of the scene.Fig. 1 shows typical brightness levels of different objects that we come across in our daily life.When objects, of very different brightness, are present together in a scene, the scene is referred as an HDR scene.In a real-life example, a sunny natural scene with shadows and light sources and/or light reflections present in it is an HDR scene.Recently, new designs of cameras have emerged which can capture much larger dynamic range of the scene than the standard cameras [5][6].However a more commonly used technique to capture HDR scenes is to take a series of shots by varying the exposure time setting of the camera, and merging them to a single HDR image using a technique generally referred as exposure fusion [7][8][9][10].The merging process requires carefully removing the effect of moving objects and many techniques have been proposed for this purpose [11][12].In the current state of imaging technology, high dynamic range of natural scenes can be captured quite reliably, and the recent focus is mainly on improving the speed of operation and reconstruction of HDR from a single LDR image using deep learning approaches [13].
Replicating an HDR scene on a standard screen is a challenging problem due to several factors, most importantly due to limitation of the hardware.New HDR displays have hit the consumer market, but their dynamic range is still much lower than the dynamic range of HDR images.An HDR image shown on an LDR display without any processing might look like one of the shots shown in Fig. 2 depending on the display settings.For a better viewing experience, a tone mapping operator (TMO) is required.The term refers to the algorithms used to match the dynamic range of the content with that of the screen.In general, the TMOs enhance the brightness of dark pixels and compress that for the bright pixels.Many TMOs use some model of HVS characteristics to produce results of better visual quality.Performing automatic mapping from a very large range of values to a limited set of values while preserving the quality of image is in general an NP-Hard problem [14][15]; therefore, the use of metaheuristics is appropriate for such a complex problem.Metaheuristics are powerful nature-inspired stochastic computational algorithms that fall under the artificial intelligence umbrella; they work iteratively to solve complex problems by generating optimal or close-to-optimal solutions which cannot be resolved using traditional optimization methods [16].Furthermore, metaheuristics do not require mathematical modelling of the problem to be optimized referred as a black-box problem [16].Metaheuristics have been successfully used in image processing and various other applications [17].In this work, we utilize a metaheuristic algorithm with a recently proposed TMO, referred by the authors as Adaptive Threshold vs Intensity based TMO, or ATT in short [18], to produce tone-mapped LDR images of high quality.The authors of ATT noted that performance of histogram based TMOs is heavily dependent on construction of the histogram bins.Therefore, they used a sensitivity model of HVS, referred as Threshold vs. Intensity (TVI) model [19] to form histogram bins mimicking the function of human eye.We start with the initial tone-mapping curve generated by the ATT in the form of a lookup table (LUT) and iteratively change the values using the differential evolution (DE) metaheuristic.The proposed optimization framework successfully enhances the quality of the tone-mapped results.Experimental evaluations are carried out on several test images of different types of content and dynamic range to test and evaluate the proposed optimized histogram-based approach, referred as OHbA hereinafter.The contributions of this paper are as follows:  An optimized histogram-based approach is proposed that applies the DE optimizer on the histogram of the input HDR image to convert it optimally to an LDR image.
 The DE algorithm is tailored to the constraints of the TMO matching problem.
The rest of the paper is structured as follows.Section II reviews the related work.The proposed approach including the ATT algorithm and the optimization process is presented in Section III.Section IV provided a brief description of the metric employed for evaluation followed by the detailed presentation of experimental results.Finally, the paper is concluded in Section V.

II. RELATED WORK
In this section, we review the works proposed previously in the domain of tone-mapping used for converting HDR images to the LDR counterparts.In this context, we classify the approaches and highlight the weaknesses and strong points of each approach.
Tone-mapping operators target a range of objectives while converting from HDR to LDR images, such as better visibility, natural color appearance, preservation of scene details, and photographic look, among others.The approaches for tonemapping can be classified into two main groups, global and local.The main characteristics of the global operators is that they preserve the relative order of pixel luminance values by using a monotonically non-decreasing mapping curve.On the other hand, local operators use neighborhood features to determine the value of LDR luminance, and therefore the relative order of pixel intensities is not necessarily maintained.Local operators can show more details in the LDR images, but they are prone to artefacts which can affect their visual appeal [20].Table I summarizes the difference and highlights the advantages of each group.Some interactive tone-mapping techniques have also been developed.They determine areas of interest using different methods such as the position of cursor or gaze of user.The authors of the work [21] proposed a method that enables users to define some boundaries on a given image and controls the details of the defined area by adjusting the tonal values.A tool designed for interactive display was presented in [22], where the main objective is to grant users the control on the level of contrast.A system presented in [23] detects the area that falls in the user's gaze by using an eye tracker and uses the contents in the area to tune the tone-mapping parameters in real-time.The system mimics the adaptation mechanism of the human eye to a wide range of brightness levels.A similar work in [24] also relies on the user's gaze, but the final objective here is to improve the performance, i.e., the speed of operation.
A biological retina model is used in [25] for tone-mapping.The main feature of this work is the use of spatial-temporal filtering for two main purposesreducing the noise and enhancing the temporal stability.In [26], a median based approach is proposed.The key goal is to make localized environmental adaptation to generate image appearance calibration.Targeting the video domain, the authors of [27] provided a post-processing approach, where the final objective is to ensure temporal stability for static TMOs.In the time domain, Ferwerda et al. [28] presented a visual adaptation model for tone mapping.They depend on visibility, color appearance, and sensitivity, which are adjusted according to some thresholds.Aiming at saving contrasts and details, the authors of [29] provided a scene dependent basis, where compression of luminance is employed within the scene.Based on both the photographic look and photographer response respectively, the authors of [30][31] provided some new methods that use human eyes' sensitivity to achieve tone mapping.

Difference
Rely basically on the value of luminance only for the mapping process.
Take properties of the neighboring pixels also into consideration in the mapping process.

Main advantage
Use a non-decreasing monotonic curve in the mapping process, and thus they are computationally efficient.
Perform well when it comes to reproducing fine details.

Main disadvantage
Important details can be missed out in the process of mapping to LDR.
Enhancement of details often leads to creation of visible artefacts.www.ijacsa.thesai.org Since the human visual sensitivity follows the Gaussian distribution, Kim and Kuatz [32] proposed an average of scene's log luminance-based method to perform the tone mapping.Within the two-dimensional and three-dimensional spaces, the authors of [33] modeled the tone mapping curves using the normalization of the RGB color space to mitigate the distortions in the color contrasts.A good level of improvement was achieved in the tone-mapped images in [34][35].The key idea of enhancement in [34] is relying on the preference model of human viewer, while in [35] the improvement is achieved through utilizing weighted least square filters and artificial intelligent method, i.e. a neural network, to preserve colors while shifting the luminance.The authors of [36] proposed a psychophysical based TMO method, which depends on both the color appearance and visual acuity.As for the response, this method is relying on quantifying of threshold visibility.In a similar method, the authors of [37] proposed a mapping algorithm that depends on both the appearance and the psychophysical model.The main difference between the two is that in [37] smoothing filters are used to simulate the adaptation.The goal of the approach presented in [38] is to preserve the contrast in the HDR image.This is linked with temporal variation problem, which is solved by using filters.The authors of [18,39] used an adaptation model of the human eye to the luminance levels.A histogram of non-uniform bins was constructed and used for tone-mapping which produced results better than traditional histogram-based methods.
Artificial Intelligence (AI) is employed in the domain of converting HDR images to LDR ones.In this context, some other recent tone-mapping algorithms proposed for better visual quality using different techniques such as deep learning and various HVS models, and some implementations on the hardware for real-time performance, as described below.
The researchers in the work [40] proposed a method that tries to solve the mapping problem by keeping the pixel counts in histogram bins within determined lower and upper ranges.
Building of histogram can be a slow manipulation when image size is huge.In this context, Scheuermann et al. [41] presented an effective programming of histogram construction on the graphical Processing Unites (GPUs) and applied this for to tone-mapping algorithm of Larson et al. [40].In addition, Khan et al. [42] executed their algorithm on the GPUs and provided a real-time performance report even for the images of huge sizes.Moreover, Ambalathankandy et al. [43] executed a local histogram equalization method for tone-mapping on FPGA with poor specifications (i.e., small memory and minimal data access requirements).
Recently Rana et al. [44] provided a Deep Convolutional Neural Network (DCNN) for tone-mapping of HDR images.The researchers gathered a large group of tone-mapped images generated by a number of existing TMOs.For each input HDR image, the outputs of all TMOs were evaluated (in terms of comparison) relying on goal quality index.Then, the one that got the best band was involved in the training phase.The deep TMO trained depending on the previous gathered set of images would supposedly learn the best characteristics of all TMOs.This reflects that the perform is better than the existing TMOs.The experimental results showed by the researchers seem to ensure this assumption.That is because the deep TMO got the best average quality score during the testing phase.

III. PROPOSED OPTIMIZED HISTOGRAM-BASED APPROACH (OHBA)
In this section, we introduce our proposed approach called Optimized Histogram-based Approach, OHbA, which takes an HDR image as input and converts it into an LDR image.Our proposed approach relies on optimization of the initial solution provided by the ATT algorithm.The initial histogram of the input image constructed by the ATT method is optimized to generate an LDR image of high quality.The flowchart given in Fig. 3 illustrates the steps of the proposed approach which are explained below in detail.

A. Histogram-Based Tone-Mapping
Recently, a new histogram-based tone-mapping algorithm, referred as ATT by the authors, has been proposed modifying an earlier histogram-based design [53], and it outperformed the existing state-of-the-art methods in several subjective and objective studies reported by the authors [18].Our proposed optimization algorithm takes the transformation curve of ATT as one of the initial solutions and uses DE to improve it for better results quality.The reason behind selecting this histogram-based algorithm is that it does not require any userdefined parameters or human intervention.Our optimization module iteratively improves the quality of results by refining the histogram bins, thus changing the clustering of the HDR pixels and hence their corresponding LDR values.
The ATT algorithm starts by constructing the luminance channel from the RGB channels of the HDR image as: Then, it designs a suitable tone-mapping curve based on histogram of this channel.The bins of the histogram are formed based on the TVI curve-based sensitivity model of the HVS [40].The TVI curve describes sensitivity of the human eye as function of the environmental light conditions and can be used to calculate just noticeable difference (JND) under different viewing conditions.A JND is the minimum difference in brightness in the given viewing environment that would be noticeable to the average human observer.ATT determines widths of the histogram bins such that the luminance range spanned by each of them is the same if measured as the number of JNDs it covers.After constructing the histogram, the tonemapping curve is designed based on a scaled version of the cumulative histogram in [0, 255] range.Histogram-based methods are known to suffer from excessive compression of low-density clusters of pixels and exaggerated enhancement of high-density clusters.The ATT algorithm refines the bin counts, again using the TVI model of HVS, to resolve these issues.This step determines the bins that are visually richer in terms of the number of perceptually distinguishable pixels and assigns them additional weights to ensure that the display levels are not allocated to the bins just based on the density of pixels.
The tone-mapping function of ATT can be represented by an LUT of 2 columns.The first column contains the HDR values H at the boundaries of the histogram bins, while the corresponding LDR values L are placed in the second column.These are essentially the values of the normalized cumulative histogram at the bin edges.It should be mentioned that traditionally the histograms use bins of equal width.However, as mentioned above, the ATT method forms non-uniform bins using the TVI model of the sensitivity of human visual system.As a result, clusters of visually similar pixels are formed and mapped to same or nearly same LDR values, generating visually pleasing output images.
The HDR and LDR pairs, H and L, in the LUT are the initial guess of tone-mapping parameters as indicated in the flowchart of our tone-mapping structure shown in Fig. 3.The process of linear interpolation with the LUT is used to tonemap the HDR image to LDR.Through the DE optimization process, stated next in this section, we iteratively modify the LUT.To simplify the process, we modify the initial LUT produced by the ATT such that the LDR values L are fixed at {0, 1, 2, …, 255} and the corresponding HDR values H are found through linear interpolation.In the modified LUT, the DE algorithm need to change the vector H only, which would mean changing the ranges of the histogram bins and hence changing the tone-mapping function.

B. Differential Evolution
Differential Evolution [54,55] is one of the most robust population-based evolutionary algorithms which improves, via an iterative process, a group of candidate solutions, called a population, through classical evolutionary operators, namely, selection, crossover, and mutation.As explained in the pseudo code (Algorithm 1), in every iteration of DE, the best solutions are selected for next iteration.Then, this group of candidate solutions is utilized for the crossover and mutation to build new candidate solutions (offsprings).This intelligent and iterative stochastic process continues until reaching the stop criteria, i.e., the maximum number of iterations, or the minimum level of desired quality in one or more offsprings.
A large number of well-known optimization problems have been solved by DE.It has been used in applications such as privacy protection, high performance computing, image processing, security and data mining tasks to name a few.An extensive survey of DE can be found in [56].In this work, we use DE to optimize the H vector obtained from the ATT design mentioned above, which is passed to the DE optimizer as one of the initial solutions.Algorithm 1 outlines the pseudo code of the proposed DE.The initial solution H is used to improve the tone-mapping of the given HDR image, and the quality of the LDR image, used for comparison purposes by DE, is measured using the Tone Mapping Quality Index (TMQI) [57], which is widely used for this purpose in the existing literature and is described later in this paper in more detail.The vectors H are evaluated at every iteration by DE using the measured TMQI values.Each iteration modifies the vectors H such that the TMQI values of the produced LDR images are ideally larger (better) than the values obtained in the previous loop.
The optimization problem, in this work, is composed of 255 variables representing the range of every bin.The problem can be modeled as a maximization problem of the quality LDR image as shown below. s.t.
where x i is the upper boundary of bin   defined by H.
It should be noted that the bin edges in vector H are sorted floating-point values in ascending order.The original DE algorithm does not require the vector to be sorted; however, we modify it to enforce this constraint.This reduces the search space of the optimizer which leads to a faster convergence.Minimum and maximum values of H are known for a given HDR image and these constrained are also applied.www.ijacsa.thesai.orgBased on our preliminary experiments we have found that the following parameters settings for the DE algorithm are more appropriate for the studied problem: The iterative optimization process mentioned above incrementally enhances the quality of the output images in each iteration.In most of the cases, the enhancement beyond 200 iterations is minimal.Therefore, we set the stop condition to 200 iterations or the minimum desirable value of TMQI, whichever is achieved first.Fig. 4 shows the progressive improvement in the quality (TMQI index) of the LDR images for 10 different HDR test images described in Table II.A gradual improvement over iterations of every image is represented by a distinctive curve in the figure.

IV. EXPERIMENTAL RESULTS EVALUATIONS
In this section, we show experimental results comparing the proposed design with some state of the art existing TMOs.We present quantitative evaluations using the TMQI metric which is also explained briefly in this section.

A. Tone-Mapping Quality Index
For evaluation purposes, we rely on the metric called tonemapped image quality index, TMQI.The TMQI metric is used to measure the structural similarity as well as the naturalness, which in turn quantify the quality of the output images in [0, 1] range.The higher value of the TMQI means a higher quality and vice versa.The naturalness index of TMQI gives useful information about the correlations between image naturalness and different image attributes [57].It is defined by the following formula.

𝑁 =
(  ×  )  (4) where, P G and   are the Gaussian and the Beta probability density functions, respectively. is the normalization factor.As for structural similarity index, TMQI calculates the local similarities between corresponding patches  and  of HDR and LDR image pairs using the following formula.
where,   .  .   are the local standard deviations and the cross correlation between the corresponding HDR and LDR patches. 1   2 are the positive stabilizing constants.
The overall TMQI is the combination of the previous two metrics and is defined as.
where, 0 <  < 1 adjusts the relative importance of the two components (i.e., naturalness and structural similarity), and  and  determine their respective sensitivities.

B. Comparitive Studies
For comparison, we selected three representative tonemapping algorithms.The global TMO by Reinhard et al. [26] is the most well-known and extensively used algorithm which has been shown to produce natural looking results.ATT [18] is a recent algorithm which was used as the initial solution for our optimization loop.ATT outperformed all other algorithms in the extensive objective and subjective studies reported in [18].The algorithm by Liang et al. [58] is a recent algorithm and authors have reported high quality results outperforming many existing methods.The experiments reported below were conducted on a computer equipped with Intel i5 1.4 GHz CPU and 4 GB 1600 MHz DDR3 RAM, running on OS X Yosemite.
In Tables II and III we show the TMQI scores obtained by the output images produced by these methods.The 10 test images used in Table II are encoded in the well-know HDR format, RGBE, and are picked randomly from the public dataset available at [59].The images used in Table III are the complete dataset of the HDR images taken from the accompanying DVD of [60] encoded in another famous HDR format, OpenEXR.It can be seen that for both sets of images, the proposed TMO obtained highest scores compared to the existing state of the art methods.
As shown in Table II, the proposed OHbA method achieved the most accurate results for 9/10 images and remained at the second position for one image by a small margin.In Table III, the OHbA achieved the most accurate results for 7/10 images and remained at the second position for remaining 3 images by a small margin.In all the cases, the OHbA outperformed the original solution generated by the ATT which shows the efficiency of the optimization methods and its benefits in HDR to LDR conversion applications.www.ijacsa.thesai.org

V. CONCLUSION
The images of high dynamic range cannot be viewed with full details on the existing displays, because the range of the screens is much less than of the captured images.In other words, the problem of converting high dynamic ranges images into low dynamic range images is pressing, to enhance the visualizing experience taking advantage of the advancements in imaging technologies.Many approaches were proposed previously to deal with this issue.However, artificial intelligence was not used in this domain to the best of our knowledge.Based on the DE algorithm, which forms the base of optimization of the output images' quality, we propose the optimized histogram-based approach (OHbA).The OHbA enhances the quality of images for all the studies cases.Compared with an existing state of the art tone-mapping algorithms, the proposed approach shows better performance based on the quality metric called tone-mapped quality index (TMQI).

Fig. 2 .
Fig. 2.An HDR Sunny Scene Captured with different Exposure Settings of a Standard Camera.

Fig. 3 .
Fig. 3. Flowchart of the Proposed Approach for Designing an Optimizationbased Tone-Mapping Operator.

Fig. 4 .
Fig. 4. DE Quality Improvement Over Time for the 10 Images Obtained from the ref [59].
If the candidate is better than the parent, the candidate replaces the parent.If the parent is better than the candidate, the candidate is discarded.Otherwise, the candidate is added in the population.2.2.If the population has more individuals than NP, truncate it.end.

TABLE II .
TMQI SCORES USING RGBE TEST IMAGES.THE HIGHEST SCORE FOR EACH IMAGE IS SHOWN IN BOLD FONT

TABLE III .
TMQI RESULTS USING OPENEXR TEST IMAGES.THE HIGHEST SCORE FOR EACH IMAGE IS SHOWN IN BOLD FONT.