Novel Approach to Estimate Missing Data Using Spatio-Temporal Estimation Method

With advancement of wireless technology and the processing power in mobile devices, every handheld device supports numerous video streaming applications. Generally, user datagram protocol (UDP) is used in video transmission technology which does not provide assured quality of service (QoS). Therefore, there is need for video post processing modules for error concealments. In this paper we propose one such algorithm to recover multiple lost blocks of data in video. The proposed algorithm is based on a combination of wavelet transform and spatio-temporal data estimation. We decomposed the frame with lost blocks using wavelet transform in low and high frequency bands. Then the approximate information (low frequency) of missing block is estimated using spatial smoothening and the details (high frequency) are added using bidirectional (temporal) predication of high frequency wavelet coefficients. Finally inverse wavelet transform is applied on modified wavelet coefficients to recover the frame. In proposed algorithm, we carry out an automatic estimation of missing block using spatio-temporal manner. Experiments are carried with different YUV and compressed domain streams. The experimental results show enhancement in PSNR as well as visual quality and cross verified by video quality metrics (VQM).


INTRODUCTION
With enhancement in wired and wireless networks, more and more users are demanding video services, including video conferencing and video streaming over the internet.However, the Internet does not provide guaranteed quality of service (QoS).Loss of data packets occur due to traffic congestion [1].In wireless networks, packet loss happens frequently due to shadowing, multipath fading, and noise disturbance of wireless channels [3].Video transmission uses compressed video streams for transmission so that video data can be transmitted even with poor network bandwidth situations [2].A loss of packet over transmission in compressed stream introduces severe distortion because the compression algorithms use spatial estimation methods and temporal to improve compression efficiency.Therefore a single distorted block within a frame may occur errors not only in present frame but also propagate error over several frames.Many decoder error concealment techniques and error resilience have been proposed to control amount of error in reconstructed frame [4].A simple error resilience approach is to use feedback channels and request for retransmission whenever there is error.This is the most prosperous technique and the recovered data would always be correct.However, it involves halting decoding process till error block of data is received again.This is an inefficient approach in terms of delay involved in process.Another way to avert errors is to embed error checks in encoded video bit streams and transmit over the channels.This method though bypasses retransmission of video; it affects compression efficiency of the encoder and thus increased usage of network bandwidth.
Hence, in this paper a post processing algorithms on the decoder side are proposed for error concealment.The preference with decoder error concealment is that it does not require any change in encoding or decoding process.It simply appends a post processing block which retrieves erroneous data.Hence, there is no increase in bit rate or delay.Fig. 1 shows block diagram for process in which packet loss occurs in channel and video sequence to recover loss of macroblocks.Therefore these methods can be used in real time video applications like video-voice over internet and streaming applications.
The complete video coding system can be organized according to the blockdiagram in Fig. 2 The building blocks of the video coding system including post processing at the decoder are summarized below which give complete idea about video coding system [17].
 Video Acquisition-Source of the video sequence which is output in a digital form.Following are the processing steps.
 Pre-Processing-Operations on the raw uncompressed video source material, such as color correction, or denoising trimming, color format conversion. Encoding-The aim of encoding is to generate a compact representation of the input video sequence which is applicable for the transmission method in the given application scenario.
 Transmission-Transmission section includes sending and delivery of the video data to the receiver side. Post-Processing-Post-processing perform operations on the reconstructed video sequence for enhancement or for adaptation of the sequence for display.These operations can e.g.include trimming, or re-sampling and color correction.
 Display-Display use for presentation of the video sequence for viewing.In real applications, example multiple pre-processing /post-processing steps in conjunctions with iterative de-and re-encoding and transmission may be used.
Therefore in this paper, we propose an approach based on combination of spatial and temporal concealment using spatial smoothing filter and wavelet transform for missing block estimation.Spatial smoothing is used to estimate missing block using spatial information present in the frame as, most of the information lies in low spatial frequency.And to add details we used wavelet based bidirectional estimation of high frequency information.The wavelet transform is used because it has ability to represent information in logarithmic way (low to high details) and hence can be used to estimate details (edges) separately.
The rest of the paper is organized as follows, Section 2 reviews the theory behind the algorithm, section 3 Theory based on proposed new algorithm to improve error concealment, section 4 shows proposed algorithm section 5 shows simulation results and section 6 concludes the paper.

A. Forward Error-Concealment Technique
There are different error resilience techniques: forward, concealment, and interactive techniques [18].Almost all forward techniques boost the bit rate since they add redundancy to data.Few of them require modification to the encoder.Interactive methods need a feedback channel between the encoder and the decoder.Interactive methods will also announce some delay and may, therefore, be improper for real-time applications like video communications.On the other hand, concealment techniques do not boost the bit rate, do not require any modifications of the encoder, and do not boost any delay.This makes them a very alluring choice for video communications [19].An error concealment method plays primary role at the encoder side.When the transport channel is not lossless, there are two kinds of distortion observed at the decoder.The first one is quantization noise imported by the waveform coder and the second is the distortion due to transmission errors.Both quantization and transmission errors are minimized by using optimal pair of transport coder and source coder such as transport protocol, packetization and FEC.Quantization error is minimizing by video codec given in available bandwidth.Shannon proposed separation theorem useful for source and channel that are memeoryless and stationary [20] and later extended to more general class of source and channel [21].Joint design of source and channel coder achieves better performance.To accomplish forward error concealment they all add redundancy either in source coder and transport coder.All these techniques achieve error resilience by adding certain amount of redundancy such as layered coding, multiple description coding, robust waveform coding, robust entropy coding.Summary of Forward error concealment techniques proposed Y.Wang [10].It explains Layered Coding with prioritized transport includes frequency domain partitioning, Successive amplitude refinement and spatial/temporal resolution refinement.Note that these techniques are not mutually exclusive; rather, they can be used together in a complementary way [10][11][12][13][14][15][16].

B. Error Concealment by Post-processing /Error Reconcealment
The loss of transmitted data packets affects the quality of the received video and this problem is caused by the limited channel bandwidth used by the mobile communication networks.It is not possible to retransmit lost packets in real time application.Generally post processing techniques is used to reduce the visual artifacts caused by bit stream error [22].Error concealment methods which will be implemented on the receiver side to restore the missing and corrupted video frame using the previously decoded video data [4].It will be noted that various post processing technique are successfully implemented such as Motion compensated temporal prediction, Spatial Interpolation, Maximally Smooth recovery, POCS, Fuzzy logic which is review by Y.Wang [10][11][12][13][14][15][16].www.ijacsa.thesai.orgIII.
THEORY BASED ON PROPOSED ALGORITHM In proposed algorithm we estimate the missing block using hybrid approach.Wherein, we first decompose the video frame using wavelet transform.To estimate the missing data we apply smoothing function on approximate wavelet coefficients and use bi-directional prediction on the detail wavelet coefficients.The reason behind using bidirectional wavelet prediction is, the detail information (edges) cannot be approximated and needs to be predicated using the available details in neighborhood frames.Once spatio-temporal predication is done we reconstruct the frame using inverse wavelet transform.
In this section we quickly revisit the basics of wavelet transform and spatial smoothing.

A. Wavelet transform
Wavelet transform have a wide range of applications ranging from analysis of image signal to data compression [5].In general wavelet transform of time varying signal x(t) is calculated by taking inner product of signal against family of wavelets.These wavelets, are labeled by scale and time location parameters a and b.In continuous wavelet transform, the wavelet corresponding to scale a and time location b is given as,

√ ( )
Where is a wavelet prototype function which can be thought of as a bandpass function.The continuous wavelet transform is given by,

{ } ∫
Where denotes complex conjugate of .Time t and time-scale parameters a, b vary continuously.Time remains continuous but time-scale parameters and sampled on a dyadic grid in time-scale (a, b) space.This is defined as,

{ }
The wavelets in this case are

⁄
The original signal can be recovered through
The discrete wavelet transform (DWT) corresponds to continuous wavelet transform of a sampled sequence , where T is a sampling period.The discrete wavelet transform applies to discrete time signals where both time and scale parameters are discrete.The DWT is represented by Where denotes complex conjugate of .The residual coefficient at J are given by ∑ Where is the analysis scaling sequence.It is used to bring input signal from initial scale to .One of the main concept of wavelet theory is the interpretation of wavelet transform in terms of multiresolution decomposition.The input signal x(n) is decomposed into approximate and detailed coefficients using a set of low pass(H) and high pass(G) filters followed by a decimator.These filters are quadrature mirror filters and are related by, where M is filter length.

B. Smoothing function
In statistics and data analysis smoothing function is used to reduce noise or small scale information while keeping most imprints of the datasets [6,7].Mathematically, noisy data can be represented as follows, (1) Where represents Gaussian noise with zero mean and unknown variance and is the mean of the signal.Accuracy of estimation of depends upon accuracy of .Also, is supposed to be smooth function that is derivatives of of particular order generally greater than 2 are continuous.Smoothing of relies upon smoothing of .Here penalized least squares approach for smoothing of data is used.Mathematically, it can be expressed as, (2) Where, RSS is residual sum of squares which is expressed as, is the penalized term, and S is scalar which indicates degree of smoothing, with increase in smoothing www.ijacsa.thesai.orgparameter, degree of smoothing also increases.In [6], it is explained that term can be expressed as, (3) Where D is a tri-diagonal square matrix defined as,

Where represents step between
To correctly estimate smoothing parameter, we minimize equation ( 2) with constraints of equation ( 3).Hence, smoothened data can be obtained as, (4) Where is identity matrix and is transpose of .
As discussed above we formulate the missing block estimation problem in approximate wavelet sub band.We treat the approximate coefficients of missing block in the given frame as missing data and hence estimate using smoothing given in equation ( 4).
Effectively the concealments in done in low frequency and high frequency band separately.Low frequency coefficients are estimated as spatial smoothing (using missing data estimation given in section 3.2 and high frequency coefficients are predicted using high frequency bands of previous and next frames).

C. Error concealment
Temporal error concealment methods assume that motion in the video is not dramatic (Sum of Absolute difference between consecutive frames is less) and do follow some linear or quadratic model.We first detect the video shot boundary to detect the scene change.If the block loss happened in the frame is not the key frame then we can use bidirectional predication and else we restrict the algorithm as a unidirectional prediction.Literature gives multiple algorithms for the video shot boundary detection [8,9].We use correlation between consecutive frames as a measure for shot boundary detection,

∑
Where, N is the size of cross correlation vector window use for calculation.
If the block loss happen in the frame which is not a key frame (frame after scene change) but having second frame after key frame then we use the first order bidirectional prediction on detail wavelet coefficients as, Where, represent lost block and is frame index.IB is the low pass band in wavelet transform.
The high frequency components of the wavelet transformed image are restored using linear combination of reference frames.
Where HB represents high frequency component of wavelet transformed image.
Therefore, to we propose a technique which uses multiple reference frames to recover frame data.Following figure explains proposed algorithm.PROPOSED ALGORITHM The proposed algorithm is as follows.We use previous and next frame as reference frames 1. Read frames from incoming video streams.2. Detect the video shot boundary.3. Decompose frames using wavelet transform.4. Estimate approximate wavelet coefficients (low frequency information) of transformed frames using robust smoothing of gridded data in one or higher dimension algorithm given in section 3.2. 5. Estimate detail coefficients (high frequency data in the current frame) by bi-liner predication given in section 3.1 and 3.3.6. Restore the frame using inverse wavelet transform (with estimated low frequency coefficients using spatial smoothing and predicated high frequency coefficients using bidirectional predication of detail wavelet coefficients).
Following are the block diagram of proposed algorithm www.ijacsa.thesai.orgThis section describes simulation results for the proposed algorithm.The following assumptions are made while conducting experiments, video sequences used for video error concealment are of QCIF and CIF resolution at 30fps, YUV format and MP4, AVI videos for compressed video error concealment.The experiments were conducted on foreman and akiyo video sequences.For our experiments we selected the values of linear coefficients as follows, These parameters are chosen empirically and tuned to achieve the maximum reconstruction quality of missing blocks.We use Daubechies 1 wavelet for testing purposes [6].
We implemented for QCIF (176 X 144) and CIF (352 X 288) different video sequences in which error block is reconceal by our proposed algorithm.Video Quality Metric is tool to measure quality of video effectively.PSNR_SEARCH is function VQM tool to calculate PSNR and video quality results effectively.We have also cross verified results using the Video Quality Metric Tool.We test various video sequence formats QCIF (172 X 144) for various video scene Carphone, Highway and Forman sequences and CIF (352 X 288) for various test sequences Akiyo, Waterfall and Tempete sequences.We have used PSNR_SEARCH measure for all the test video mention above.We have observed improvement in PSNR in Table I.To show the comparative study we have used the algorithm given in [4], as it is conceptually similar to the proposed algorithm.Fig. 18.Shows the concealment results for

VI. CONCLUSION
In this paper, we propose a low complexity approach for bi-directional spatio-temporal error concealment.The proposed method exploits the information from multiple reference frames as well as information available in current frame to recover lost block of data.We have applied the method on Carphone, Highway, Forman, Akiyo, Waterfall and Tempete video sequence and also on compressed AVI video.Proposed Algorithm does not require very complicated computation and hence usable for various application.Results clearly indicate that the method outperforms existing methods in terms of PSNR as well as cross verified by video quality metrics (VQM) and visual quality.

Fig. 1 .
Fig. 1.Block diagram of process in which packet loss occurs in channel

Fig. 2 .
Fig. 2. Block diagram of video coding system including post-processing at the decoder In transmission packet loss occur in video.In fig.2 loss occur in frame while transmission due to lossy network or compression of video sequences. Decoding-Encoding usually use compression to achieve the target transmission bitrate constraints due to which losses occur.The decoded video constitutes a resemblance of the original source video.If inaccessible transmission losses have occurred, the decoder implements concealment strategies to recover the corrupted video sequence as much as possible.

Fig. 3 .
Fig. 3. Computation of DWT In our algorithm we use wavelet decomposition and then use prediction on detail coefficients to estimate details of missing blocks (high frequency information in form of edges).

Fig. 6 .Fig. 6 .
Fig.6.shows visual result QCIF (176 X 144) Carphone video sequence at resolution 30 fps which having 380 frames in video shows a) Original Carphone video frame b) It shows corrupted video frame in which 20Macroblock is lost randomly in frame c) It shows Reconceal video frame with help of proposed method

Fig. 12 .Fig. 7 .Fig. 8 .
Fig.12.shows visual CIF (352 X 288) Akiyo video sequence at resolution 30 fps which having 380 frames in video shows a) Original Akiyo video frame b) It shows corrupted video frame in which 20Macroblock is lost randomly in frame c) It shows Reconceal Akiyo video frame with help of proposed method.

Fig. 14 .
Fig. 14. Results shows CIF Tempete Video Sequences at resolution 30 fps showing Original video, Error block video and Reconceal video Re-concealed Frames "Tempete" Sequence www.ijacsa.thesai.orgone frame using reference and proposed algorithm.Fig.19Shows the PSNR comparison.

Fig. 18 .Fig. 19 .
Fig. 18. Results for Compressed Sequence (Bottom right is proposed and bottom is as per algorithm in [4])

TABLE I
Fig. 15.Error Frame Index Vs PSNR for 'Akiyo' Sequence Fig. 16.Error Frame Index Vs PSNR for 'Waterfall' Sequence Fig. 17.Error Frame Index Vs PSNR for 'Tempete' Sequence