Speech-to-Text Conversion in Indonesian Language Using a Deep Bidirectional Long Short-Term Memory Algorithm

Now-a-days, speech is used also for communication between humans and computers, which requires conversion from speech to text. Nevertheless, few studies have been performed on speech-to-text conversion in Indonesian language, and most studies on speech-to-text conversion were limited to the conversion of speech datasets with incomplete sentences. In this study, speech-to-text conversion of complete sentences in Indonesian language is performed using the deep bidirectional long short-term memory (LSTM) algorithm. Spectrograms and Mel frequency cepstral coefficients (MFCCs) were utilized as features of a total of 5000 speech data spoken by ten subjects (five males and five females). The results showed that the deep bidirectional LSTM algorithm successfully converted speech to text in Indonesian. The accuracy achieved by the MFCC features was higher than that achieved with the spectrograms; the MFCC obtained the best accuracy with a word error rate value of 0.2745% while the spectrograms were 2.0784%. Thus, MFCCs are more suitable than spectrograms as feature for speech-to-text conversion in Indonesian. The results of this study will help in the implementation of communication tools in Indonesian and other languages. Keywords—Speech-to-text; Deep Bidirectional Long ShortTerm Memory (LSTM); spectrogram; Mel frequency cepstral coefficients (MFCC); word error rate


I. INTRODUCTION
Speech is a longitudinal wave that propagated through a medium, which can be solid, liquid, or gaseous [1]. Humans utilize speech as a primary component of communication to exchange information. Today, humans communicate also with computers; generally, this communication requires the conversion of speech into text [2]. This process involves various stages of conversion and outputs data consisting of numbers that can be processed by a computer into text [3]. Speech-to-text conversion can be implemented in various applications, such as communication tools for deaf people [2], smart homes [4], and translators [5].
Some studies have investigated speech-to-text conversion in various languages. Ahmed et al. utilized a hidden Markov model (HMM) for English and Arabic speech recognition [6]. Hotta [7] and Othman [8] performed speech-to-text conversion using neural networks in Japanese and Jawi, respectively. Kumar et al [9] used a recurrent neural network (RNN) for speech-to-text conversion in Hindi, and Laksono et al. [10] used connectionist temporal classification (CTC), which is usually applied on top of an RNN, for speech-to-text conversion in Indonesian and Javanese. Abidin et al. presented an approach to obtain Indonesian voice-to-text data set using Time Delay Neural Network Factorization (TDNNF) [11].
Mon and Tun [12] proposed the HMM method, which uses Mel frequency cepstral coefficients (MFCCs) as features. Because they used a large dataset of English words, the HMM was ineffective owing to the high probability of similarity between words. Zhang [13] used a combination of the deep neural network (DNN) and HMM model for English speech recognition and showed that DNN-HMM was superior to the traditional Gaussian mixture model (GMM)-HMM method. Nevertheless, it still had low accuracy. Liu et al. [14] had shown that the RNN together with Long Short Term Memory (LSTM) improved the performance of speech recognition on the ChiME-5 dataset. Meanwhile, Wu et al. [15] and He [16] utilized RNN-LSTM for Chinese dataset, and the accuracy of speech recognition was improved.
Most studies on speech-to-text conversion were limited to the conversion of words or incomplete sentences from a dataset, and very few studies considered speech-to-text conversion in Indonesian. Laksono et al. [10] used DNN and CTC with MFCCs as the features for speech-to-text conversion in Indonesian and Javanese with a small number of Indonesian and Javanese words. However, the result showed low accuracy for both Indonesian and Javanese; thus, they might not be suitable for speech-to-text conversion.
In this study, we perform speech-to-text conversion in Indonesian using a deep bidirectional long short-term memory (LSTM) algorithm. We determine the features suitable for the deep bidirectional LSTM and consider complete sentences consisting of subject, predicate, object, and adverb spoken by some respondents.
The rest of this paper is organized as follows. In Section 2, the research method used in this study is presented. Section 3 reports and discusses the results. Finally, the paper is concluded in Section 4.

A. Data Collection
The speech data were obtained from ten speakers (five males and five females). Every speaker uttered ten sentences in Indonesian consisting of a subject, predicate, object, and adverb, as presented in Table I. Each sentence was uttered 50 times; thus, a total of 5000 sentences were recorded. Data were manually divided as follows: 70% for training, 20% for validation, and 10% for testing. Thus, 3500 training, 1000 validations, and 500 testing data were obtained. The data were recorded in the Control and Robotics Laboratory, Universitas Sriwijaya. Fig. 1 shows a block diagram of the proposed speech-totext conversion process. The speech is recorded using a FIFINE K669B microphone with a sampling frequency of 16 kHz. Speech data undergo the preprocessing stage, which involves normalization, silence removal, and pre-emphasis, to correct the speech signal by reducing noise and removing the silence area on the speech signal. Then, the speech features are extracted into spectrograms and MFCCs. The features are fed to the deep bidirectional LSTM to determine the probability of each label. In the deep bidirectional LSTM training process, CTC is used to determine the loss. Subsequently, the network performs the decoding for the process of labeling from the output of the deep bidirectional LSTM network and language model obtained from the Kompas newspaperFinally, text if obtained as the output.

C. Evaluation
The word error rate (WER) is used to determine the percentage of success of speech-to-text conversion. It is determined by calculating the number of insertions, subtractions, and substitutions of the word used to convert speech into text as follows: where (ℎ( )) is the number of insertions, subtractions, and substitutions of the word in the target sentence and represents the total words in the reference which were actually said [17].

A. Preprocessing Signal
The preprocessing stage involved normalization, silence removal, and pre-emphasis, which were performed using the Python library Pyrus. Normalization is performed by dividing the data in the speech signal by the maximum value of the amplitude to equate the amplitude of the speech signals.
Owing to the recording process, the speech signal may have different intensities and consequently, different amplitude values. Silence removal is performed to determine the silence area to be erased on the speech signal. Finally, pre-emphasis is important to remove the noise while maintaining the frequency of the speech signal. show the speech signal before and after the preprocessing stage, respectively, displayed using the audacity software. From the figure, it can be seen that before preprocessing, the speech signal has an amplitude of less than 0.5 and silence areas are at the beginning and end of the speech. On the other hand, after preprocessing ( Fig. 2(b), the amplitude of the signal is approximately 0.5, which is the ideal value for speech signals [18], and the silence areas are smaller than those before the preprocessing stage; the duration of the speech signal changes from 3 to 2.4 s. Furthermore, the noise in the speech signal was reduced by using a high pass filter to eliminate speech signals with frequency below 250 Hz.

B. Feature Extraction
Using the contrib audio library of TensorFlow, we extracted the log power spectra, i.e., spectrograms, and MFCCs as features and determined which is more suitable as input to the deep bidirectional LSTM.
To obtain the spectrograms, the preprocessed speech signal was divided into sections with window lengths of 32 ms and window steps of 16 ms. A fast Fourier transform (FFT) was performed to convert the speech signal from the time domain to the frequency domain. 512 frequency bins were used, and only half of the frequency bins plus one (257 bins) were used. Then, the log power spectra, which were the density of the FFT spectra, were used as input for the training process. The visual representation of log power spectra is known as spectrogram. Fig. 3 shows an example spectrogram. As shown in the figure, an x-axis shows the time length and a y-axis is the power spectrum of the speech signals.  The MFCCs are generated using the result of the spectral density, which is filtered with a Mel scale and filter bank to obtain the energy at each point. The resolution of the Mel filter bank was 40 with lowest and highest frequencies of 0 Hz and 8 kHz, respectively. The Mel spectrum, which is the output from the Mel filter bank, is converted into the time domain using a discrete cosine transform (DCT) with a coefficient of 13. The output from the DCT process is called an MFCC plot. Fig. 4 shows an example MFCC plot.

C. Language Model
A language model is used to perform the decoding process on the network output for the speech-to-text conversion process. The decoding process is performed by calculating the probability of appearance of each word based on the exact word order [19]. This probability is calculated from the word chunks based on word order in the N-grams model. The language model can be built using Corpus text with a large amount of data, e.g., words and sentences in a newspaper. Accordingly, in this study, Corpus text in .txt format derived from the Kompas newspaper [20] was used. The Kompas newspaper has published more than 5000 articles, which were merged to create the language model. Before using, the Corpus text is sorted in alphabetical order from A to Z, and invalid data such as space and blank () are removed.
A 5-gram language model was built with outputs in lm.arpa format using the KENLM library. The lm.arpa output was transformed into binary format to be processed and read by the computer. Then, a trie that works by tracking the minimum probability for the word prefix was created as a data structure to assist in using the memory to construct the language model.

D. Training Dataset
The DeepSpeech library was used in the training process of the deep bidirectional LSTM algorithm. This algorithm consists of a combination of bidirectional RNN and LSTM, commonly known as bidirectional LSTM. The bidirectional LSTM exploits long-range context dependencies in the past (t − 1) steps and future (t + 1) steps. This algorithm has a deep architecture, which can performe high-level representations of acoustic data [21]. The DeepSpeech architecture consists of the input layer derived from the extracted features, i.e., spectrograms and MFCCs. Then, there are five stacked hidden layers: three linear hidden layers, one LSTM hidden layer, and one linear hidden layer. The last layer is an output layer that uses the Softmax activation function to determine the probability of a transcript label. Fig. 5 shows a schematic of the DeepSpeech architecture. The training performance can be determined from the loss values on the network. To prevent overfitting on the network, we use the early stopping technique, which involves the comparison of the loss value of the network during validation. The output of the network will be processed by CTC to perform the decoding process with the prefix beam search method according to the probability generated by the SoftMax layer and language model. CTC is used to model the training results obtained by a network. Because it can classify labels without having to know the alignment given, it is suitable for the deep bidirectional LSTM [22]. The CTC value decreased as the number of layers used increased, a phenomenon known as the CTC loss. The CTC loss value is a representation of the 227 | P a g e www.ijacsa.thesai.org accuracy of the training results; a smaller CTC loss value indicates a higher accuracy. Nevertheless, an excessively small CTC loss value leads to overfitting. In the training, we used five scenarios denoted as A, B, C, D, and E, as presented in Table II, to find the most suitable parameters.
In scenarios B, D, and E, the training process is continued until the epoch is ended, while in scenarios A and C, it is stopped early. We used 3500 speech data for training and a batch size of 70; thus, the number of steps in each epoch was 50. These steps were repeated for all data in each epoch. The training process resulted in a model called output graph, which could be stored and used to transcribe the data. Table III presents the results of each training scenario.
From Table III, it can be seen that when spectrograms were used as the input to the deep bidirectional LSTM and early stopping was activated (scenario A), training ended after 12 epochs within 5 h, 13 min, and 23 s with training and validation loss values of 0.746044 and 4.590043, respectively. When MFCC were used as the input and early stopping was activated (scenario C), training ended after 9 epochs within 2 h, 53 min, and 5 s with training and validation loss values of 0.424811 and 0.914198, respectively. These results indicate that MFCCs are more suitable as input to the deep bidirectional LSTM than spectrograms. These results may also imply that the MFCC features are better in terms of the computation time and loss value, and provide more useful information for the classifier.
On the other hand, when spectrograms were used as the input to the deep bidirectional LSTM and early stopping was not activated (scenario B), the loss values were lower than in scenario A. Furthermore, when MFCCs were used without early stopping (training scenario D), the loss values were lower than in scenarios A, B, and C. In particular, in scenario B, the training and validation loss values were 0.084932 and 2.765626, respectively, and training was completed within 16 h, 6 min, and 40 s; in scenario D, the training and validation loss values were 0.016846 and 0.505358, respectively, and training was completed within 15 h, 13 min, and 14 s. Therefore, when MFCCs are inputted, the training and validation loss values are smaller than when spectrograms are inputted. Although overfitting occurred in training scenario D, the training process could be re-adapted as shown in Fig. 6. Overfitting may have occurred owing to the presence of noise.
To prevent overfitting, we reduced the number of epochs before the occurrence of overfitting (scenario E). The process of training in scenario E is performed to determine the best training results before overfitting. Training ended at epoch of 24 within 7 h and 23 min with training and validation loss values of 0.077836 and 0.494393, respectively. Fig. 7 shows the plot of the loss values in scenario E. Compared to training scenario D, scenario E has higher training loss value but lower validation loss value. In scenario E, the training process took only 24 epochs, which is less than in scenario D.

E. Testing Model
The model obtained from the training results was tested. The test involved speech-to-text conversion of 500 speech data samples not included in the training process. As a measure of the accuracy, we considered the WER, which has a range of 0-1; a smaller WER value indicates a higher testing accuracy. Table IV shows the results obtained from a different model of training scenarios.  From Table IV results with a WER of 2.0784% in training scenario B, testing with MFCCs yielded the best results with a WER of 0.2745% in training scenarios D and E. These results indicate that the MFCCs are more suitable than spectrograms for speech-to-text conversion with the deep bidirectional LSTM algorithm. Training with MFCCs can identify linguistic content and remove unimportant parts of speeches, which may contain noise. Furthermore, training with MFCCs can demonstrate the vocal tract of human speech in the form of a power spectrum. Fig. 8 shows the accuracy of each model.

F. Testing Process using Speech Variations
Testing was also performed using five speech variations, namely regular (normal) conversation, speech with high intonation, speech with low intonation, speech with fast rhythm, and speech with slow rhythm. The test involved two speakers: a male and a female. The speakers uttered sentences 1-5 (see Table I) with the five styles listed in Table V. We tested the model obtained from training scenario E (using MFCCs as features) because it yielded the best results among the five training scenarios. Table V shows the WER obtained from the testing using different speech styles.
The results reported in Table V indicate that the model detected normal speech effectively. It can be seen that the intonation of the speech did not significantly affect the results. The model also detected the speech with fast rhythm well. However, the WER value obtained with the slow-rhythm speech is 40%, which indicates that many errors occurred during speech-to-text conversion. This may be due to the timelapse between the words spoken and the large number of silence areas on the speech signal. These results indicate that the rhythm of the speech tends to affect the speech-to-text conversion process, while the intonation does not.

G. Testing Process with Secondary Datasets from TITML-IDN
The model obtained with training scenario E was also tested using five sentences obtained from the TITML-IDN dataset [23]. Table VI presents the results of speech-to-text conversion using speech corpus data for males and females. The results indicate that the model could detect the words in the TITML-IDN. For example, the target transcript "dia tidak datang ke sekolah" spoken by the male and female speakers is converted to "diadi kakak ke sekolah" and "diadi eka ke sekolah eu," respectively. Therefore, the model successfully detected the word "ke sekolah" and it detected the word "dia" as "diadi." However, the model did not recognize all words successfully because they were not in the transcript of the training data. Besides, the number of words in the model was much smaller (only 39 different words) than that in the TITML-IDN dataset. In addition, speech data from TITML-IDN have a different structure of sentences used in our primary data. ITITML-IDN either contains complete sentences (subject-predicate-object-adverb) or non-complete sentences or only phrases, as these data were obtained from the text corpus. Meanwhile, the speech in our primary data recorded from 10 respondents consists of complete sentences, as described in the section Data Collection.

IV. CONCLUSIONS
This study tested the performance of a deep bidirectional LSTM algorithm on speech-to-text conversion in Indonesian using MFCCs and spectrograms as features. The data used were complete sentences consisting of subject, predicate, object, and adverb spoken by some respondents. With the MFCCs and spectrograms, the algorithm achieved the highest WER of 0.2745% and 2.0784%, respectively, indicating the higher performance of the MFCCs on speech-to-text in the Indonesian language.
The algorithm was shown to successfully convert speech with different intonation and rhythm and achieved reasonable accuracy when applied to the TITML-IDN dataset.
However, the variation of words used in this study is still limited. Thus, in the future, the algorithm should be tested with speech with the higher variation of words and rhythms to increase its universality.