An Information Theoretic Analysis of Random Number Generator based on Cellular Automaton

Realization of Randomness had always been a controversial concept with great importance both from theoretical and practical Perspectives. This realization has been revolutionized in the light of recent studies especially in the realms of Chaos Theory, Algorithmic Information Theory and Emergent behavior in complex systems. We briefly discuss different definitions of Randomness and also different methods for generating it. The connection between all these approaches and the notion of Normality as the necessary condition of being unpredictable would be discussed. Then a complex-system-based Random Number Generator would be introduced. We will analyze its paradoxical features (Conservative Nature and reversibility in spite of having considerable variation) by using information theoretic measures in connection with other measures. The evolution of this Random Generator is equivalent to the evolution of its probabilistic description in terms of probability distribution over blocks of different lengths. By getting the aid of simulations we will show the ability of this system to preserve normality during the process of coarse graining. Keywords—Random number generators; entropy; correlation information; elementary cellular automata; reversibility


I. INTRODUCTION TO RANDOMNESS
Realization of randomness has great importance both from theoretical and practical perspective.The successful application of randomness for guiding search processes in spaces which scales exponentially to the input size shows the theoretical importance of having access to a proper source of randomness [1].One can show generally that the existence of proof for theorem T with specific length n can be reduced to combinatorial problem of finding a tour of length that reaches all cities while [2].
Although the concept of Randomness has entered literarily by the theory of probability, this theory cannot define the randomness associated with individual objects whether finite or infinite.Probability theory is a theory about sets of objects not individual objects.Therefore it cannot distinguish between two sequences of length generated on binary alphabet in which one of them consists of just ones and the other corresponds to the tossing of a fair coin.It turns out that realization of randomness is very challenging and this concept resists theoretical investigation.Historically scientists have tried to define it in a specific domain.Fortunately one can observe the source of problem as a main thread in all of these domain specific approaches.
Von Mises [3] was the first person who tried to define randomness mathematically based on an intuitive aspect of unpredictability.He described randomness as an inability to predict the elements of an infinite binary sequence over with probability better than while the elements in the string are chosen randomly.Then he tried to improve his definition by replacing the random selection of elements with an acceptable selection rule.Evidently this change did not increase the mathematical clarity of his definition.Subsequently Wald [4,5] introduced the notion of countability of selection function in order to make this definition more clear.Finally Mises's definition was refined in the light of computability of selection functions by Church [6].Therefore acceptable selection rules were replaced by computable functions and as such the theoretical realization of randomness integrated with the notion of computability.This type of evolved definition is known as Mises-Wald-Church definition [7].Although Mises-Wald-Church definition of randomness was criticized by other thinkers like Ville [8], it kept its theoretical effect on subsequent works in this area.
The relation between randomness and computability has been deepened in the light of modern interpretation of defining and detecting randomness relatively to the amount of computational sources which have been used [9].The notion of randomness is measured in its modern formulation for finite objects by Kolmogorov complexity [10], [11] which is the result of Solmonof, Kolmogorov and Chaitins theory [12] and for infinite objects by the Mrtin-Lof measure of randomness [13], [14].
These new definition of randomness are deeply connected to the theory of computation.The Kolmogorov complexity of a string is defined as the size of the shortest program that produces it.Obviously random strings must have higher Kolmogorov complexity due to their incompressibility.Therefore the Kolmogorov complexity of random string , denoted by would be approximately equals to length of string .Formally and random strings cannot be compressed [10].Introducing finite random objects as incompressible objects connects the notion of randomness to the philosophical interpretation of theory which was presented for the first time by Leibniz [15].
Philosophically, theory is recognized as a compression form of statements which can describe a set of large experimental data in the shortest way.Otherwise the theory with the same size of data set, which it explains would be www.ijacsa.thesai.orguseless.In section V of Discourse de Metaphysique, Leibniz explains the comprehensibility of the world as the result of God's creation, in which the greatest possible diversity of phenomena are controlled by the smallest set of ideas.Today this fact can be rephrased in the language of algorithmic randomness.It means in spite of this apparent diversity in the world the set of rules which are responsible for all these phenomena is small.Therefore we can follow a scientific method to discover the theory which explains a set of wide natural phenomena.Based on this perspective the set of random data would be theory-less.Chaitin's Ω number is recognized as an incompressible sequence of zeros and ones.This number is the probability of halting for program P generated by a successive independent tosses of a fair coin.

∑ 
Ω is an example of algorithmically irreducible or random string.The bits of Ω cannot be compressed.In other words, its bits are true for a reason not simpler than itself.This breaking of Leibniz's famous principle of sufficient reason is one of the most controversial aspect of randomness in philosophy.It must be mentioned that Chaitin's result about halting probability is in fact another type of Godel's famous Incompleteness theorem [16].Today we know that proving the randomness of a finite string , is an example of incompleteness phenomena.
The computability of selection function in the Mises-Wald-Church definition of randomness for finite objects can be observed in terms of effective measure for continuous objects in Martin-Lof definition [14].A real number is considered random, if is not contained in any event of effective measure zero [14].
The unpredictability as an intuitive notion of randomness plays a great role in other approaches for defining randomness.Ville [8] tried to define randomness by gambling approach.In his definition, it is impossible to win an infinite amount of money by betting on the bits of a random binary sequence.The amalgamation of the theory of randomness and the theory of computation has provided the opportunity of using theoretical apparatuses developed in the theory of computation to analyze the randomness in a deeper way.Today we know the set of strings which are random in the sense of Kolmogorov complexity is not even computably enumerable and because of that, the statement of the form: is a random string, is not provable [16].
In addition to the previous results, randomness in its modern formulation is considered relatively.In the light of the theory of computability, we interpret randomness in contrast to the amount of computational sources, used to detect it [17].
Although randomness has been always seen as a sign of complexity especially in the lack of causal model, in recent years, it has been realized that randomness is responsible for emergence of many complex phenomena by providing the opportunity of having interactions between many agents in systems.There is a tendency to disentangle randomness from complexity.For a recent work on this please refer to [18].It is believed that complexity in its own true form is the result of directed interactions between elements of system.Randomness and its mysterious aspects have played a great role especially in fertilizing the multidisciplinary studies at the crossroad of mathematics, physics and computer science.Using random resources for solving hard problems in Randomized Algorithms has been very advantageous [19] both in substantial reduction of time complexity and also in deepening of our understanding of the nature of hard problems.
After the seminal work of Russel impagliazzo and Avi Wigderson about the tradeoff between hardness and randomness [20], today we know the importance of having randomized algorithm for solving hard problems as a theoretical key for designing deterministic algorithms.
There is no doubt that nature uses randomness extensively in the evolution.In recent years, Gregory Chaitin has started working on Metabiology [21].He is studying the random evolution of artificial software in order to realize Busy Beaver function as a fitness function.The soul of his technique has been based on applying randomness as it has been applied in evolution in terms of mutation.This paper has been organized into 6 sections.After this introduction about randomness, Random generators and their categorization and applications will be reviewed in Section 2. In Section 3, Cellular Automaton is formally defined and its application as Random generator will be discussed.Primary information Theoretic measures are explained in Section 4. These measures are used to analyze our inhomogeneous ECA60 as a Random Generator in Section 5 and finally in Section 6, concluding remarks will be presented.

II. RANDOM GENERATOR
The difficulties of giving a complete definition of randomness were discussed briefly in the previous section.Randomness in its modern formulation is considered as a relative concept.Therefore Random Generators must satisfy criteria which guarantee their efficiency for the specific application.
The success of simulations which are based on Monte Carlo method is highly dependent on the quality of their random generators [22]- [25].The security of information transfer on internet and the cryptography need random generators [26].Randomness and Random generators are used extensively in solving hard problems in the framework of stochastic optimization and naturally inspired algorithm in order to bypass the problem of getting stuck in local extremes [27]- [29].
Random generators play a key role in many techniques of program validation [30] and Machine learning [31].The rational behavior in strategic zero sum games needs to use randomness and Random Generators to mislead the opponent [32].In recent years, Random Generators are used for Analyzing and simulation of interactions in complex social, economical and political systems at different scales [33]- [35].
Historically three approaches have been followed for generating randomness although there are many controversial www.ijacsa.thesai.orgissues about the possibility of producing intrinsic randomness.True Random Number Generators (TRNG) use physical phenomena with inherent stochastic mechanism for generating random numbers [36].There are many phenomena which can be used for TRNGs, for example unstable nuclear decay processes [37], cosmic background radiation [38], quantum based systems [39]- [41] and more exotic examples like superconducting nanowires and Josephson junctions near superconducting critical current [42].
The randomness of physical source can be amplified in some cases [43] provided that fundamental theoretic limits are preserved [44], although Santha and Vazirani proved that randomness amplification is impossible using classical resource [45].
The next family of generators is called Random Number Generators (RNG).In these generators randomness is transformed from a priori distribution in source to the desired posterior distribution [46], therefore these generators have an access to sources of randomness.
RNGs are categorized to three main groups: Von Neumann RNG [47] in which an identically independent priori distribution is transformed to the unbiased random numbers.Knuth and Yao RNG [48], in which an identically independent prior distribution is transformed to any desired distribution in output and finally Roche and Hoshi RNG [49], [50] which transform an arbitrary random distribution in source to an arbitrary distribution in output.
The third family of generators for generating randomness is called Pseudo-Random Number Generators (PRNG) in which arithmetical methods are used in order to produce randomness.Although John Von Neumann once said [47] "Anyone who considers arithmetical methods for producing random digits is, of course, in the state of sin", Chaos theory shows how randomness can emerge from deterministic systems, while we have a hypersensitive dynamic to the initial states [51], [52].
Therefore, in the light of Chaos Theory, there is a deep dichotomy between order and randomness.This fact provides the opportunity of applying deterministic algorithms for generating randomness.Hence, PRNGs try to generate randomness without having access to any source of randomness [53]- [56].

III.
CELLULAR AUTOMATA AND RANDOM GENERATORS Cellular Automata were created by John Von Neumann, in his attempt to create a self-replicating machine [57].He tried to show that, these machines are universal constructors and can generate even themselves.Cellular Automata have found a better place in theoretical studies when Stephen Wolfram published his book, called A New Kind of Science [58].In this book he tried to present an extensive analysis of these systems and their effectiveness to realize a wide range of natural phenomena which are common to exhibit a particular type of behavior known as Emergent behavior.
Elementary Cellular Automata (ECA) is defined on alphabet set as 1-dimensional cellular array of size .The state of each cell at time is denoted by The global state or lattice configuration of Cellular Automaton at time t is represented by , where ( is the size of lattice) [59].
All cells in the lattice are updated according to local update function which generally has arguments, where is the radius of local function ( in Elementary Cellular Automata).Formally is defined by local function .


Applying to all cells simultaneously, leads to the formation of next global state and maps to under the action of global function .It has been conjectured that applying very simple local function at micro scale can give rise to a very complicated behavior in macro scale [58].This highly interesting behavior is called Emergence and has inspired an extensive type of studies [60].
Cellular Automata were used for the first time as Pseudo-Random Generator by Wolframe [61], [62].Actually he used the unpredictable emergent behavior of the global function in Cellular Automata resulted from simultaneous application of local function on different cells, as a main mechanism for generating randomness.
Afterwards, people tried to improve the quality of CA's random generator by using a combination of controllable cells [63], increasing the dimensionality of Cellular Automata [64]- [66], changing the neighborhood of cells [67], using Cellular Automata with additive rules [68], applying the evolutionary principles for designing Cellular Automata [69] and focusing on the parallelism associated with the evolution of global state in Cellular Automata for generating randomness [70].
Stephen wolfram in his well-known book "The New Kind of Science" categorized elementary Cellular Automata into four families [58].The states of cells in elementary Cellular Automata are selected from simple binary alphabet set and the radius of their local function is equal to one.It is believed that in spite their simplicity, Elementary Cellular Automata (ECA) show all types of behaviors which can be observed in Cellular Automata in higher dimensions and with more complex local functions.Since the number of possible Elementary Cellular Automata is limited to , these systems have been examined from different perspectives [71], [72].
As an evidence for the power of ECA and the mysterious aspects of emergent behavior at macro scale in complex systems, like Cellular Automata, coordinated by simple interactions between cells at micro scales, please refer to [73] in which Emergent behavior has been used in combination with other complex system's properties for generating pseudorandomness.It turns out that this strategy can generate randomness which does not have any dependency on the initial values of the system.
In this paper, a modified type of an Elementary Cellular Automaton (ECA60) would be used as random generator and its evolution from random initial state would be analyzed information theoretically.In the next section, primary Information measures will be introduced briefly.www.ijacsa.thesai.orgIV.
INFORMATION THEORETIC MEASURES Our goal in this paper is to analyze the capability of a modified type of Elementary Cellular Automaton 60, the number of which is assigned according to Wolfram's rule, as a random generator.The system is initialized by a random binary sequence on alphabet set {0, 1}.During the evolution of Elementary Cellular Automaton, this random initial sequence of size is transformed to other sequence of size , while it preserves the initial randomness and simultaneously shows dynamical reversibility.It avoids building correlations in the evolution while all of the initial information at each time step is transformed to the next global state and because of this conservative behavior and dissipation-less dynamic the initial state of the system would be observed again.The periodicity of this conservative behavior is .In order to analyze this behavior, we use information theoretic measures.
Elementary Cellular Automaton with rule number 60 has been used as a random generator in which the central cell in local configurations 101,100, 011 and 010 is transformed to 1 and the central cell in the remaining four configurations 000,001,110 and 111 is transformed to 0, except for the first cell , which is transferred without change to the next generation.As we will observe in the following section, this direct transfer of first cell is responsible for the reversibility of system.The evolution of our random generator from random initial state is shown in Fig. 1.The system is initialized by random global state, consisting of zeroes and ones (here our alphabet set is ). Randomness implies that, starting from the beginning of system, one cannot predicate the next symbol in the sequence by observing its previous symbols.Statistically it means:  Here, randomness has been interpreted as a uniform distribution on the characters of alphabet which is led to the maximum possible entropy rate.
The entropy function which measures the average of uncertainty can be defined over substring of global state of length , provided that conditions in (4) and ( 5) are satisfied [74].

 ∑ 
Furthermore, it is assumed that sequences are generated by the stationary stochastic process.The Ergodicity of generator is preserved due to the nature of local function in Elementary Cellular Automata [74].
These substrings are in fact the microstates of the system upon which macro state is built.Technically, the famous method of describing macro state in terms of probabilistic mass function over its constituent microstates in statistical physics has been applied.Therefore, the probabilistic description of the system's global state called , defines the Probability Mass Function (PMF) over system's microstates.
Having in mind such probabilistic description of the system, the entropy of system is defined by ( 6) provided [75].
[ ] ∑  Actually quantifies the disorder of n-length subsequences of the global string at generation .Usually increases as (the size of substrings or microstates) grows, but the action of local function makes certain correlations, which mitigates the growth of disorder proportional to the length of microstates.The maximum entropy associated with each character which is selected from alphabet set , is and consequently in the case of pure randomness, the maximum entropy of for sequence of length over alphabet set is expected to be observed.
The correlation can be detected as the result of becomes less than (the maximum amount of entropy per symbol when the characters are chosen from set ).But this reduction ( may be the result of correlations which have been built at much lower lengths.In order to detect the correlation specifically at length n, one can compare the consecutive gaps in entropy, computed on blocks of lengths and .Consequently, correlation at length in ( 7) is calculated.


All correlations which have been formed at lengths less than are considered in , hence if there is correlation at length , must be less than and their difference is purely related to (Correlation at length ).Although there are different approaches for calculating correlations, one can observe the simple phenomenon which produces it at specific distance.Formally the emergence of correlation at length requires (8) to be met.


The maximum amount of entropy per symbol is decomposed to the entropy rate of the system and all the www.ijacsa.thesai.orgcorrelations which have formed at different lengths.

V. ANALYZING THE RANDOM GENERATOR
The modified version of ECA60 has been used as a Random Generator which transforms an initial random sequence into another random sequence in the next generation.This transformation is done against our normal expectation to observe the increment of regularity in the sequence, due to the applying of deterministic local function.Fortunately the system is able to preserve the initial randomness during its evolution in spite of having a considerable hamming distance between consecutive global states of the system.It means that, this conservation of randomness would not decrease the activity of system.
As Lindgren [74] showed in Elementary Cellular Automata, due to the deterministic nature of local function, we expect the global entropy to be decreased during the evolution of system.Mathematically it means this fact can be realized intuitively, since applying local function in Elementary Cellular Automata is usually accompanied with the omission of variations in the system's global state and would force system to converge to the very small subset of all possible global states.Obviously moving toward regularity and reduction of variations would lead to decrementing of the initial randomness.This behavior is not suitable for our purpose.On the other side it can be shown that is zero for systems with conservative nature [74].
It is not hard to realize that would be zero for almost reversible systems, i.e. entropy is constant during the evolution.A Cellular Automaton with rule R and range r is called almost reversible if R can be decomposed in the following way [74]:


It means R is one to one if one can support it by giving it the information about its first or last argument.It is easy to show that our random generator is an almost reversible rule since its local function can be written as:


Therefore is reversible if we have an access to the value of or .In order to recover the global state at time from the global state at time , we can use the following equation inductively for , when is the size of system, if we have an access to the first bit of global state at time ).


Please remember that our Random Number Generator is an inhomogeneous type of ECA60 in which the state of first cell in the system is transformed to the next generation without change.In fact, having access to this single bit from the previous generation makes us able to recover all the bits of previous state by applying (13), iteratively.Generally in Almost Reversible Rule with range , one can recover the previous state by having access to the -bits of the previous generation [74].
As a matter of fact we are taking the advantage of emergence to produce next random sequence from the current one.This emergence here manifests itself as complex global function induced by simultaneous applying of local function to all cells in the system.Generally there is no mathematical method to find the relation between global function and local function in such systems.
Conservation of information in this Random Generator which is leaded to the conservation of initial randomness due to the ( 9) is related to the global reversibility of Cellular Automata.
Studies by Hedlund [76] and Richardson [77] have shown that for Cellular Automaton A, A is injective if it is invertible (Reversible).Therefore reversibility in CA is equivalent to the injectivity of its global function.Furthermore A is injective if it is surjective and there is a close relation between injectivity and surjectivity of the Cellular Automaton A.
Injectivity and Surjectivity for global functions of CA's were studied by Moore [78] and Myhill [79] for the first time in the way of analyzing Garden of Eden (a global state without predecessor).Studies about the reversibility of dynamical systems have attracted many attentions on the part of scientists.We know that physical world at micro scale is governed by reversible rules.Surprisingly what can be observed at macro scale is irreversible.It is believed that this irreversibility at macro scale motivated by reversible rules at micro scale can be interpreted as a sign of Emergence in the system [60].
In addition to that, studies about Reversibility has found an extra importance after the seminal work of Landauer [80].He found a relation between consuming energy and irreversibility in dynamical systems.In other words, he showed that dissipation of heat in the system which is the main source of consuming energy has its root in erasing information during the process in the system.Erasing information happens in irreversible systems, in which one cannot recover the information of previous state by having the information of current state.Although it is not hard to imagine that every irreversible process would be accompanied by erasing information, Landauer [80] predicted the minimum amount of energy dissipation associated with erasing one bit.Recently his prediction has been verified experimentally [81].It has been shown by Bennett [82] that it is possible to do any computation reversibly.It means we have a reversible analogue for Universal Turing Machine.Then people started to analyze the advantages of reversibility in computing system [83].It is exciting to know that in principle, reversible computation can be done with zero energy consumption.Many fundamental www.ijacsa.thesai.orglimits to the computation process were realized better in the light of these studies [84].Due to the importance of reversibility, many algorithms have been developed in order to detect it in the variety of dynamical systems [85], [86].It has been shown that deciding the reversibility or its equivalent property in systems with dimension higher than one is impossible [87], [88].
In spite of conservative nature of this Random Generator, the initial random configuration is transformed into another random configuration with considerable hamming distance with previous global state.Furthermore the normality of random binary sequence is kept during the evolution of system.Let's look at the definition of normality and normal sequence.
Normality demands the balanced form of appearance for all patterns which means every block of digits of the same length occurs with the same frequency when all digits in the expansion are considered.Normality can be interpreted in the light of information theoretic measures.Although there are other similar concepts which are related to notion of normality, one has to know that random sequences must be normal in order to satisfy their expected unpredictability.
It is expected that unpredictability demonstrates itself as our inability to forecast the dominant frequency of observing specific pattern.Normality can be described with the aid of other measures like block complexity, block entropy, etc.
Definition [89]: the block complexity of a sequence with values in a finite alphabet is the function , where is the number of different blocks of length that occur in the sequence.
Clearly for a sequence over alphabet set with length , block complexity satisfies the (14).


For normal sequences, maximum block complexity is expected.Obviously low block complexity would not be seen in random sequence and can be responsible for generating periodic behavior.It is interesting to realize how block complexity as a local measure for sub-sequences can be used to predict the global behavior of the sequence.Morse and Hedlund proved the following theorem about this relation.
Theorem [90]: if the complexity of a sequence satisfies , then the sequence is ultimately periodic.
The block complexity can easily be related to block entropy.In section 4, the block entropy for the global state of a Cellular Automaton was defined when the process responsible for generating it is ergodic.When the block complexity for blocks of length over alphabet set is equal to , it can be concluded that every possible pattern of length over this alphabet set has been generated; therefore, maximum entropy would be expected.There is a simple relation between block complexity and block entropy.Considering their definition we can simply reach to (15).


Here is a binary set therefore ( 14) can be simplified into .As we discussed before, the entropy rate would be calculated when the length of block tends to infinity and all correlations are considered.Thus we have:


When the length of block tends to infinity, the correlations among symbols at different lengths exhibit themselves as a restriction of freedom for choosing among the possible characters of alphabet.Therefore in the case of having correlations, the block complexity is not increased proportionally to the length of the block and the normality is broken.In other words for a sequence of length with correlations among its characters we have .
Since the correlation at specific length is accompanied by the reduction of growth in block complexity at length in contrast to its growth at length it can be detected when the entropy rate is decreased at some specific length.In Fig. 2, the entropy rate over blocks of size 10 for a system of size 10^6, over 100 generation has been shown.The entropy rate shows fluctuations in the scale of 10^ (-5) over generations.Furthermore the lower value of entropy rate for blocks of size 10 is bigger than 0.99 which approves of the random nature of sequences generated by this random generator since the maximum entropy rate for our system is .
In this Random Generator, the block complexity is much more than the lower threshold predicated by Morse & Hedlund [90], [91] and it is around Surprisingly here in spite of keeping considerable local variations and nearly maximum block complexity, the global state of system shows periodicity due to its conservative nature of system and the trajectory of global state starting from initial configuration meets the initial state again.Fortunately in its entire route from initial state, the normality at different lengths is preserved because the entropy rate does not depict considerable difference calculated on blocks of different sizes.In Fig. 3, the entropy rate for system of size 10^6 over 200 generation has been displayed when the entropy rate is calculated over blocks of different lengths ranging from 5 to 10. www.ijacsa.thesai.orgFortunately the conservation of information does not impose restriction on the micro-dynamic of system and consecutive global states have considerable hamming distance.Please see the Fig. 4.

VI. DISCUSSION AND CONCLUSION
The work of this Random Generator can be analyzed from different perspectives such as the Conservation of initial information during the evolution leads to reversible dynamic, the intrinsic parallelism (as the result of simultaneous updates of cells in the system), the efficient use of initial randomness and the ability of generating acceptable number of sequences which are equivalent to the initial configuration considering their randomness or their entropy rate.In other words this Random Generator is able to show varied behavior while it keeps its initial information during its evolution.
Taking into account (9) as the Hamiltonian of this Random Generator, the maximum amount of entropy rate of the system can be decomposed into the entropy of the system or its random parts plus its regular parts which are the result of summing all of its correlations at different lengths.Actually in this system, the random part of the current global state is mapped into the random part of the next global state.The fluctuations in the entropy rate calculated over blocks of different lengths are negligible and can be the result of encoding randomness from one length into another.Furthermore due to its avoidance of building correlations or empowering them, the Normality of generated sequences is preserved during the evolution at all scales.
Preservation of Normality at all scales and the conservative nature of this Random Generator, in spite of having considerable micro-activity turn this system into remarkable Random Generator.In addition, this Random Generator gets the benefits of intrinsic parallelism and proves to be technically easy to implement due to its simple logical local function.

Fig. 1 .
Fig. 1.The evolution of ECA60 from random initial configuration of size 200 during 50 generations.

Fig. 2 .
Fig. 2. The entropy rate for system of size 10^6 over blocks of size 10 for 100 consecutive generations.

Fig. 3 .
Fig. 3.The entropy rate for system of size 10^6 over blocks of different lengths ranging from 5 to 10 during 200 generations.

Fig. 4 .
Fig. 4. the hamming distance between consecutive generations for system of size 10000 during 10^5 generations.