Abstract: a critical point is a point at which the derivatives of an error function are all zero. It has been shown in the literature that critical points caused by the hierarchical structure of a real-valued neural network (NN) can be local minima or saddle points, although most critical points caused by the hierarchical structure are saddle points in the case of complex-valued neural networks. Several studies have demonstrated that singularity of those kinds has a negative effect on learning dynamics in neural networks. As described in this paper, the decomposition of high-dimensional neural networks into low-dimensional neural networks equivalent to the original neural networks yields neural networks that have no critical point based on the hierarchical structure. Concretely, the following three cases are shown: (a) A 2-2-2 real-valued NN is constructed from a 1-1-1 complex-valued NN. (b) A 4-4-4 real-valued NN is constructed from a 1-1-1 quaternionic NN. (c) A 2-2-2 complex-valued NN is constructed from a 1-1-1 quaternionic NN. Those NNs described above do not suffer from a negative effect by singular points during learning comparatively because they have no critical point based on a hierarchical structure.

Tohru Nitta, “Construction of Neural Networks that Do Not Have Critical Points Based on Hierarchical Structure” International Journal of Advanced Computer Science and Applications(IJACSA), 4(9), 2013. http://dx.doi.org/10.14569/IJACSA.2013.040911

@article{Nitta2013,
title = {Construction of Neural Networks that Do Not Have Critical Points Based on Hierarchical Structure},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2013.040911},
url = {http://dx.doi.org/10.14569/IJACSA.2013.040911},
year = {2013},
publisher = {The Science and Information Organization},
volume = {4},
number = {9},
author = {Tohru Nitta}
}