Construction of Neural Networks that Do Not Have Critical Points Based on Hierarchical Structure

a critical point is a point at which the derivatives of an error function are all zero. It has been shown in the literature that critical points caused by the hierarchical structure of a real-valued neural network (NN) can be local minima or saddle points, although most critical points caused by the hierarchical structure are saddle points in the case of complex-valued neural networks. Several studies have demonstrated that singularity of those kinds has a negative effect on learning dynamics in neural networks. As described in this paper, the decomposition of high-dimensional neural networks into low-dimensional neural networks equivalent to the original neural networks yields neural networks that have no critical point based on the hierarchical structure. Concretely, the following three cases are shown: (a) A 2-2-2 real-valued NN is constructed from a 1-1-1 complex-valued NN. (b) A 4-4-4 real-valued NN is constructed from a 1-1-1 quaternionic NN. (c) A 2-2-2 complex-valued NN is constructed from a 1-1-1 quaternionic NN. Those NNs described above do not suffer from a negative effect by singular points during learning comparatively because they have no critical point based on a hierarchical structure.

@article{Nitta2013,
  title     = {Construction of Neural Networks that Do Not Have Critical Points Based on Hierarchical Structure},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {4},
  number    = {9},
  year      = {2013},
  publisher = {The Science and Information Organization},
  author    = {Tohru Nitta},
  doi       = {10.14569/IJACSA.2013.040911},
  url       = {https://doi.org/10.14569/IJACSA.2013.040911}
}