Improved Generalization in Recurrent Neural Networks Using the Tangent Plane Algorithm

The tangent plane algorithm for real time recurrent learning (TPA-RTRL) is an effective online training method for fully recurrent neural networks. TPA-RTRL uses the method of approaching tangent planes to accelerate the learning processes. Compared to the original gradient descent real time recurrent learning algorithm (GD-RTRL) it is very fast and avoids problems like local minima of the search space. However, the TPA-RTRL algorithm actively encourages the formation of large weight values that can be harmful to generalization. This paper presents a new TPA-RTRL variant that encourages small weight values to decay to zero by using a weight elimination procedure built into the geometry of the algorithm. Experimental results show that the new algorithm gives good generalization over a range of network sizes whilst retaining the fast convergence speed of the TPA-RTRL algorithm.

@article{May2014,
  title     = {Improved Generalization in Recurrent Neural Networks Using the Tangent Plane Algorithm},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {5},
  number    = {3},
  year      = {2014},
  publisher = {The Science and Information Organization},
  author    = {P May and E Zhou and C. W. Lee},
  doi       = {10.14569/IJACSA.2014.050317},
  url       = {https://doi.org/10.14569/IJACSA.2014.050317}
}