Abstract: The tangent plane algorithm for real time recurrent learning (TPA-RTRL) is an effective online training method for fully recurrent neural networks. TPA-RTRL uses the method of approaching tangent planes to accelerate the learning processes. Compared to the original gradient descent real time recurrent learning algorithm (GD-RTRL) it is very fast and avoids problems like local minima of the search space. However, the TPA-RTRL algorithm actively encourages the formation of large weight values that can be harmful to generalization. This paper presents a new TPA-RTRL variant that encourages small weight values to decay to zero by using a weight elimination procedure built into the geometry of the algorithm. Experimental results show that the new algorithm gives good generalization over a range of network sizes whilst retaining the fast convergence speed of the TPA-RTRL algorithm.

P May, E Zhou and C. W. Lee, “Improved Generalization in Recurrent Neural Networks Using the Tangent Plane Algorithm” International Journal of Advanced Computer Science and Applications(IJACSA), 5(3), 2014. http://dx.doi.org/10.14569/IJACSA.2014.050317

@article{May2014,
title = {Improved Generalization in Recurrent Neural Networks Using the Tangent Plane Algorithm},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2014.050317},
url = {http://dx.doi.org/10.14569/IJACSA.2014.050317},
year = {2014},
publisher = {The Science and Information Organization},
volume = {5},
number = {3},
author = {P May and E Zhou and C. W. Lee}
}