Parallel Implementation of Gradient Descent Algorithm for Backpropagation Networks
DOI:
https://doi.org/10.26438/ijcse/v5i10.266272Keywords:
Backpropagation, Supervised Learning, CUDA, parallelAbstract
The problem of computational efficiency in adaptive algorithms, which is current and pressing, can be solved through their implementation in parallel frameworks, like CUDA, OpenCL, etc. The approach taken to parallelize any complex operation requires its separation into several distinct and independent sub-operations. We employed the same procedure to parallelize the BP (or Backpropagation) network algorithm. The function breakdown of the BP network involved breaking its overall operation into Feed-forward and Back-Propagate sub-operations, which was further divided into smaller independent execution groups. We applied parallel constructs on those independent execution groups and used the MNIST dataset to compare the algorithm’s performance with respect to the sequential algorithm. Comparing their performances, we found that the efficiency of the algorithm depended on the size of the BP network. In the large network with massive number of weight connections, we saw a significant improvement in the convergence time. This makes our algorithm preferable in feedforward networks having large number of hidden layers, neurons and weight connections.
References
G. S. Almasi, A. Gottlieb, “Highly Parallel Computing”, Benjamin-Cummings Publishing Co., Inc., USA, 1989.
D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986.
C. Li, C. Yu, “Performance evaluation of public non-profit hospitals using a BP artificial neural network: The case of Hubei province in china,” International Journal of Environmental Research and Public Health, Aug 2013.
Y. Li, Y. Fu, H. Li, and S. W. Zhang, “The improved training algorithm of back propagation neural network with self-adaptive learning rate,” in 2009 International Conference on Computational Intelligence and Natural Computing, pp. 73–76, 2009.
J. Zhu, P. Sutton, “FPGA implementations of neural networks–a survey of a decade of progress,” Field Programmable Logic and Application, pp. 1062-1066, 2003.
I. B. D. Steinkraus, P.Y. Simard, “Using GPUs for machine learning algorithms,” Document Analysis and Recognition, pp. 1115-1120, 2009.
C. M. Bishop, “Pattern Recognition and Machine Learning”, Springer-Verlag New York, USA, 2006.
Y. Yuan, “Step-sizes for the gradient method,” AMS/IP Studies in Advanced Mathematics, 1999.
R. A. Jacobs, “Increased rate of convergence through learning rate adaptation,” Neural Networks, vol. 1, 1988.
M. J. Flynn, “Some computer organizations and their effectiveness,” IEEE Transactions on Computers, vol. C-21, no. 9, pp. 948–960, 1972.
E. Kussul, T. Baidyk, “Improved method of handwritten digit recognition tested on MNIST database,” Image and Vision Computing, vol. 22, no. 12, pp. 971 – 981, 2004.
J. D. Owens, M. Houston, D. Luebke, S. Green, J. E. Stone, J. C.Phillips, “GPU computing,” Proceedings of the IEEE, vol. 96, no. 5, pp. 879–899, May 2008.
U. Ray, T.K. Hazra, U.K. Ray, "Matrix Multiplication using Strassen’s Algorithm on CPU & GPU", International Journal of Computer Sciences and Engineering, Vol.4, Issue.10, pp.98-105, 2016.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.
