Parallel Implementation of Gradient Descent Algorithm for Backpropagation Networks - Devkota, K; Bhattarai, P (International Journal of Computer Sciences and Engineering, 2025)
The problem of computational efficiency in adaptive algorithms, which is current and pressing, can be solved through their implementation in parallel frameworks, like CUDA, OpenCL, etc. The approach taken to parallelize any complex operation requires its separation into several distinct and independent sub-operations. We employed the same procedure to parallelize the BP (or Backpropagation) network algorithm. The function breakdown of the BP network involved breaking its overall operation into Feed-forward and Back-Propagate sub-operations, which was further divided into smaller independent execution groups. We applied parallel constructs on those independent execution groups and used the MNIST dataset to compare the algorithm’s performance with respect to the sequential algorithm. Comparing their performances, we found that the efficiency of the algorithm depended on the size of the BP network. In the large network with massive number of weight connections, we saw a significant improvement in the convergence time. This makes our algorithm preferable in feedforward networks having large number of hidden layers, neurons and weight connections.