A Review of Optimization Methods in Deep Learning
DOI:
https://doi.org/10.26438/ijcse/v6i4.440447Keywords:
Artificial Neural Network, Deep Learning CNN, RNN, Optimization Methods, Gradient Descent, ADAM, Framework, mageClassificationAbstract
Deep learning technique is an emerging field of machine learning. In recent years, it has been successfully used in different fields, such as image classification, natural language processing, computer vision, speech reorganization, etc. When compared to the machine learning, deep learning has a high learning ability to extract features of large datasets. Deep learning came into existence in 1971 when Ivakhnenka used group method of data handling algorithm (GMDH) to train 8-layered neural network [1]. This paper focuses on the artificial neural network, learning techniques and optimization methods of deep learning like stochastic gradient descent, batch gradient descent, mini-batch gradient descent and ADAM.
References
A.G. Ivakhnenko, “Polynomial theory of complex systems” IEEE Transaction on System, Man and Cybernetic vol. 1,no 4, pp. 364-378, 1971.
Xuedam Du,Yinghao Cai, Wang, and Leijie Zhang “ Overview of Deep Learning” 31st Youth Academic Annual Conference of Chinese Association of Automation Wuhan China November 11-13-2016.
Siddhartha Sankar Nath, Janynyaseni Kar, Girish Mishra, Sayan Chakraborty , Nilanjan Dey “ A Survey of Image Classification Methods and Techniques ” ICCCICCT 2014.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. Neural Information Processing Systems, Nevada, 2012
Henrik Petersson, David Gustafsson and David Bergstroom “ Hyperspectral Image Analysis using Deep Learning- a Review” IEEE 2016.
Adrian Carrio, Carlos Sampedro, Alejandro Rodriguez Ramos and Pascual Campoy “A Review of Deep Learning Methods and Applications for unmanned Aerial Vehicles ” Hindawi Journal of Sensors 2017.
H. Kamitomo and C. Lu, “3-d face recognition method based on optimum 3-d image measurement technology,” Artificial Life and Robotics, vol. 16, no. 4, pp. 551–554, 2012.
Walaa Hussein Ibrrahim, Ahmed AbdelRhman Ahmed Osman, Yusra Ibrahim Mohamad” MRI Image Classification Using Neural Network” ICCEEE, 2013.
S.Kim, B.Park, B.S Song, and S.Yang, “ Deep belief network based statistical feature learning for fingerprint liveness detection, ” Pattern Recog. Lett., vol 77, ,pp. 58-65,2016.
Gang Liu, Liang Xiao, Caiquan Xiong “ Image Classification with deep belief network and improved gradient descent” IEEE 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradientbased learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
Travis Williams, Robert Li “ Advanced Image Classification using Wavelets and Convolution Neural Network” IEEE 2016.
Narek Abroyan , “ Convolutional and Recurrent Neural Network for real time data classification” The Seventh International Conference on Innovative Computing Technology (INTECH 2017).
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, “Deep Learning”, Book in preparation for MIT Press, 2016, on-line version available at http://www.deeplearningbook.org.
Michael A.Nielsen, “Neural Networks and Deep Learning”, Determination Press, 2015.
Marek Dabrowski, Justyna Gromada, Tomasz Michalik Orange Centrum “A Practical study of neural network –based image classification model trained with transfer learning method” FedCSIS 2016.
C. Lu and X. Tang, “Surpassing human-level face verification performance on lfw with gaussianface,” arXiv preprint arXiv:1404.3840, 2014.
L. Deng and D. Yu, “Deep learning: methods and applications,” Foundations and Trends in Signal Processing, vol. 7, no. 3-4, pp. 197–387, 2013.
J. A. Hertz,” Introduction to the theory of neural computation.“ Boulder, USA: Westview Press, 1991.
J. Suykens and J. Vandewalle, “Least squares support vector machine classifiers,” Neural Processing Letters, vol. 9, no. 3, pp. 293–300, 1999.
Y. Xiong and R.Zuo, “ Recognization of geochemical anomalies using a deep autoencoder network,”Computer Geosci-UK, vol.86, pp. 75-82, 2016.
Zejian Shi, Minyong Shi, Chunfang Li,” The prediction of character based on recurrent neural network,” IEEE computer society, Wuhan China, 2017
Taro Ishitakl, Ryolchlro Obukata, Tetsuya Oda, Leonard Baroll, “Application of deep recurrent neural network for prediction of user behavoiur in Tor Network,” 31st International Conference on Advanced Information Networking and Application Workshops, 2017.
Marek Daabrowski, J. Gromada, T. Michalik,” P practical study of neural network-based image classification model trained with transfer learning method,” Federated Conference on Computer Science and Information Systems pp. 49-56, 2016.
B. Wang, K. Yager, D.Yu, Minh Hoai,” X- ray Scattering image classification using deep learning,” IEEE Winter Conference on Application of Computer Science,2017
Nur Anis Mohmon and Norsuzila ya acob,” A review on classification of satellite image using artificial neural network (ANN),” IEEE 5th Control and System Graduate Research Colloquium,2014.
R.Jyothi, Y.K. SundaraKrishna, V. Srinivasa Rao,” Paper Currency recognition for color images based on Artificial Neural Network,” International Conference on Electrical , Electronics and Optimization Techniques ( ICEEOT), 2016.
M.Abadi, A. Agarwal, p. Barham, E. Brevdo, Z.f. Chen, C. C itro,et at.,” Tensorflow : Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv: 1603.04467,2016.
R. Collobert, S.Bengio, and J.Mariethoz, “ Torch: a modular machine learning software library,” Idiap, No.EPFL-REPORT-82802, 2002.
R. AI-Rfou, G.Alain, A.Almahairi el at.,” Theano ; a python framework for fast computation of mathematics expression,” arXiv preprint arXiv : 1605.02688,2016.
http://www.wpclipart.com/medical/anatomy/cells/neuron/neuron.png.html
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.
