A Survey on Map Reduce Algorithm for Big data analysis using Hadoop, Pig and Hive utility tools

Authors

  • Kadre A Dept. of Computer Science and Engineering, MITS, Bhopal-India
  • SR Yadav Dept. of Computer Science and Engineering, MITS, Bhopal-India

Keywords:

Map reduce, big data, Hadoop, HDFS, Pig & Hive, Flight data

Abstract

Data is growing at a rate which cannot be handled by the traditional methods of computing. To store and process such data new data analysis and storage techniques have emerged over the last few years. Hadoop is one such parallel processing open source framework which provides distributed storage and processing of big data. This paper introduces Big Data, a new platform which enables accessing, manipulating, analyzing, and visualizing data residing on a Hadoop cluster. In this paper a survey is done on big data analysis using Hadoop and other utility tools like Pig and Hive. The majority of large-scale data intensive applications executed by data centers are based on Map-Reduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a large fraction of the data center’s overall costs. Therefore to minimizing the energy consumption when executing Map-Reduce jobs is a critical concern for data centers. In this survey Flight data has been analyzed in terms of the mentioned parameters such as time complexity and energy consumption information’s are retrieved using Hadoop.

References

Toshimori Honjo, Kazuki Oikawa "Hardware acceleration of Hadoop Map-Reduce" in 2013 IEEE International Conference on Big Data.

Ming Meng, Jing Gao, Jun-jie Chen "Blast-Parallel: The Parallelizing Implementation Of Sequence Alignment Algorithms Based On Hadoop Platform" in 2013 6th International Conference on Biomedical Engineering and Informatics (BMEI 2013).

Madhury Mohandas, Dhanya P M "An Approach for Log Analysis Based Failure Monitoring in Hadoop Cluster" in 2013 IEEE.

Ilja Kromonov, Pelle Jakovits, Satish Narayana Srirama "NEWT - A Resilient BSP Framework for Iterative Algorithms on Hadoop YARN" in 2014 IEEE.

Lena Mashayekhy, Mahyar Movahed Nejad, Daniel Grosu, Dajun Lu, Weisong Shi "Energy-aware Scheduling of MapReduce Jobs" in 2014 IEEE International Congress on Big Data.

Oscar D. Lara, Weiqiang Zhuang, and Adarsh Pannu "Big R: Large-scale Analytics on Hadoop using R" in 2014 IEEE International Congress on Big Data

Kiran M., Amresh Kumar "Verification and Validation of Parallel Support Vector Machine Algorithm based on Map-Reduce Program Model on Hadoop Cluster" in 2013 International Conference on Advanced Computing and Communication Systems (ICACCS -2013), Dec. 19 – 21, 2013, Coimbatore, INDIA

Downloads

Published

2025-11-10

How to Cite

[1]
A. Kadre and S. Yadav, “A Survey on Map Reduce Algorithm for Big data analysis using Hadoop, Pig and Hive utility tools”, Int. J. Comp. Sci. Eng., vol. 3, no. 10, pp. 52–57, Nov. 2025.

Issue

Vol. 3 No. 10 (2015): IJCSE October Edition

Section

Survey Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.