Analyzing Failures of a Semi-Structured Supercomputer Log File Efficiently by Using PIG on Hadoop - Palle, Madhuri Srinivas; Jyothsna, Konisa; B. Anusha (International Journal of Computer Sciences and Engineering, 2014)
Data sets used to fuel the recently popular concept of business intelligence are becoming increasingly large. Conventional database management software is no longer efficient enough however; parallel database management systems and massive data-scale processing systems like MapReduce indeed look promising. Although, MapReduce is a good option, it is difficult to work with, as the programmer would have to think at the mapper and reducer level. In this paper, we present a simple yet efficient way to mine useful information where a program can be written as a series of steps. We have queried a supercomputer log file using Apaches Hadoop and PIG, obtained results as to when and why the supercomputer had failed and compared these results to that of a traditional program.