IoT Data Analytics Pipeline using Elastic Stack and Kafka

Authors

  • Jayashri V Dept. of Computer Science and Engineering, R V College of Engineering, Bangalore, India
  • K Badari Nath Dept. of Computer Science and Engineering, R V College of Engineering, Bangalore, India

DOI:

https://doi.org/10.26438/ijcse/v8i5.144148

Keywords:

Apache Kafka, Elastic Stack, Filebeat, Logstash, Elasticsearch, Kibana, IoT

Abstract

With Internet of Things enabling advanced connectivity of devices and systems, different types of sensors are being used in various use cases. These generate huge volumes of data which can be processed and analysed over distributed systems for the benefit of industries like healthcare, supply chain, agriculture, transportation and so on. Elastic Stack is a set of open source solutions from Elastic. It is designed to help users move data of any kind from any type of source and index, search, analyze, and visualize that data in real-time. It consists of Filebeat for shipping the logs, Logstash for extracting and indexing, Elasticsearch for storing, searching and analysing, and Kibana for visualizing. It can be used along with Apache Kafka which is an open source distributed streaming platform generally used to build data streaming pipelines that move data between systems reliably. This paper explores the fundamentals of Apache Kafka and the Elastic stack and how Kafka can be used in conjunction with the Elastic stack for collecting and analyzing the data generated by networked sensors for IoT applications. It lays out the design for a system based on open source components, that monitors the data from the sensors on devices, performs analytics and alerts the concerned teams when required.

References

[1] M. Sheik Dawood, M. Jehosheba Margaret, R. Devika, “Review on Applications of Internet of Things (IoT)”, International Journal of Advanced Research in Computer Engineering & Technology, Vol. 7, Issue 12, pp: 841-845, December 2018

[2] Mantripatjit Kaur, Anjum Mohd Aslam, “Big Data Analytics on IOT: Challenges, Open Research Issues and Tools”, International Journal of Scientific Research in Computer Science and Engineering, Vol.6, Issue.3, pp.81-85, June 2018.

[3] Jordi Serra, David Pubill, Angelos Antonopoulos, Christos Verikoukis, “Smart HVAC Control in IoT: Energy Consumption Minimization with User Comfort Constraints”, The Scientific World Journal, 2014. doi:10.1155/2014/161874

[4] Poonam M. Mahajan, “WSN: Infrastructure and Applications”, International Journal of Scientific Research in Network Security and Communication, Vol.06, Issue.01, pp.6-10, 2018.

[5] Hong-Linh Truong, “Integrated Analytics for IIoT Predictive Maintenance Using IoT Big Data Cloud Systems”, 2018 IEEE International Conference on Industrial Internet (ICII), pp: 109-118.

[6] Jay Kreps, Neha Narkhede, Jun Rao, “Kafka : a Distributed Messaging System for Log Processing”, 6th International Workshop on Networking Meets Databases (NetDB 2011), Athens, Greece. Jun. 12, 2011.

[7] Marcin Bajer, "Building an IoT Data Hub with Elasticsearch, Logstash and Kibana", 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), Prague, 2017, pp. 63-68.

[8] D. Kalyani, Dr. D. Mehta, “Paper on Searching and Indexing Using Elasticsearch”, International Journal of Engineering and Computer Science Vol. 6, Issue 6, pp: 21824-21829, June 2017

[9] Paro A., ElasticSearch Cookbook - Second Edition, Packt Publishing

Ltd, Jan 2015

[10] Sunny Advani, Meghna Mridul, Prof. S. R. Vij, Manil Agarwal, Loya Palak A., Kasturkar Sanketa S, “Log Analytics Using ELK Stack on Cloud Platform”, International Journal of Advanced Research in Computer and Communication Engineering, Vol. 5, Issue 4, pp: 50-52, April 2016

[11] Manuj Aggarwal, TetraTutorials Team, “ElasticSearch, LogStash, Kibana (the ELK Stack) #3”, Packt Publishing, Jan 2018 [Online]

Downloads

Published

2020-05-31
CITATION
DOI: 10.26438/ijcse/v8i5.144148
Published: 2020-05-31

How to Cite

[1]
J. V and K. B. Nath, “IoT Data Analytics Pipeline using Elastic Stack and Kafka”, Int. J. Comp. Sci. Eng., vol. 8, no. 5, pp. 144–148, May 2020.

Issue

Vol. 8 No. 5 (2020): IJCSE May Edition

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.