IoT Data Analytics Pipeline using Elastic Stack and Kafka - V, Jayashri; Nath, K Badari (International Journal of Computer Sciences and Engineering, 2020)
With Internet of Things enabling advanced connectivity of devices and systems, different types of sensors are being used in various use cases. These generate huge volumes of data which can be processed and analysed over distributed systems for the benefit of industries like healthcare, supply chain, agriculture, transportation and so on. Elastic Stack is a set of open source solutions from Elastic. It is designed to help users move data of any kind from any type of source and index, search, analyze, and visualize that data in real-time. It consists of Filebeat for shipping the logs, Logstash for extracting and indexing, Elasticsearch for storing, searching and analysing, and Kibana for visualizing. It can be used along with Apache Kafka which is an open source distributed streaming platform generally used to build data streaming pipelines that move data between systems reliably. This paper explores the fundamentals of Apache Kafka and the Elastic stack and how Kafka can be used in conjunction with the Elastic stack for collecting and analyzing the data generated by networked sensors for IoT applications. It lays out the design for a system based on open source components, that monitors the data from the sensors on devices, performs analytics and alerts the concerned teams when required.