As more and more IT infrastructures switch to the cloud, the need for public cloud security tools and log analytics platform has increased manifold. An organization generates a huge amount of data daily, which usually contains unstructured log. To structure these logs, the companies require an analysis tool. Elastic Stack is a complete log analysis solution that helps in deep searching, analyzing, and visualizing logs generated from different machines.
What is ELK Stack?
ELK is a combination of three open source tools: Elasticsearch, Logstash, and Kibana. All three tools are explained separately below:
Elasticsearch: As a full-text distributed NoSQL database, Elasticsearch is a free, open-source tool that allows for real-time data search and analysis using documents instead of schema or tables. Some of its characteristics are:
As a distributed search engine, Elasticsearch index is split into elements called shards. These shards are distributed across multiple nodes and are managed automatically by Elasticsearch. Following are some features of shards:
ELK Plugins: There are a number of plugins available for Elasticsearch to make it more efficient.
Logstash Functionalities: As mentioned earlier, Logstash is used to collect data from different sources and sends it to Elasticsearch. Some of Logstash functionalities are listed below:
How Logstash works?
Logstash configuration file location: /etc/Logstash/conf.d/Logstash.conf
path => (path of the file)
Apply filter based on the requirement
Elasticsearch details go here
Logstash Filter Plugins:
Logstash Output Plugins:
While some organizations use ELK, there are a few who use EFK (Elasticsearch, Fluentd, and Kibana). ELK requires Beats (explained in the later section) to send logs to Logstash, whereas Fluentd itself runs a daemon set and sends log directly to Elaticsearch. Below are the differences between Logstash and Fluentd.
Kibana: Kibana is used for visualizing and analyzing data in a graphical way. Following are the features of Kibana. It helps in searching, viewing, and interacting with the data stored in Elasticsearch indices; execute queries on data and visualize results in charts, tables, and maps; add or remove widgets; and share/save/load dashboards (import/export).
Picture 4: Kibana Panel
Let me explain each component in the left panel in detail:
Beat: Collecting logs and metrics for ELK Stack
ELK stack has an essential component, called Beats, that helps in the collection of data to be sent to Elasticsearch of Logstash. There are different types of beats available to collect a variety of data and it runs in each node as a daemon set.
ELK stack can be implemented in almost all kinds of business structures as it has the capability of enhancing and visualizing the raw data. E-commerce is the talk of the town these days and with millions of transactions going simultaneously, it generates lots of data in the log format and requires cloud security tools. Implementing ELK stack in an e-commerce platform can yield very productive results as it can easily catch the fault captured in logs by giving it a proper format and visualization, thus, saving a lot of time and effort.