Exploring AWS !!

Day 61

ELK — ElasticSearch, Kibana, LogStash

Kibana: analytics and visualization platform, which lets you easily virtualize data from ElasticSearch and analyze it to make sense of it. It’s like an ElasticSearch dashboard where you can create visualization such as pie charts, line charts and many others. You can plot your web’s visitors onto map and show traffic in real time. Kibana is also where you configure change detection of forecasting. It can manage ports of Elasticsearch and Logstash, for Elasticsearch such as authentication and authorization. It uses data from Elasticsearch and sends queries using same REST API, it provides on interface for building those queries and lets you configure how to deploy results. This can save a lot of time because you don’t have to implement all yourself. You can build dashboards for metrics and for system administrators that monitors performance of servers such as CPU and memory usage, it also monitors errors for KPIs(Key Point of Interest) like sales, revenue, etc. Using Elasticsearch as analytical platform along with Kibana is perfect use case.

LogStash: Traditionally, it has been used to process logs from applications and send them to ElasticSearch, now its evolved to a data processing pipeline. The data that Logstash receives is handled as events like log file entries, ecommerce orders, customer chats, etc. These are processed and shipped by Logstash to one or more destinations. LogPipeline consists of 3 stages: inputs, filter and outputs. each stage can make use of plugin. An input file could be a file that logs will read events from given file. There are many input plugins like Kafka or HTTP endpoints. We can parse csv, xml, etc files. Output plugins are called stashes. So, main work is to receive access log entries, process them and send them to Elasticsearch or any stash.

X-Pack: pack of features that adds additional functionality to Elasticsearch and Kibana.
-Security: adds authentication and authorization to Kibana and Elasticsearch. In regards to authentication, Kibana can integrate with LDAP, AD or other provide authentication. We can add roles and users as well.
-Monitoring: enables you to monitor performance of Elasticsearch, logs, Kibana, CPU, memory disk space usage and other metrics. Can set up alerting to be notified.
-Alerting: Create alerts
-Reporting: Export Kibana visualization and dashboards to pdf files.
Elasticsearch SQL: query documents with proper query language named Query DSL. The Query DSL is flexible, but it might be a bit verbose. SQL queries can be sent over HTTP to Elasticsearch or use JDBC drivers. Elasticsearch translates SQL into Query DSL behind the scenes.

Beats: collection of data shippers. They are lightweight agents with single purpose that you install on servers which send data to logstash or Elasticsearch. Eg: FileBeat collection log files and log entries off to Logstash or Elasticsearch. It ships with modules for common log files, such as Nginx, Apache web servers. Types: log files Metric Beat: system and server metrics, Packet Beat: network data, Winlog Beat: windows event log, Audit Beat: audit data from Linux, Heart Beat: monitor service uptime.