Logstash is a free and open source, data processing pipeline that collect the data from the various source and send it to the destination where data is needed. If we want to the send the log data or the message data to the ElasticSearch than logstash work as a pipeline for sending the data to the ElasticSearch.
Logstash is written on JRuby programming language which runs on the JVM, so we can run Logstash on different platforms. We can collect data from the different resources like Logs, Packets, Events, Transactions, Timestamp Data, etc.
Some of the General Features of logstash
- Logstash can collect data from different sources and send to multiple sources using pipeline.
- Logstash can also handle many types of log data like Apache Logs, celery Logs, Kibana Logs and many more.
- Logstash provides a variety of filters, which helps the user to find only that data which they need.
- Logstash can take unstructured data and provide it a structure.
- Logstash is free and open source and available under the Apache license version 2.0.
Some of the Key Concepts of logstash
When the data is provides to the logstash than it perform some operation over the data. This operation is known as Event. Their may be different types of event like filter the data. When the operation is perform than we get remove the unwanted data and can add addition field in it. This operation is perform after Input and before Output.
Input is the first stage in the Logstash pipeline, which is used to get the data in Logstash for any type of processing. It provides different types of plugins for collection of data, filtering the data, sending data etc. Some of the most commonly used filter plugins are – Grok, Mutate, Drop, Clone and Geoip.
This is the last stage in the Logstash pipeline, where the output events can be formatted into the structure form. The output file is sent to the different plugins. Some of the most commonly used plugins are – Elasticsearch, File, Graphite etc.
Difference between ElasticSearch and Logstash.
ElasticSearch work as a Search engine and Analytics engine where as Logstash work as a pipeline for processing the data and sending it to the ElasticSearch.
How Logstash work ?
Logstash work as a plugin based data processing that collect the data from the various source. It performs the data processing over the data and forward that data to the ElasticSearch or to the other resources where there is needed.