Logstash Reviews & Insights

We investigated (and use) Logstash as apart of our ELK (ElasticSearch, Logstash and Kibana) stack. We input data from websites that post daily updates. This data is read into Logstash from RSS feeds, transformed in Logstash, then Sent to ElasticSearch / Kibana for visualization and reporting. Our problem was we needed to scrape large amounts of unstructured data from multiple sites that provided users the ability to post information. This information was free form. To make matters more interesting, these sites did not have an open API to query the data directly, so writing a simple cURL bot would not suffice. Therefore, we turned to ELK, due to previous projects where we employed ELK successfully. So the primary Return On Investment (ROI) is the fact that we had a fast, tightly Cohesive (but loosely coupled) stack of software that we knew how to configure and integrate well with our Laboratory.

Perfect for projects where ElasticSearch makes sense: if you decide to employ ES in a project, then you will almost inevitably use LogStash, and you should anyways. Such projects would include: 1. Data Science (reading, recording or measure web-based Analytics, Metrics) 2. Web Scraping (which was one of our earlier projects involving LogStash) 3. Syslog-ng Management: While I did point out that it can be a bit of an electric boo-ga-loo in finding an errant configuration item, it is still worth it to implement Syslog-ng management via LogStash: being able to fine-tune your log messages and then pipe them to other sources, depending on the data being read in, is incredibly powerful, and I would say is exemplar of what modern Computer Science looks like: Less Specialization in mathematics, and more specialization in storing and recording data (i.e. Less Engineering, and more Design).

My primary use case for Logstash is ingesting log files into a local Elasticsearch&Kibana Docker container so that I can easily search though the logs better. My favorite feature is the grok parser as it is easy to decompose complex regular expressions into simplified patterns. Logstash has a plethora of available plugins, but the out of the box connections have addressed all my needs thus far.

Logstash is well suited for tight integration into the ELK stack, but it is also flexible enough to support other ingestion workloads similar to any other message bus or queueing framework. Compared to a message queue, though, Logstash also supports various filter and enrichment plugins that allow you to manipulate data as it passes through the system.

We were introduced to Logstash via the ELK stack. Our application generates many data points, and one of many patterns we had seen was to store it in elastic search. Logstash was the router which actually sent data to ES, and received data from our applications. The power of Logstash was quickly realized when we increased the number of source applications generating different kinds of data, but the ingestion point remained same. Since then Logstash has been one of my team's favorite tools and is one of the few things which you can set once and forget. We are happy users of Logstash and expect to use it more in the future.

Logstash is a must in an ELK stack, which I am sure is going to be the #1 case. At any point when you have several sources, Logstash can be the common point to aggregate, and categorize those data. Then send this new data to its destination. Very handy. It is free and open source.

It may not be appropriate to analyze data-sets dependent on each other but from a different data source. Reason being Logstash works on data at hand, and not wait for other data to arrive. It would be unwise for Logstashh to handle complicated, long-running transformations because this is injected and ejected. The faster you do it, the safer.

Logstash is a part of the ELK stack in the private Telcocloud datacenter I used in Nokia. The Telco datacenter infrastructure has a lot of logs to be analyzed, and Logstash acts as a log shipper on the Opensatck-based infrastructure. It helped collect logs from various sources and then processed them in the required format to be displayed on Kibana for analysis & Grafana for visulisation for graphs

Logstash is a good choice for a full-fledged VM-based or bare-metal server data center because there is no resource crunch in a production-grade data center. It collects multiple inputs like logs, metrics, and events from diverse sources (files, syslog, Beats, Kafka, cloud services). It only loses some points in K8s-based infrastructure because there is still scope for improvement for Logstash in supporting K8s observability and auto scaling.