Today we are officially releasing Graylog v3.1.
This release brings a whole new alerting and event system that provides more flexible alert conditions and event correlation based on the new search APIs that also power the views. In addition, some extended search capabilities introduced in Graylog Enterprise v3.0 are now available in the open source edition in preparation for unifying the various search features.
Support for building search workflows with parameters remains a Graylog Enterprise function and will be enhanced in future releases once the search unification work is completed.
Please read on for detailed descriptions of each feature.
- DEB or RPM packages are available in our repositories. Check our documentation for details.
- Docker image
- OVA / Appliance
- Tarball (manual installation)
Please report bugs and any other issues in our GitHub issue tracker. Thank you!
New: Alerts and Events
Turn your log messages into normalized events by defining filters and aggregations and alert on them directly or correlate multiple events over time to express more complex conditions.
Log messages can take many different forms and usually contain a lot of noise. At the same time, not everything noteworthy in logs is necessarily an alert. In order to overcome this gap, Graylog now allows you to define events, through conditions such as filters and aggregations, which form the elements of alerts.
Because those events are also stored in Elasticsearch, all Graylog functionality can be used to query, organize and archive them, long beyond the lifetime of the raw, noisy logs that created them.
In Graylog Enterprise you can also correlate events over time to trigger notifications when a combination of events appear or even a certain combination of events does not happen in a certain amount of time.
For example if you are collecting logs about software deployments, you can define events that describe when service was stopped or started from its application server logs. By themselves these are not alerts, but describe something noteworthy about the service that is extracted from the noise of all the other informational messages in those logs.
By correlating stop and start events Graylog Enterprise lets you express a new alert event when, for example, a service that has stopped does not start again within 30 seconds. This new alert can be passed to Graylog’s notification system to trigger your incident management system, send emails or trigger the appropriate Slack channels.
Like other events, correlation events are also stored in Elasticsearch, and are available for further filtering, aggregation or correlation rules, allowing you to build very powerful alert expressions over your data.
In an information security context, events could describe successful and failed logins across a variety of platforms and correlation rules can alert you when a large number of failed logins are followed by a successful login for the same user, indicating a possible brute force attempt.
All events (and thus alerts) in the system have what is called a key: The key tells Graylog what the topic of the event is. In the case of logins it can be the user name and target IP address; in the case of software services it can be the host name and the name of the deployed application. In Enterprise Edition, different events that have the same key can be correlated, in other words they can be treated as a group.
Prior versions of Graylog could not produce keyed events thus requiring overly specific alert conditions and creating more work to produce the alerts you wanted. With the addition of keys, the same type of event can now describe different topics. In the context of login events this allows us to write a single condition describing what makes up a login event, and then scoping that to the subject of the event: the user that logged in. This means writing fewer alerts and less time required to define those alerts.
Events and alerts are all run as searches and Graylog now keeps track of which underlying data it has seen, which time frames it produced events and alerts for already, and it will ensure that there are no gaps in coverage - even if the search cluster is temporarily slow or disconnected.
Graylog Enterprise will also run event and alert searches across all Graylog nodes in the cluster in parallel, thereby greatly increasing the throughput when a large number of alerts are defined in the system.
Enhanced: Visualization and Aggregation
In Graylog 3.1 many visualization improvements have been completed. These visualization improvements range from creating different aggregation charts, to quickly marking key data in the search pane.
Starting off, we have introduced multi-point aggregation on data. Allowing grouping of the charts on multiple fields, refining the data set. An example would be searching on destination ports, summing total number of attempts into an aggregation chart. Now the source or destination IP address can be a second data point, breaking down all port 80 attempts by host.
On the Views search page, you can now highlight any value by clicking on the field in question and selecting “Highlight this value”. Under the left side and Formatting & Highlighting, you can adjust the color you would like to use as well.
Here you can see highlighting the `clientip` field and how it looks after highlighting has been performed. This allows for the rest of the logs on the page to have the same color for quick identification.
Coloring can also be changed in the charts by clicking on the legend below the chart as shown below
Inside the Aggregation window, you can now have a “percentile()” metric function for calculating the top percentile of your logs. Also, we have added the option for Bar Charts to have the data stacked.
You can now duplicate tabs under any current search. Duplicating your search tab, allows for quick manipulation of your query, while keeping all your current work in place. All your charts and filters will be copied to the new tab so you can continue your investigation. Don’t forget to save after you have made your perfect search!
Finally, Graylog now has syntax highlighting in the pipeline rule editor. Build your rules via the WebUI, and see in real time if they are formatted properly!
Let us know what you’d like to have included in our GitHub issue tracker.