Introduction to Data Forwarder
When you have a distributed environment with many remote locations and one centralized IT staff, you want the logs in one central spot—without killing your network. Graylog helps with this issue with the Graylog Data Forwarder.
Data Forwarder provides the ability to forward messages from one Graylog cluster to another over HTTP/2. This centralizes logs messages from a distributed architecture into one cluster, allowing for centralized alerting, reporting and oversight into your logs.
In the Enterprise Integrations plugin after Graylog version 3.0.1, you can now create a forwarder output on the remote cluster, and a forwarder input on the destination cluster.
How It Works
The Forwarder Output (Graylog Source Cluster) is configured to forward messages to the destination cluster. First a message is collected near the source cluster, and then it is written to the on-disk journal to ensure no messages are lost incase of network outage. The messages will stay in the journal until the destination cluster is able to receive the message.
Messages are forwarded to the destination cluster, after they are done being processed by the pipeline rules of the source cluster, and at the same time they are written to Elasticsearch on the source cluster.
The Graylog forwarder is capable of sending logs at very high throughput rates. These can be affected by things like CPU Clock Speed, CPU Cores, available memory and network bandwidth. There are many configuration options to help with any network.
Once the logs have been received by the destination cluster, they are tagged to know which cluster they came from for quick differentiation.
Frequently Asked Questions
Can logs be compressed?
Yes, upon creation of the Forwarder Output, there is an option to enable compression for transport.
Can you use load balancing?
The Forwarder uses HTTP/2 (gRPC) for transport. If using more than one Concurrent Network Sender, then load balancing is supported. For more information see Load Balancing gRPC.
Can you use encryption?
Graylog Forwarder, can be encrypted with SSL/TLS encryption. This can be done with X.509 Certificates on each side of the transport.