Introduction to Data Forwarder
When you have a distributed environment with many remote locations and one centralized IT staff, you want the logs in one central spot—without killing your network. Graylog helps with this issue with the Graylog Data Forwarder.
The Data Forwarder provides the ability to forward messages from one Graylog cluster to another over HTTP/2. This centralizes logs messages from a distributed architecture into one cluster, to support both local teams and enable organization-wide data analysis where needed.
For example, an international financial company headquartered in San Francisco with a centralized IT staff working from there wants all of the logs in one central spot. The company is running 3 Graylog clusters in 3 different regions: San Francisco, New York City, Singapore, Frankfurt, and Switzerland. They use the Graylog Data Forwarder to centralize their logs messages from the distributed architecture into one cluster so that they can monitor, alert, report, etc. on their logs and still identify the origin cluster. In addition, by centralizing the log data, the team can perform searches across the different clusters. This is especially useful when you need real-time answers because of an outage, performance issue, security alert, etc.
How It Works
The Forwarder Output (Graylog Source Cluster) is configured to forward messages to the destination cluster. First a message is collected near the source cluster, and then it is written to the on-disk journal to ensure no messages are lost incase of network outage. The messages will stay in the journal until the destination cluster is able to receive the message.
Messages are forwarded to the destination cluster, after they are done being processed by the pipeline rules of the source cluster, and at the same time they are written to Elasticsearch on the source cluster.
The Graylog forwarder is capable of sending logs at very high throughput rates. These can be affected by things like CPU Clock Speed, CPU Cores, available memory and network bandwidth. There are also many configuration options to help with any network.
Once the logs have been received by the destination cluster, they are tagged in order to differentiate which cluster they came from. This allows the analysts to focus their efforts on a specific cluster which could help reduce the time required to analyze the log files.
Frequently Asked Questions
Can logs be compressed?
Yes, upon creation of the Forwarder Output, there is an option to enable compression for transport.
Can you use load balancing?
The Forwarder uses HTTP/2 (gRPC) for transport. If using more than one Concurrent Network Sender, then load balancing is supported. For more information see Load Balancing gRPC.
Can you use encryption?
Graylog Forwarder, can be encrypted with SSL/TLS encryption. This can be done with X.509 Certificates on each side of the transport.