Centralized Docker Container Logging With Native Graylog Integration
Docker containers have solved many technical challenges, but also introduced a few new ones. Log collection from Docker containers has been difficult because containers do not persist for a long time, and thus don’t leave a reliable trail of historical information. As a result, tasks that rely on log data, such as application debugging, have become more difficult.
To solve the Docker log management problem, we added our our own GELF sender as a logging driver to the Docker 1.8 core platform. The GELF logging driver enables a container to send STDOUT and STDERR messages formatted as GELF messages to a remote, centralized system like Graylog. The GELF logging driver replaces log forwarders or manual methods for collecting logs inside or outside of a container. As log forwarder containers are removed, users will see a decrease in configuration requirements, image bloat, and CPU/memory utilization.
Collecting Container Logs with Docker 1.8
Start with an existing Graylog instance or use our pre-configured Docker image.
docker run -t -p 9000:9000 -p 12201:12201 graylog2/allinone
Next, create a UDP GELF input through the Graylog web interface.
Now Graylog is listening for your data and you are ready to send logs from any container to Graylog.
docker run --log-driver=gelf --log-opt gelf-address=udp://<GraylogIP>:12201 busybox echo Hello Graylog
We can collect not only the log message information from the container but also details around the container itself, which is important for troubleshooting. Additional context is very useful for figuring out which application and service was impacted if there was an issue.
Below is a list of additional log message fields included in every log message from Docker:
- Hostname – Name of the Docker host
- Container ID – Full ID of the container
- Container Name – Human readable name of the container
- Image ID – ID of the image used to create this container
- Image Name – Human readable image name
- Command – Command or entrypoint that is executed inside of the container
- Tag – A tag that was given on creation time to identify containers easily
- Creation time – A timestamp when this container was started
- Log level – Was the message send to STDOUT or STDERR?
These additional fields combined with the search functionality of Graylog, gives the user the ability to analyze the output of containers across multiple host systems.
Figuring out what went wrong with a Docker container is now much easier, you can search by host, image or even the container itself. Below is an example of searching by container ID.
Since we can collect all of the log messages centrally, it’s a simple process to search across all log messages from all Docker hosts and containers.
Another example would be to display all logs from a Redis cluster running in multiple Docker containers. Assume we run a cluster with more than one Redis instance and we don’t know the exact container IDs. It is possible to simply search for the image name because all containers were built from the redis image.
In the same way all other fields can be used to find the necessary information in a tangle of log data.
To do further breakdowns of log data, use Graylog’s quick values feature. With one click a user can analyze every field data from Docker. This can be used to get an overview of most used images or commands executed by users.
Additionally Graylog provides statistical data for numerical fields. With this it is possible to get mean, min, max or standard derivation for certain log data.
All searches from above can be used to create widgets and dashboards in Graylog to provide a better overview for Docker admins. Just click on the top-right to add the data in question to a Docker dashboard. This dashboard can then be shared with other users or put on wall-mounted display.
Graylog is able to send out alerts based on certain log events. For example, if we want to send out alerts whenever a Tomcat container runs out of memory, we can create a simple stream looking for ‘OutOfMemoryError’ in the Tomcat container message. Based on the type of error and the number in a given time period we can send out alert emails, notifications or other kinds of alerts to different systems i.e. Slack, PagerDuty etc.
Log management used to be a challenge with Docker, but it’s now a lot easier since native GELF support is integrated into the core Docker platform. Graylog users will also get the benefit of advanced log management features working right out of the box like alerting, log retention and dashboards.
System Administrators and Application Developers are now able to quickly zero in on and analyze problems in their Docker environments without the additional complexity and configuration/management overhead of the past.
Tell us about your experience! We’d love to hear how your Docker logging experience changes with this new addition to the core Docker platform.