Log Indexing and Rotation for Optimized Archival in Graylog
You have Gigabytes or Terabytes of logs coming in on a daily basis, but now what do you do with them? Should I keep 10 days, 30 days or 1 year? How do I rotate around my logs and configure them in Graylog? Let's talk about the best practices around log retention and how to configure them in Graylog.
Log rotation can be done for various reasons ranging from meeting a compliance goal, keeping the size of the index down for faster searches or to get rid of data after a set amount of time. Graylog enables you to rotate the indexes based on a few methods. Message count, will rotate the index after a number of messages have been written into the index. Index size rotates the index after the size defined has been reached and Index time rotates the index after the specified time. All have their uses, with the most common being index time. With index time, you can meet the three months online, but setting the index to rotate every week, and keeping 12 of them online, or every day and setting the max indices to 90.
To get into your index rotation strategy you need to go to: System -> Indices and select Edit next to the index you would like to modify. In this example below, we have a 1-month rotation and are keeping 12 indexes for a full year of data.
After you set your rotation strategy, you will also need to select your retention configuration. Let’s have a quick overview of each choice and why you might want to pick that option.
● Delete: When you delete the indices you are having the minimal resource consumption by Elasticsearch and removes the index from disk thus saving disk space. This would be a good setting for operational data, which after a couple of weeks has no value (System Metrics, Flow Data, etc.)
● Close: Closing an index, blocks Elasticsearch from writing more data into it, but keeps it online and maintains the index’s metadata so you can still search it.
● Do Nothing: No resource saving on elasticsearch, and will keep the index open and on disk until manual removal.
● Archive: Discussed Later
If you would like to archive your data you can use the Enterprise version of Graylog to set up a backend storage location, allowing older indexes to be moved and compressed for long term storage. After you have a mount location on your server and configured that as the storage location of your archives, you can then set up the “Archive Index” retention strategy and give the number for when it will start moving them to the archive. In this example, the 21st index will be moved to the archive location, and the index closed. Many will choose to delete the index after archiving, as the archive feature allows for restoration if needed.
Understanding your log retention and rotation strategy is essential in any deployment of Graylog. With correct rotation strategies, your logs will be collected and maintained as expected, and allow for a fast and useful log aggregation tool.