ContactSupportBlogPartner Portal

Understanding Log Management: Issues and Challenges

November 15, 2019

In order to fully understand all the log management issues that can occur, we must first take a step back in order to provide a detailed examination of what log handling is and what it entails.

Log messages - which are also known as event logs, audit records, and audit trails, to name a few of their other commonly used names – are events as they pertain to computing. These are generated or triggered by the software or the user. Like, for example, when the user interacts with the system by inputting a command on the keyboard. This means that almost every action on a system, user or software-made, creates an accompanying log message of that event.

From vital server processes that affect thousands, even millions, of computers and digital devices, all the way to the inner workings of a tiny iPhone app – everything creates a trail of logs. For large organizations, this can mean hundreds of gigabytes of raw log data every day, which can be both difficult to store as well as make sense of. But this is only one of the reasons that highlight the importance of log management.

So, with all that in mind, let’s explore all the biggest log management challenges of modern IT, and the solutions for these problems.

Log Management Issues - Log Correlation

As previously stated, businesses - especially big ones - generate an astounding amount of data  on a daily basis. So while most log management services are capable of collecting this data, correlating it is another matter entirely.

Simply put, this is the difference between “good” and “bad” data activity. By itself, log management software can’t differentiate between the two of them, it merely gathers the data. Without appropriate protocols and advanced analytics in place, a false positive can seem like a big problem, while a serious breach of security can mask itself to appear as innocuous standard activity.

The ability to sort through these threats according to the level of seriousness and weed out the genuine complications from the unimportant issues is needed in every log management tool. For example, one of the ways this is done is through threat intelligence. TI analyzes the indicators of compromise to find possible loopholes to your security and help you contain them on time. The option to further aggregate this data is also critical, with content packs and other structured machine data file types and extensions being taken into account when collecting and analyzing data.

In this age of big data (that is becoming bigger and bigger with each passing second), log correlation is a vital part of log management - one that is used for everything from cybersecurity, system administration, and staying compliant with mandatory auditing procedures.

Log Management Issues - When (And What) To Automate

Log management is, like most types of software, heavily reliant on automatization. Data that is practically impossible to sort through by (human) hand is ordered, sorted, and analyzed by the program. Left to its own devices, your log management tool (LMT) will do its best to carry out the job according to its (pre)set parameters, but this is hardly an ideal solution.

New threats and problems are cropping up every day, and while your LMT is designed to help you identify and deal with them, a dedicated human in charge of steering its functioning and setup is needed to bring out the real benefits of log management best practices.

Knowing what to automate and what to do by yourself is an acquired skill in itself. And, like any other skill, it requires practice, time, training, and human devotion to get the most of it.

Log Management Issues - Storage and Archiviation

Archiving log files is a necessary way to reduce the amount of data you have to keep on your local servers and hard drives. Depending on your needs, log data is usually saved locally for up to 30 days, but you will often need to search for something that has happened several months (or even years) ago to identify the entry point of the problem or incursion.

This is done not just for security and system problems, but primarily for auditing purposes. Some regulatory audits require you to keep your log data for three to five years, and others may even mandate it be kept, for all intents and purposes, forever.

Of course, these logs aren’t stored in their raw form, but are compressed - in a lossless format - to reduce their size. You can choose to import them into your program whenever you want to, or have to, make use of them. This makes auditing a much more painless and agile procedure.

One of the biggest problems here is scaling - many log management solutions will charge you not according to how much data you process and store, but a flat package which can vary wildly. A difference between ten and fifty users (or ten and fifty gigabytes) is staggering. It is easy to go over the limit, and then you will have to pay the more expensive package. A good log management tool will keep this in mind - with scalability being one of its core features and pillars of design architecture.

Log Management Issues - Lack of a User-Friendly Interface

It may seem like a bit of a nitpick, but in actuality - few things can frustrate the user as much as an unintuitive, badly made interface.

User experience goes beyond mere interface design, but it is seen as a sort of foundation for how the user interacts with the application. A user interface (UI) that isn’t immediately clear and precise in its visual language can - and will - lead to mistakes, human error, and oversights.

The options - especially the most important features - should never be “hidden” from sight, but always be presented front and center, so that even an inexperienced user understands what the applications and its options do.

Speaking of which, a good UI has to have several viewing options. These can take the form of graphs and pie charts - which are much more visually distinct than numerical data, and can be used to quickly summarize information in company meetings, or to people who may not be as tech-savvy.

Log Management Issues - Reporting and Search Features

Underdeveloped searching and reporting features are common problems that still plague many log management tools.

With log file data that can easily measure in the terabytes, having the option to perform an in-depth and fast search is of paramount importance. For this reason, multi-threaded search is a must-have if you want to quickly and efficiently find pertinent data. Search queries should be easy to perform and provide all-encompassing information in their results.

Likewise, setting up reports has to be both intuitive and functional. Regardless of the time of day (or night), things can always go wrong and it is imperative that reports are sent out as soon as there is a problem, and that they reach the right people in charge. Reporting also has to be customizable, with the option to send out daily/weekly/monthly reports by email as needed.

Conclusion

There are many issues and challenges when it comes to log management. While it is an indispensable part of modern IT, not every log management tool is created equal and many lack the necessary, sometimes even basic, features required to provide an exhaustive log management solution.

Graylog comes in both open source and commercial versions, and has all of the above-discussed crucial features, as well as many more, and is a full-scale log management solution that is constantly being updated and expanded upon.

Written By
Nick Carstensen

Nick has been in the security industry for over fifteen years with experience in Security and the Log/SIEM Industry. Nick is currently a Technical Product Evangelist for Graylog, creating content and helping with their social presence.

@
NickCarstensen1
Add Graylog to your RSS feed
How to use RSS
RSS feeds allow you to see when websites have added new content. You can get the new content as soon as it's published, without having to visit the website. To start getting RSS feeds you will need a RSS feed reader on your device.
Back to Blog Posts

Stay In The Know

Get Graylog email updates and be the first to know about new content, product updates, and tips and tricks!