Application logs hold valuable information. They can reveal a security breach or disclose attempts at one. They can provide an early warning of poor performance. Studying them might show that more disk space or a bigger memory allocation is necessary. Sometimes they’ll reveal unauthorized or excessive usage.
But they’re a pain to manage. Logs are usually an afterthought for software developers and are rarely cleanly designed. They serve many purposes, including debugging, recording usage, reporting security issues, and analyzing performance. All this information is jumbled together.
You have to dig through lots of irrelevant information to find the important stuff. When you find it, it’s apt to be cryptic. Different applications have different log formats. How do you stay on top of the important information without devoting all your time to reading logs?
Reading unfiltered log files is rarely a productive use of your time. Think of them as the raw material for notifications. Scripts and notification systems can help you to extract the important information and learn about critical events quickly.
Normalizing logs
The more you can normalize your logs, the easier they’ll be to manage. This can take quite a bit of effort up front, but it will save time after that.
Study the logging options for each application you’re interested in. Figure out what the most useful logging level is. “Info” and “debug” entries generally are overkill, unless you’re debugging the app. Beyond that, it’s not an exact science. One application may put everything important into “severe” entries, while another may use “warning” for dangerous situations that don’t actually crash. You have to learn that from experience.
If an application can use standardized logging with the operating system or a server, that makes life easier. Applications that follow the Syslog standard are especially easy to integrate with processing tools. They treat logging as sending a message to a receiver, rather than necessarily writing to a file. Logging servers use Syslog to collect entries from multiple machines on a network. Entries have a standard format, consisting of several headers and structured data. It’s a messaging format, not a print format, and log files usually don’t reproduce full Syslog messages.
Many Windows applications have an option for writing to the Application Event Log, or they use it by default. You can use Event Viewer to examine logs for all these applications in a consistent way.
Mac OS offers Unified Logging and Activity Tracing APIs to standardize log output. The Console application lets you view the combined logs or search for particular kinds of entries.
Choose your log formats with an eye to further processing. Plain text is generally the most human-readable option, but other options, such as Syslog, XML, or CSV, are easier to script or to use with available tools.
Scripting logs
When you’re dealing with diverse logging formats, scripting can make them more manageable. Converting them to a common representation, making time stamps consistent, and extracting the most important entries can make reviewing them easier.
In a lot of cases, repetitive and uninteresting messages bloat logs, and there might not be any way to configure logging to get rid of them. A filter can remove or highlight any entry that contains a key phrase.
Software tools
When the logs are in standardized formats, tools are usually available off the shelf, saving you the trouble of writing scripts.
Seq is a popular tool for .NET application logs or Node.js applications. This log server receives data over HTTP and has powerful filtering options to separate the noise from the important information. It includes integration for sending events to Slack or HipChat, letting you view all important entries in one place.
Nagios Log Server is a log analysis tool available for Windows, VMWare, and Linux. It monitors logs and sends alerts based on defined patterns and thresholds. Logging information from multiple machines can be viewed in one place through a GUI.
Sumo Logic is a cloud-based log server with an emphasis on analytics. It supports searches and custom alerts. Syslog-compatible applications work well with it.
Notification options
Current tasks take up all the time you can give them. In spite of your best intentions, you aren’t going to find time every day to read the logs. Unless you’re running into constant issues, they’re boring. You’d rather be notified when something important happens. Some logging systems support notifications, and log processing applications or custom scripts can add more.
Catching an urgent log entry quickly can avert major trouble. It could indicate that users are encountering problems, that an application is failing to perform some functions, or that there’s a critical security issue. Malware often runs undetected for months because no one notices the warnings. Important messages go unnoticed because they’re buried among routine ones. They can be very costly if they don’t reach someone’s attention.
Email notification is a common option. An application’s log configuration may let you get the highest-priority entries as email messages, either singly or as periodic updates. SMS notification is another widely available option. Make sure to use it conservatively if you don’t want your phone to fill up!
Notification to chat-oriented applications such as HipChat and Slack is increasingly popular. If your workday is already built around one of those applications, it might be the best place to get important log information.
A lot of administrators just find logs too much to deal with, so they rarely look at them. They could be missing important security problems and performance issues. Spending some time to get your logs in order makes them useful, and you’ll be able to catch critical problems that would have slipped by.
