You can help Splunk get more out of your logs by following these best practices.
One of the most powerful features of Splunk is its ability to extract fields from events when you search, creating structure out of unstructured data. To make sure field extraction works as intended, use the following string syntax (using spaces and commas is fine):
key1=value1, key2=value2, key3=value3 . . .
If your values contain spaces, wrap them in quotes (for example, username="bob smith").
This might be a bit more verbose than you are used to, but the automatic field extraction is worth the size difference.
Avoid using complex encoding that would require lookups to make event information intelligible. For example, if logs are in a binary format, provide tools to easily convert them to a human-readable(ASCII) format. Don't use a format that requires an arbitrary code to decipher it. And, don't use different formats in the same filesplit them out into individual files instead.
The correct time is critical to understanding the proper sequence of events. Timestamps are critical for debugging, analytics, and deriving transactions. Splunk will automatically timestamp events that don't include them using a number of techniques, but it's best that you do it.
Unique identifiers such as transaction IDs and user IDs are tremendously helpful when debugging, and even more helpful when you are gathering analytics. Unique IDs can point you to the exact transaction. Without them, you might only have a time range to use. When possible, carry these IDs through multiple touch points and avoid changing the format of these IDs between modules. That way, you can track transactions through the system and follow them across machines, networks, and services.
Avoid logging binary information because Splunk cannot meaningfully search or analyze binary data. Binary logs might seem preferable because they are compressed, but this data requires decoding and won't segment. If you must log binary data, place textual meta-data in the event so that you can still search through it. For example, don't log the binary data of a JPG file, but do log its image size, creation tool, username, camera, GPS location, and so on.
Developers like the ability to receive a stream of data over HTTP/S when possible, and with data structured so that it can be easily processed.
For some interesting reading about logging in JSON, read this entry in Paul's Journal.
Put semantic meaning in events to get more out of your data. Log audit trails, what users are doing, transactions, timing information, and so on. Log anything that can add value when aggregated, charted, or further analyzed. In other words, log anything that is interesting to the business.
Categorize the event. For example, use the severity values INFO, WARN, ERROR, and DEBUG.
Include the source of the log event, such as the class, function, or filename.
Multi-line events generate a lot of segments, which can affect indexing and search speed, as well as disk compression. Consider breaking multi-line events into separate events.
These operational best practices apply to the way you do logging:
If you log to a local file, it provides a local buffer and you aren't blocked if the network goes down.
Splunk knows how to index. It will catch up where it left off so you won't lose logging data.
Logs can take up a lot of space. Maybe compliance regulations require you to keep years of archival storage, but you don't want to fill up your file system on your production machines. So, set up good rotation strategies and decide whether to destroy or back up your logs.
Collect all the data you can, because the more data you capture, the more visibility you have. For example, collect these when you can:
Using Splunk universal forwarders, you can access log events that are saved to files and broadcast over network ports. But you aren't limited to files or streamsif you have log data that is buried in an application, device, or system, you can get to the data if you make it accessible via a transport, protocol, or API. Here are some examples of liberating your log data: