Logging overview

You can use Splunk out-of-the-box in your everyday development activities in order to improve the quality of your applications and reduce the time it takes to resolve issues. With Splunk you can quickly and easily search your log files. You can also make some slight code changes to improve the readability of your logs across the entire organization.

Logs can be a hassle to deal with: they don't have a real structure, you might not know what's in them if you didn't create them yourself, and the sheer number of files and types of logs can be overwhelming. On the other hand, logs are really useful: they contain a gold mine of analytic information, they can help you find problems, and they can tell you about your IT infrastructure, the behavior of your users, and identify potential attackers.

Chances are, you're already using logs for debugging purposes. But you can do more with your log files. Keep reading to learn how to get the most from your logs and how to set them up right. The results are worth it.

What you can learn from logs

There are two types of logging:

• Logging for debugging. As an application developer, you can use Splunk to look at your own application logs for debugging rather than hunting and pecking through files. Executives and others might be interested in viewing information from debugging logs to identify trends in system stability, categorize issues, and more.
• Semantic logging. You can also log "semantic" level events, which are events that record the state of business processes (web clicks, financial trades, cell phone connections and dropped calls, audit trails, and so on).

Typically, IT organizations develop systems that focus on specific silos of technologies, functions, people, or departments. Individually, these silos provide a limited view. But they all generate data—data that is non-standard, unstructured, high-volume, and growing every millisecond of every day. Whatever you're logging, every environment has some set of IT data to leverage:

• Logs contain a record of customer activity and behavior, product and service usage, and transactions.
• Logs contain a record of activity for IT components, such as applications, servers and network devices.

Often, companies must write their own applications from scratch to perform statistical analysis on the data contained in these events. Similarly, companies build dashboards by hand to display that analysis for historical comparisons, future projections, performance tuning, billing, security, and so on.

Splunk doesn't care about the format or schema of your data—queries and searches can be ad-hoc, and your data can come from any textual source. With Splunk, you can use the logs generated from your application to achieve the same powerful analytics without building special-purpose software. Logging semantic data in addition to debugging data can significantly enrich virtually all applications.

The following stub code shows a purchase transaction:

void submitPurchase(purchaseId)
{
    log.info("action=submitPurchaseStart, purchaseId=%d", purchaseId)
    // These calls throw an exception on error:

    submitToCreditCard(...)
    generateInvoice(...)
    generateFullfillmentOrder(...)
    log.info("action=submitPurchaseCompleted, purchaseId=%d", purchaseId)
}

This pseudo-code logs the purchase ID for each action. So with the two events generated from the above pseudo-code, you could learn a lot about your purchase transactions. You could:

• Graph your purchase volume by hour, by day, and by month.
• Find out how long purchases take during different times of the day and days of the week.
• Find out how long purchases are taking now compared to last month.
• Find out whether systems are getting slower.
• Find out how many purchases are failing, and graph these failures over time.
• Find out which specific purchases are failing.

Each of these items is a simple search in Splunk that you can save, schedule, and share by email. And there is no need to architect a complex solution—the more data you log, the more you get out of it. It's really that simple. Use Splunk instead of complex and inflexible architectures like RDBMS/SQL.

But it is important to set up your logging in the right way in order to get the maximum benefit with your application. You must start by thinking about how other people will view your logs. Events might get separated from the logs. Certain events might get copied and passed along to a variety of people inside and outside of an IT organization. Without using robust logging techniques, you can lose meaningful data, making it harder to derive insight from the logs. The logging best practices that follow can help make your debugging logs more relevant and powerful for you and others.

Now, let's get into the specifics with Logging Best Practices.