Acting on data

You've got your data in, searched it, and transformed and viewed it. You know how Splunk Enterprise makes it easy to recognize, at a glance, when your data is telling you that something is wrong and action is needed. With alerts, there's no glance required: Splunk Enterprise can tell you the moment something happens--an action is logged, a value is reached, a threshold is passed, and so on--and even trigger actions based on what's happening.

Data is generated around the clock. Splunk Enterprise is always monitoring your data, and it gives you numerous different ways to observe trends and visualize your data at any time. But you don't have to monitor your data constantly to be able to identify when you need to act on it. With alerting, you can tell Splunk Enterprise when to inform the appropriate stakeholders to take action, or even tell Splunk Enterprise to initiate those actions itself. In this chapter, we talk about what alerts are in the context of Splunk Enterprise, the types of alerts available and how to manage them, what alert actions are and what Splunk Enterprise can do when an alert is triggered. We'll also talk a bit about creating custom alert actions and integration with the Splunk SDK for JavaScript.

DEVIf you're just interested in creating custom alert actions, which are developer-crafted, customizable actions that are initiated when an alert is triggered, you might want to go directly to the "New adventures require new tools: alerting" chapter in the Journey.

All about alerts

An alert is a notification mechanism to let you know when an event of interest has occurred. You can configure Splunk Enterprise to trigger an alert whenever a search returns a result that matches a threshold or trend setting that you specify.

When getting started with alerts, consider the following:

  • Conditions: What are the events that you want to know about?
  • Type and frequency: How often do you want to be made aware of events?
  • Alert action: What should happen when an alert occurs?

Consider also the different types of alerts available in Splunk Enterprise:

  • A per-result alert is based on a real-time search, and is triggered whenever the search returns a result that you specify. For example, a per-result alert could be triggered by a failed login attempt.
  • A scheduled alert runs a search according to a schedule that you set, and is triggered by search results that you specify. For example, you could create a scheduled alert that runs a daily search of nationwide song airplay and notifies you which ten songs were played the most.
  • A rolling-window alert is based on a real-time search, and is triggered by conditions that occur during a window of time, both of which you specify. For example, you could create a rolling-window alert that notifies you if an employee's keycard is used to access company resources more than ten times within a one-hour period.

If you want more information about the different alert types, along with example scenarios for each, see "Alert types and scenarios" in the Alerting Manual.

Alert scenarios

You've probably already come up with a few possible uses for alerting in your Splunk Enterprise deployment. Here are a few more to get you thinking even more about how to make alerting work for you:

  • When a server hits a predefined load or an extreme burst in activity occurs, have Splunk Enterprise notify you.
  • When web server performance slows to below a certain average response time, have Splunk Enterprise notify you and execute a script to disable certain functionality for trial users so that your web site remains usable for paying customers.
  • When the load of your application is very low, have Splunk Enterprise start an action to perform certain background processing tasks that are not time critical, but resource intensive. 
  • When a specific type of error occurs on any host, have Splunk Enterprise notify you.
  • When CPU on a host hits above 90% for a certain period of time, have Splunk Enterprise notify you through multiple channels.
  • When the usage patterns of a particular user are flagged as anomalous, have Splunk Enterprise notify you and lock the account.
  • Once a day, trigger an alert that notifies you when the number of items sold that day (or some other benchmark figure) is less than a certain number.
  • Once an hour or so, trigger an alert that notifies you when the number of 404 or other errors exceeds a certain threshold.
  • Use a rolling window alert to trigger an alert when a user has three consecutive failed logins or unsuccessfully tries to access restricted network assets within a certain time period.
  • Use a rolling window alert to trigger an alert when a host is unable to complete a repeating automated task such as an hourly file transfer to another host.
  • Use a webhook to open a bug in your company's bug tracking system when a certain debugging statement is logged.
  • Use a webhook to post a message to your IT department's chatroom when a server is unreachable or an unusual number of errors has been returned.

More alert action scenarios are available in topic "Alert examples" in the Alerting Manual.

Creating a new alert

You always start the alert creation process in the same way: create a search and then save it as an alert.

Search results screen

Enter a search query. In the upper-right corner, click Save As, and then Alert. The Save As Alert window appears.

Save as alert screen

In the Settings section, you enter a title for the alert, an optional description, specify whether the alert is shared with others or private, and what type it is.

  • For a per-result alert, choose Real-time next to Alert type, and then Per-Result next to Trigger alert when. Per-result alerts are always real-time alerts, because you want to be notified as soon as the trigger condition occurs.
  • For a scheduled alert, choose Scheduled next to Alert type. Next to Trigger alert when, choose whether to trigger the alert based on the number of search results, hosts, or sources during a scheduled search, or choose Custom to enter a custom condition to evaluate against the search results at the scheduled time.
  • For a rolling-window alert, choose Real-time next to Alert type. Next to Trigger alert when, choose whether to trigger the alert based on the number of search results, hosts, or sources during a rolling window of time, or choose Custom to enter a custom condition to evaluate against the search results during the window. You'll then need to enter values for the number of results and the time window.

Select the Throttle checkbox to suppress alert triggering for a period of time that you define. This prevents you from being overwhelmed with triggered alert actions, should your alert be triggered more frequently than you had expected.
The final step in creating an alert is specifying a trigger action, or alert action. We go into detail about alert actions in the next section.

Alert actions

Alert actions are initiated when an alert is triggered. You specify an alert action in the final step of creating an alert.

Available alert actions in Save as alert screen

You have several options when it comes to alert actions:

  • Add to Triggered Alerts: Great for testing out alert triggers, this action simply adds the triggered alert to the Triggered Alerts list in Splunk Enterprise. To access the Triggered Alerts list, go to Activity > Triggered Alerts.
  • Run a script: This alert action invokes a custom script. Store custom scripts within /bin/scripts or within an app's /bin/scripts directory. Keep in mind, however, that running a script when an alert is triggered has been deprecated in Splunk Enterprise 6.3, and therefore is not recommended for use.
  • Send email: By far the most common alert action, this one sends an email when an alert is triggered. For more information about this alert action, see "Email notification action" in the Alerting Manual.
  • Webhook alert action: This alert action lets you invoke a webhook when the alert is triggered. Webhooks allow you to make HTTP POST requests on a particular web resource. These resources can include ones you create yourself or that are developed and hosted by services such as Zapier or Twilio. For more information on webhook alert actions, see "Use a webhook alert action" in the "Alerting Manual." You should also check out the section on webhook alert actions in the "New adventures require new tools: alerting" chapter in the Journey.
  • Custom alert actions: These are available if they've already been installed and configured on your Splunk Enterprise instance. Custom alert actions are enabled by the custom alert action framework. With the framework, developers can create, package, and distribute customized trigger actions based on any alert use case imaginable.

To manage or configure the installed alert actions, go to Settings > Alert Actions. From here, you can click Browse more to search for and browse custom alert actions that others have created and posted to Splunkbase.

List of alert actions

The next section describes custom alert actions in more detail and tells you where to go to learn how to create them.

Custom alert actions

The custom alert action framework lets developers add new functionality and integration for alerts. Custom alert actions, like alerts, can be access control list (ACL)-managed, packaged, and distributed within apps, but they are fully modular, and can be reused throughout Splunk Enterprise.

DEVA custom alert action is packaged within an app, and is made up of several files laid out in a set directory structure. A generic directory structure for a typical custom alert action appears within detailed instructions for creating a new custom alert action in "Custom alert actions overview" in the Developing Views and Apps for Splunk Web Manual. We also list the structure of our reference app's custom alert action--along with an account of our real-life experience creating a custom alert action from scratch--in the "New adventures require new tools: alerting" chapter in the Journey.

The process of creating a custom alert action is straightforward but multifaceted. To ease the process, we've created a custom alert action template and included it with the PAS reference app download package. In the spikes folder, open the alertaction_app_template folder and you'll see all of the necessary files, already placed in the correct file structure. From there, follow the four basic steps for building a custom alert action into an app:

  1. Create configuration files. (We've done this step for you, but you should know what is in the configuration files and what they do before you proceed.)
  2. Create a script.
  3. Define a user interface.
  4. Add optional components.

UXDepending on how you engineer your custom alert action, you may need your users to set it up first. For example, our reference app's custom alert action requires users to set up their JIRA server and other configuration settings before they can assign the alert action to an alert. Be sure to add a setup step to your documentation.

Splunk SDK for JavaScript integration

The Splunk SDK for JavaScript gives you programmatic access to your Splunk Enterprise instance's triggered (or "fired," in the SDK's lexicon) alerts. Once you integrate the SDK's alerting classes into your JavaScript app, you can retrieve fired alerts, fired alert groups, fired alert group collections, and their properties. You can also retrieve information about fired alerts of specific saved searches.

DEVThe term "app" in this context refers to an external application developed in JavaScript that interacts with your Splunk Enterprise instance. It does not refer to a Splunk app that runs on Splunk Enterprise.

For detailed information about how to use the SDK for alerting, including code examples, see "How to work with alerts using the Splunk SDK for JavaScript" in the Splunk SDK for JavaScript documentation. The Splunk SDK for JavaScript download package also contains complete code examples that you can run in your browser, in the files examples/firedalerts.js and examples/firedalerts_async.js.