Adding code: using JavaScript and Search Processing Language

We chose to use the Simple XML with extensions model to build the PAS app. Therefore, we use JavaScript to add in the extension logic and behavior where necessary. This chapter also discusses some of the searches we define using the the Splunk® search processing language (SPL™).

Practically everything in Splunk Enterprise is driven by searches behind the scenes. Searches are truly the workhorses of Splunk apps. Splunk Enterprise has its own search processing language, SPL. It includes a wide variety of useful commands: from those that correlate events and calculate statistics to those that generate, manipulate, reformat and enrich your data, build visualizations, and more.

DEV In a nutshell, a search is a series of commands and arguments. Commands are chained together with a pipe "|" character to indicate that the output of one command feeds into the next command on the right.

Whether you are new to the Splunk search language or not, the Splunk > Quick Reference Guide will be helpful. We used it as a reference frequently while creating the PAS app. In addition, Exploring Splunk offers a primer into SPL and a collection of recipes.

ARCH SPL is also extensible. You can define your own search commands for custom processing, data generation, and calculations. For more information, see the "Write Custom Search Commands" section in the Search Tutorial.

Case study: Building a complex query with lookups and time data overlays 

During our journey, one of the stories we implemented involved creating a new dashboard named Off-Hours Document Access. The intent of this dashboard is to let a user identify possible suspicious attempts to access the document repository, such as attempts by an employee to access the repository outside of their normal working hours. This requires some information that is not included in the log files: the logs record when an employee tried to access the repository and the status of the attempt (succeeded/failed), but not the employee's work schedule. We need to look up each employee's normal work schedule and correlate it with the information in the log file. We maintain this information in a separate list of employees. For this app, we've chosen to use a CSV file as our employee database, but in a real deployment we would look up this data in an external system or store the information in the KV store and use a batch process to keep it up to date with information from the external system. To package the employee data for use in the PAS app, we created a new Splunk app called pas_hr_info that contains the file and a transforms.con file to define the name of the lookup:

filename = employee_details.csv

For more information about defining a lookup in a static file, see Configure field lookups.

The following shows some sample records from the data in the file:


The shift_start and shift_end columns show the hour the employee starts and stops work. Notice how some of the shifts span midnight into the next day. The status column indicates the employment status. Later we will create a search and visualization to help identify attempts to access to the repository by terminated employees.

When we have our employee data, our query must relate the time the user tried to access the repository to the employee's official working hours to identify out-of-hours access.

The data model in the main PAS app now references the employee details lookup data to make it available in pivot searches. The following screenshot shows the detail of how the data model performs the lookup:

The data model also includes several eval expressions that use the lookup data to determine if a particular document access by an employee is outside of their normal working hours. Notice how some eval expressions refer to others:

Field name Eval expression
overnight if(shift_start>shift_end,1,0)
hour tonumber(strftime(_time,"%H"))
valid_time if((overnight==1 AND (hour >= shift_start OR hour <= shift_end)) OR
(overnight==0 AND hour>=shift_start AND hour<shift_end),1,0)
shift_day strftime(if(overnight==1 AND hour < shift_end,relative_time(_time,"-1d"),_time),"%a")
workdays_mv split(workdays,",")
valid_day mvfind(workdays_mv,shift_day)
valid_time_access if(valid_time==1 AND valid_day>=0,1,0)

The Invalid_Time_Access child event in the data model uses the calculated valid_time_access attribute as the constraint to identify the events that represent out of hours access attempts:

ARCHIt's better to perform any calculations in the SPL rather than in your JavaScript code. It's faster and easier to maintain.

The searches in the Off-Hours Document Access dashboard can now use the data model to search for out of hours access attempts, for example:

    <title>Documents Accessed Outside Working Hours</title>
    <searchString>| pivot ri_pas_datamodel Invalid_Time_Access count(Invalid_Time_Access) AS count SPLITROW _time AS _time PERIOD auto SORT 0 _time ROWSUMMARY 0 COLSUMMARY 0 NUMCOLS 0 SHOWOTHER 1</searchString>
    <option name="charting.chart">line</option>

The following screenshot shows an example of the chart that snippet above renders:

DEVIf you are testing or debugging a complex search that references app specific objects such as data models, be sure to use the Search dashboard in your app, rather than the generic one in the Splunk Search & Reporting app. In the PAS app, we can reach the Search dashboard at this address: http://localhost:8000/en-US/app/pas_ref_app/search. This is because the knowledge objects are saved in the app that you are using.

Combining pivots with searches

During our journey we uncovered a requirement to combine the results from several pivot searches against our data model with several standard searches into an aggregate set of results to use in a single visualization. We identified several possible approaches: write a search string that combines the pivots and regular searches, convert the regular searches to pivots and then aggregate all the pivots, or convert the pivots to regular searches and combine all the regular searches.

Combining pivots with searches in a single search

We can combine pivots together using the following approach. Given these two pivots:

| pivot ri_pas_datamodel Invalid_Time_Access count(Invalid_Time_Access) AS count SPLITROW _time AS _time 
| pivot ri_pas_datamodel Terminated_Access count(Terminated_Access) AS count SPLITROW _time AS _time 

Then provided the data model is accelerated, we can combine them using the following technique:

| tstats count from datamodel=ri_pas_datamodel where nodename=Invalid_Time_Access OR nodename=Terminate_Access groupby nodename _time span=auto | ...

For more information on the tsats command, see Splunk Search Command Reference.

We could then combine this with a regular search by using "|append []" command to attach the regular query.

However, it is difficult to identify opportunities for optimizing this search and this approach is best used to maintain backwards compatibility within an application when different developers have used different approaches (pivots and regular searches) in the past to build dashboards.

Converting everything to regular searches

We could convert our existing pivots to regular searches and then combine all the regular searches together. However, in our scenario we thought that this would result in a complex search that would be difficult to optimize manually.

Converting everything to pivots

The option we chose was to convert everything to pivots. This required some changes to our data model, but then it was easier to create the query that aggregates all the results using the technique shown previously. For the PAS app, optimizing for performance is more important than maintaining internal backwards compatibility. 

Example: Combining multiple searches

The search that drives the donut visualization on the summary screen is an example of a complex search that combines results from multiple sources. We converted the main searches to pivots before combining them together. The base search in the summary.xml file combines the results from three pivot commands by using the append command and then performs a lookup for additional information about the policy violations. Each pivot searches for different policy violation types: one for out of hours access, one for access attempts by terminated employees, and one for excessive access requests by an employee.

<search id="policy_violations_search">
                   | pivot ri_pas_datamodel Invalid_Time_Access SPLITROW department count(Invalid_Time_Access) as Invalid_Time_Access
                   | eval ViolationType="Invalid_Time_Access"
                   | rename Invalid_Time_Access as ViolationCount
        | append [ | pivot ri_pas_datamodel Terminated_Access   SPLITROW department count(Terminated_Access)   as Terminated_Access
                   | eval ViolationType="Terminated_Access"
                   | rename Terminated_Access as ViolationCount ]
        | append [ | search tag=pas tag=change tag=audit
                   | fields _time, user
                   | bucket _time span=1h
                   | stats count as ops by _time, user
                   | where ops!=0
                   | stats stdev(ops) as SD, avg(ops) as MEAN, latest(ops) as LAST_OPS by user
                   | where LAST_OPS > (MEAN + SD*1)
                   | lookup employee_details user
                   | stats count as ViolationCount by department
                   | eval ViolationType="Excessive_Access" ]
        | lookup violation_info ViolationType
        | eval TotalViolationWeight = ViolationCount*ViolationWeight

The lookup determines the color and weight of each violation type.

DEVFormatting the search string across multiple lines and aligning similar elements makes it much easier to read, understand, and maintain.

The second part of the query prepares the data for the visualization calculating the proportion of each color to display for each department along with the total value to display in the center of each donut:

<search id="policy_violations_color_summary" base="policy_violations_search">
        | stats sum(TotalViolationWeight) as TotalWeight,
                sum(ViolationCount)       as ViolationCount
          by department, ViolationColor
        | eval NumYellows=if(ViolationColor="Yellow", ViolationCount, 0)
        | eval NumReds=   if(ViolationColor="Red",    ViolationCount, 0)
        | stats sum(NumReds)     as NumReds,
                sum(NumYellows)  as NumYellows,
                sum(TotalWeight) as TotalWeight
          by department
        | table department, NumYellows, NumReds, TotalWeight

DEVThis search uses the table command to ensure that the query returns the fields in a specific order. The visualization that depends on this query makes assumptions about the field order.

For more information about how the donut control works and how it uses the data from this query, see the section Using a third-party visualization library (take two, creating the donut) in the chapter UI and visualizations: what the apps look like.

Example: Optimizing a search

The following code snippet shows the third search in the policy_violations_search discussed in the previous section: 

| search tag=pas tag=change tag=audit
    | fields _time, user
    | bucket _time span=1h
    | stats count as ops by _time, user
    | where ops!=0
    | stats stdev(ops) as SD, avg(ops) as MEAN, latest(ops) as LAST_OPS by user
    | where LAST_OPS > (MEAN + SD*1)
    | lookup employee_details user
    | stats count as ViolationCount by department
    | eval ViolationType="Excessive_Access"

This illustrates several useful optimizations:

  • We use the fields command to eliminate, as soon as possible, as many columns as we can from the search pipeline.
  • We use the fields command, and not the table command, to minimize the amount of data brought over the network in a distributed deployment.
  • The stats command reduces the number of events in the search pipeline.
  • The where ops!=0 removes any empty buckets from the pipeline.
  • When we begin our calculations using the stdev function, we have already reduced the amount of data we are working with as much as possible.
  • We have removed a number of eval operations from earlier iterations of the query design and inlined them.
ARCHYou can often gain insight into how Splunk Enterprise executes a search by using the job inspector. In the Splunk Web UI, visit the Jobs page from the Activity menu.

Loading custom JavaScript code

The PAS app uses JavaScript extensions to the Simple XML model for building dashboards and uses several different types of JavaScript resources. These range from relatively simple JavaScript code written by us and attached to a specific dashboard, to complex, third-party libraries and components that we have integrated with our code.

Attaching JavaScript to a Simple XML dashboard

Many of our dashboards in the PAS app use JavaScript extensions to add behavior and functionality. For example, the Summary dashboard uses JavaScript code in the summary.js file. The following code snippet from summary.xml shows how we attach the page specific JavaScript file to this dashboard using the script attribute of the form tag:

<form script="summary.js" stylesheet="summary.css,tagmanager.css">

The location of the summary.js file is in the appserver/static folder. We use a naming convention (summary.xml, summary.js, and summary.css) to make it easy to identify which resources in the appserver/static folder are associated with which Simple XML dashboards in the default/data/ui/views folder.

In our JavaScript extension code, we typically start with code like this:



], function(

DEVRequireJS is a popular JavaScript file and module loader, optimized for in-browser use. For more information, see

The purpose of this code is to ensure that the SplunkJS library is fully loaded before we execute any of our own code that may depend on the library being available. In some cases (for example in user_activity.js) the require function looks like this:




], function(


The second ready! Is to ensure that the code waits for all of the panels on the Simple XML dashboard to load. This is useful if you encounter JavaScript errors when you execute code to obtain a reference to a panel such as the following:

var zoomSearch = mvc.Components.get("zoom_search");

Integrating with a third-party JavaScript component 

We want to enhance the flexibility of the Summary screen by enabling a user to filter the data on the dashboard without any knowledge of the Splunk search syntax. For example, an investigator might want to exclude a particular heavily used document from the results because it is making it hard to see what's happening with other documents. To implement this feature, we first added a context menu to the Top Documents and Top Users panel as shown in the following screenshot:

After the user clicks on the Exclude option, the dashboard looks like this showing the filter as a tag in the Filter Criteria text box:

The Filter Criteria shows the details of the filter applied, and the search results no longer include the document. The dashboard now substitutes the filter token values ($filter$ and $exclude$) in the trend-search search manager:

pivot ri_pas_datamodel Root_Event count(Root_Event) AS count SPLITROW _time AS
  _time PERIOD auto SPLITCOL command FILTER command isNotNull $filter$ $exclude$

In this example, the dashboard now runs the following search to return the results:

pivot ri_pas_datamodel Root_Event count(Root_Event) AS count SPLITROW _time AS
  _time PERIOD auto SPLITCOL command FILTER command isNotNull FILTER object isNot "C:\\var\otb\docs\2770150694874303.dmgr"

In the JavaScript extension, summary.js, we initialize and define the context menus for the bar chart and the two tables as shown in the following code snippet:

context.init({preventDoubleContext: false});
context.attachToChart(barchart, menuData); 
context.attachToTable(user_table, menuData); 
context.attachToTable(document_table, menuData);

The context object is defined in the context.js file we added to the app in the static folder (the code in summary.js does load this file directly: summary.js uses the require function to load filter_component.js, which in turn uses the require function to load context.js). You can learn more about the open source Context.js script at

Loading the custom component

The extension summary.js also references two other JavaScript source files: filter_component.js and tagmanager.js. The filter_component.js file contains custom code written for the PAS app to manage the token values $filter$ and $exclude$ that appear in the search manager instances on the dashboard. The code in the filter_component.js file uses the Tags Manager jQuery plugin to display the filter values on the dashboard in the Filter Criteria text box. Tags Manager ( is an open source jQuery plugin for displaying a list of tags, letting a user edit the tag list, and that offers an API for managing the tag list programmatically.

These third-party JavaScript components make it easy to add custom behavior to the built-in visualizations. In this example, by adding a context menu to let an investigator filter the results in the screen without any knowledge of the Splunk Enterprise search syntax.

During this stage of the journey, we add a custom component to the Summary dashboard that provides filtering capabilities to the user. We use the following code to ensure the page loads the necessary JavaScript files and their dependencies:

    paths: {
        "tagmanager": "../apps/pas_ref_app/tagmanager",
    shim: {
        "tagmanager": {
            deps: ["jquery"]

], function(...

The require function loads our custom filter component (along with three other modules) that is defined in the file filter_component.js. Notice the path we need to use (../apps/pas_ref_app) to correctly load this file from the appserver/static folder. We have written our filter component module to be compatible with the require.js open source module loader. However, the third-party tag manager module (a jQuery plugin) is not compatible with require.js. To work around this, we use require.config to define a fake compatible module, named tag manager, that wraps the incompatible third-party module.

DEVThe require.config function is available as part of the SplunkJS Stack. It is part of the require.js open source module loader. You can find out more at

Working with tokens in a custom component

The file user_activity.js illustrates how our custom component can interact with the tokens on the dashboard. In a Simple XML dashboard there are two token models: the "default" token model substitutes token values into most visualizations on the page and the "submitted" token model substitutes token values into most search expressions on the page. For our app, we would like to ignore this distinction. If we set the token $trendTime.earliest$ to a value we want it to update everywhere, in searches and visualizations. Therefore, we define a fake token model called tokens that behaves like the regular default and submitted models, except that when we write to it, it updates both models. If we read from it, it reads from the default model. This enables us to simplify any code that just needs to update the tokens in both token models. The user_activity.js file includes the following façade definition to implement this behavior:

var defaultTokens = mvc.Components.get("default"); 
var submittedTokens = mvc.Components.get("submitted"); 
var tokens = {
    get: function(tokenName) {
        return defaultTokens.get(tokenName);
    set: function(tokenName, tokenValue) {
        defaultTokens.set(tokenName, tokenValue);
        submittedTokens.set(tokenName, tokenValue);
    on: function(eventName, callback) { 
        defaultTokens.on(eventName, callback); 

On the User Activity dashboard, we then need to propagate token values to ensure that all the visualizations display correctly:

    "trendTime.earliest": tokens.get("time.earliest"),
    "trendTime.latest": tokens.get("time.latest")
tokens.on("change:time.earliest change:time.latest", function(model, value) {
        "trendTime.earliest": tokens.get("time.earliest"),
        "trendTime.latest": tokens.get("time.latest")

Working around a bug in a library

The JavaScript extension code for the User Activity dashboard contains some slightly unusual require related code to work around a bug in the css! plugin that's bundled with Splunk Enterprise. Originally, this extension loaded the Calendar Heatmap control using the following code, using the standard path prefix ../apps/pas_ref_app:

], function(
) ...

However, this caused the css! plugin to get confused when you try to directly include a module name that contains "..", such as ../apps/pas_ref_app/components/calendarheatmap/calendarheatmap. Our workaround uses a path alias defined in require.config() so that the directly included module name, pas_ref_app/components/calendarheatmap/calendarheatmap, does not contain "..":

    paths: {
        "pas_ref_app": "../apps/pas_ref_app"
], function(
) ...

Making a JavaScript file "require compatible"

Originally, the filter_component.js file was a simple JavaScript file that contained several function definitions such as:

  • function generateFilterComponent(mvc) { ... } 
  • function filter(tokens, field_name, field_value) { ... } 
  • function exclude(tokens, field_name, field_value) { ... } 
  • function drilldown(tokens, base_search, field_name, field_value, earliest, latest) { ... } 

The summary.js file used the following code to load filter_component.js and its dependencies:

    paths: {
        "tagmanager": "../apps/pas_ref_app/tagmanager",
        "filter_component": "../apps/pas_ref_app/filter_component"
    shim: {
        "filter_component": {
            deps: [
        "tagmanager": {
            deps: ["jquery"]

], function( ...

In addition to the complexity of this code, it is not good practice to load functions into the JavaScript global namespace if you can avoid it. Therefore, we have rewritten filter_component.js to be "require compatible." The summary.js file now uses the following, much simpler, code to load filter_component.js and its dependencies:

    paths: {
        "tagmanager": "../apps/pas_ref_app/tagmanager",
    shim: {
        "tagmanager": {
            deps: ["jquery"]

], function( ...

For this to work, we made some major changes to the structure filter_component.js to convert it to a JavaScript module that require.js can load directly:

define(function(require, exports, module) {
    var mvc = require("splunkjs/mvc/mvc");
    var FilterComponent = {
        initialize: function() {
        _filter: function(tokens, field_name, field_value) {
        _exclude: function(tokens, field_name, field_value) {
        _drilldown: function(tokens, base_search, field_name, field_value, earliest, latest) {
    return FilterComponent;

We use this structure to create an Asynchronous Module Definition that defines a module that require.js can load directly. For more information, see the "WHY AMD?" page on the site.

Incorporating a third-party visualization

Incorporating a third-party visualization in a dashboard is similar to incorporating a custom component. The User Activity dashboard includes a calendar heatmap visualization from The resources for this visualization are in the appserver/static/components/calendarheatmap folder.

The panel on the dashboard is defined in the Simple XML file as shown in the following snippet:

<panel id="activity_levels_panel">
    <h3>Activity Levels</h3>
    <div id="activity_levels"></div>

In the JavaScript extension file user_activity.js, the require function loads the resources:

], function( ...

The following JavaScript code then renders the visualization:

new CalendarHeatMap({
    id: "activity_levels",
    managerid: "activity_levels_search",
    domain: "month",
    subDomain: "x_day",
    el: $("#activity_levels")

Because this visualization was originally designed to work with Splunk Enterprise, it is easy to incorporate: for example, the manager property refers to a Splunk search. However, we made a few changes to the visualization to customize its UI to our requirements by editing the calendarheatmap.js file.

Sharing code between dashboards

It is possible to attach the same JavaScript file (for example, setup_check.js) to every dashboard in the app by adding it to the list of scripts as follows in each Simple XML dashboard file:

<form script="setup_check.js,summary.js" stylesheet="summary.css,tagmanager.css">

In this case it's simpler to add the code to the dashboard.js file that is automatically attached to every dashboard. Therefore, our form elements in our Simple XML dashboard files can be a little simpler:

<form script="summary.js" stylesheet="summary.css,tagmanager.css">

However, this did introduce a problem in the PAS app. The code in the dashboard.js file checks if the user has completed the app setup process. If not it redirects the user to the setup dashboard. Unfortunately, this caused a loop because when the user visited the setup dashboard, the code determined that they had not completed the setup process and redirected them to the setup dashboard again. We solved this issue by adding the following code to the dashboard.js file to identify if the user is visiting the setup dashboard.

var isExemptFromSetupCheck = (window.location.href.indexOf('setup') !== -1);

if (!isExemptFromSetupCheck) {

Updated: Periodically reloading a visualization

NEW FOR PAS 1.5.2: This section has been changed since the original publication of Building Splunk Solutions, and applies to version 1.5.2 of the PAS reference app. For more information, see What's new in Building Splunk Solutions.

We refresh the Summary dashboard every five minutes to force the dashboard to re-run the searches on the page and display up-to-date data. Whereas in prior revisions of the PAS app, in summary.js we used the JavaScript window.setInterval() method to run the window.location.reload() method to refresh the dashboard every 300,000 milliseconds, for the current version of the PAS app, we realized that we could instead set the refresh rate for our dashboard using the built-in refresh=<seconds> attribute in summary.xml:

<form ... 

For more information, see Build a dashboard using advanced XML in the Splunk Enterprise Developing Views and Apps for Splunk Web manual.

ARCHWe chose to use a scheduled search instead of a Splunk Enterprise real-time search for two reasons. We don't need real-time data (refreshing every five minutes is good enough), and we don't want the performance overhead associated with a real-time search.

Using the Splunk JavaScript SDK to interrogate other apps

The User Activity dashboard includes a panel that displays information about user activity associated with different data sources (providers). The following screenshot shows 367 events from the Application provider and 1101 events from the File provider:

To display this information, the PAS app needs to dynamically discover which other apps are its data providers, and then add the activity count for each one to the panel. In the example shown in the screenshot, you can see information from the Application and File data provider apps. Our initial approach was to identify provider apps for the main PAS app by adding a special pas_provider.conf file to them. This mechanism is one way we can define a data API in Splunk Enterprise, the existence of the special .conf file advertises that this app can be used in a particular way by other Splunk apps. Other Splunk apps will make assumptions about the capabilities of this app because of the presence of the special .conf file. In the PAS app, the content of this file defines the title that the PAS app uses on the User Activity dashboard (File and Application in this example) and the name of the source type provided by the app:

title = File
sourcetype = ri:pas:file

The code in the provider_stats.js file performs this task using the Splunk JavaScript API inside the function loadProviderInformation. The following code snippets outline this process:

// Get access to Splunk objects via the JavaScript SDK
var service = mvc.createService();

// Look for apps that contain a pas_provider.conf file
var appsCollection = service.apps();

The next section of code fetches all the installed apps and iterates over the collection looking for those that contain a pas_provider.con file. The parallel logic in this code is hard to follow and we plan, if time permits, to reimplement it using a JavaScript async library:

appsCollection.fetch(function(err) {
    var providerTitleForSourcetype = {};
    var apps = appsCollection.list();
    var numAppsLeft = apps.length;
    _.each(apps, function(app) {
        if ( { 
            // Avoid querying information about disabled apps because 
            // it causes JS errors. 
        var configFileCollection = service.configurations({
            owner: "nobody",
            sharing: "app"
        configFileCollection.fetch(function(err) {
            if (err) {
            } else {
                var configFile = configFileCollection.item("pas_provider");
                if (!configFile) {
                    // Assume config file is missing, meaning that this app
                    // does not represent a PAS provider

Here it reads the contents of the pas_provider.con file and reads the title and sourcetype values from the provider stanza.

                } else {
                    configFile.fetch(function(err, config) {
                        var providerStanza = configFile.item("provider");
                        var stanzaContent =;
                        providerTitleForSourcetype[stanzaContent.sourcetype] = stanzaContent.title;

This last function checks if we have processed all the apps.

    function finishedCheckingApp() {
        if (numAppsLeft === 0) {

This approach lets us add new provider apps, and have our PAS app automatically recognize them and incorporate their data into the User Activity dashboard.

Issues with this approach

A security review of our app highlighted some concerns over the approach we implemented using the special .conf files to manifest that this app can be used in a particular way by other Splunk apps. The specific issue is that using JavaScript code to cross application boundaries is not a security best practice. Therefore, we had to investigate other solutions and choose an alternative approach to discovering our provider apps from the main PAS app. Furthermore, our original approach relied on the main PAS app running with administrator rights to be able to access other apps' data.

SECIt's possible for a security administrator to further lock down the behavior of an app. For example, by restricting an app to being allowed to only read specific indexes.

Our solution is to use the sourcetype in a search result to identify which provider add-on specific events come from:

var providerStatsSearch = mvc.Components.get("provider_stats_search");"results", {
    condition: function(manager, job) {
        return ( || {}).isDone;
}).on("data", function(resultsModel) {
    var rows =;
    if (rows.length === 0) {
        view.html("No data providers found.");
    } else {
        _.each(rows, function(row) {
            var sourcetype = row[0];
            var count = row[1];
                provider_title: sourcetype,
                event_count: count

This approach means that we can delete all the provider.con files from our provider add-on apps.

We also need to change the provider_stats_search in the user_activity.xml Simple XML file that retrieves all the events for the specified user from the pas index to format the data correctly:

<search id="provider_stats_search">
        tag=pas tag=change tag=audit user=$user|s$ | stats count by sourcetype 
        | replace "ri:pas:file" with "File" in sourcetype 
        | replace "ri:pas:database" with "Database" in sourcetype
        | replace "ri:pas:application" with "Application" in sourcetype
        | replace "google:drive:activity" with "Google Drive" in sourcetype
        | table sourcetype, count

Notice how we use the replace command to pretty-print the names of the sourcetypes on the dashboard.

ARCHThis approach to displaying the sourcetype names is not ideal because it hardcodes the original names. If we change a name or add a new provider app, we will need to edit this search.

Example: Adding a new provider add-on app

At a later stage in the journey, we add a new provider app to our sample. This new provider app pulls information about document access from a Google Drive account. We don't need to make any changes to the main PAS app in order to use it because:

  • The main PAS app automatically discovers the new Google Drive app because it is adding entries to the pas index. These events have a source type set to google:drive:activity.
  • The Google Drive app uses mappings to convert its data into the structure that our data model defines.
  • The Google Drive app defines a modular input to retrieve data from the Google Drive Activity Stream.

The following snippet shows how we map the data from Google Drive to our data model in the props.con file:

pulldown_type = 1

FIELDALIAS-command = event AS command
FIELDALIAS-object = doc_title AS object


EXTRACT-user = email=(?<user>\w+)

The googledrive_addon defines a modular input to enable Splunk Enterprise to read log data from Google Drive. We implement this in Python and use the Google Drive Python API to connect to Google Drive and access the log data from the Google Drive activity stream. We define the modular input in the file. There is some additional custom code in the file to support the OAuth authentication scheme that Google Drive uses. The following screenshot shows the modular input configuration page for the app in Splunk Enterprise where the user adds their OAuth credential information:

How we work #2: Pairing between stakeholders 

One of the stories we began to implement during this stage of our journey was anomaly detection, where the PAS app identifies and flags anomalous behavior on a dashboard such as an unusual number of accesses to the document repository by an employee. The implementation of this story began with an architect evaluating several approaches to detecting anomalous behavior. His decision was to try out a statistical approach as "the simplest thing that could possibly work" rather than trying to implement a sophisticated machine learning based solution. This statistical approach is based on a calculation for each employee of the mean and standard deviation (SD) of the historic daily number of document accesses in the repository, and then applies a multiplier to the SD to calculate a threshold value. If, on the most recent day in the period under analysis, the number of document accesses for the employee exceeds the threshold value for that particular employee, the PAS app flags an anomaly for further investigation in case the anomaly represents a security violation.

The architect next validated his approach with the business user and this raised a number of questions and issues related to this statistical method.

First, the business user was interested to know if the approach could indicate the degree of anomaly instead of just flagging it. The code snippet below shows a column definition TIMES_OVER_NORM that calculates a measure of this by comparing the variation with the mean. This enables the app to display anomalies using traffic light visualizations: highlighting large anomalies in red and less significant ones in orange. The business user suggested the following as an initial way to categorize the anomalies.

Size of anomaly Severity
> 2 * SD> High (red)High
1.5 * SD to 2 *SD Moderate (orange)Moderate
< 1.5 * SD Low (green)Low

Second, the business user pointed out that this statistical approach may not work well with certain patterns of access. For example, if the search calculates the mean and SD over the previous month with the data shown in the following chart, the spike at the end of July is detected as an anomaly:

If the search calculates the mean and SD over the previous three months, the spike at the end of September may not be detected as an anomaly:

Furthermore, if we expect a spike at the end the end of each month, then this statistical approach does not spot the anomaly that the spike is missing at the end of September:

The architect and business user agreed that for the PAS app, using the simple statistical approach is acceptable if the user can specify the time range over which the app calculates the mean and SD. However, in other scenarios a more sophisticated, pattern-aware approach such as machine learning may be necessary.

ARCHYou need to understand your data before you can use it effectively.

The architect tested his algorithm out with his own data set before bringing it to one of the developers to develop further in a pairing session. The first step for the developer was to understand the search and then modify it to work with the sample data we use with the PAS app. This revealed a problem with our sample data in that there is currently very little variation over time in the number of times each sample employee accesses the repository each day. We had to use a very low multiplier value before we saw any anomalies. Therefore, as a separate task, we need to modify our sample data generator to add more variation into the access patterns. Once the search was working with our sample data, the developer incorporated the search into a dashboard.

The architect then suggested that the search should be parameterized. As part of the application setup, a user should be able to specify the multiplier value for the threshold calculation and the time range over which the app calculates the mean and SD. The following code snippet shows the final search definition that includes the $multiplier$ token:

tag=pas tag=change tag=audit | bucket _time span=1d | stats count as ops by _time user |
  stats stdev(ops) as SD, avg(ops) as MEAN, sparkline  latest(ops) as LAST_OPS by user | eval sig_mult=SD*$multiplier$ |
  eval threshold = sig_mult+MEAN| table user MEAN SD threshold LAST_OPS sparkline| eval DELTA=LAST_OPS-threshold |
  eval TIMES_OVER_NORM = round(DELTA/MEAN,0) | eval ANOMALY= if(LAST_OPS>threshold,"true","false") |
  sort -ANOMALY

One important thing to notice about this search is that it does not currently use our data model, but queries the index directly. The issue here lies with the $multiplier$ token because it's not possible to perform calculations in a data model based on a token value or any other type of parameter. We will investigate further to see if there is any way we can use our data model and the PIVOT command in this search manager. The dashboard currently sets the time range for the calculation in code:

var timepicker = new TimeRangeView({
    id: "timepicker",
    managerid: "anomalous_manager",
    preset: "Last 24 hours",
    el: $("#timepicker")

We implement the final version of this dashboard in the PAS app using Simple XML with no JavaScript, and reformat the query to make it more legible:

    <label>Anomalous Activity</label>
    <fieldset autoRun="true" submitButton="false">
        <input type="time" searchWhenChanged="true">
                      tag=pas tag=change tag=audit
                    | bucket _time span=1d
                    | stats count as ops by _time user
                    | where ops!=0
                    | stats stdev(ops) as SD,
                            avg(ops) as MEAN,
                            latest(ops) as LAST_OPS by user
                    | eval sig_mult=SD*1
                    | eval threshold = sig_mult+MEAN
                    | table user MEAN SD threshold LAST_OPS sparkline
                    | eval DELTA=LAST_OPS-threshold
                    | eval TIMES_OVER_NORM = round(DELTA/MEAN,0)
                    | eval ANOMALY= if(LAST_OPS>threshold,"true","false")
                    | sort -ANOMALY

How we work #3: Exploratory programming

Splunk Enterprise 6.2 introduces a feature - the app key value store. We decided to do a spike using this new feature to see if we can use it to persist our application configuration settings from the setup screens.

Note: Splunk KV Store JS library (with Backbone classes) is released out-of-band. We've included it in our codebase under the pas_ref_app/appserver/static/components/kvstore_backbone folder.

We installed the KV Store Backbone app during a pairing session and discovered that the app includes a comprehensive JavaScript test suite that uses the Mocha unit test framework. This test suite proved invaluable in learning how to use the key value store from JavaScript. The following screenshot shows the test dashboard:

This page displays the JavaScript code that the test executes (in this example, it is retrieving an item previously uploaded to the KV store) making it easy to see how to complete a particular task. However, this page does not show all the code associated with the tests. To see how to upload an item to the KV store, we had to look at the source file that contains the test and locate the code that performs the setup for this "can be created" test in the model-tests.js file. The following code snippet shows the relevant function definition from this file:

// Insert one model before each test
beforeEach(function(done) {
  cleanCurrentUserCollectionAsync(function() {
    model = new TestModel({name: 'test'}, { namespace: { owner: currentUser } });
      .done(function(data, textStatus, jqXHR) {
      .fail(function(jqXHR, textStatus, errorThrown) {
DEVUnit tests are a great resource for helping you to learn how to use an API. Don't forget that you can help other developers understand your code base by proving your unit tests.

You can find a useful introduction to unit testing JavaScript code with Mocha in this blog post "Using Mocha JS, Chai JS and Sinon JS to Test your Frontend JavaScript Code".

Some tips and tricks

This section includes some useful tips, tricks, and recommended practices we use in our code.

This and that

Occasionally in our code you will see something similar to this snippet from the summary.js file that uses the variables this and that:

_start: function() {
    var that = this;
    // Perform initial rendering
    // Rerender whenever the page resizes
    window.addEventListener("resize", function() {
    }, false);

The trick of assigning this to that is very common in JavaScript. this is a built-in variable that refers to the current context or object, so in the _start function it refers to the object that this function is attached to. The addEventListener function has a different context, so this inside its function would refer to a different object than this in the _start function. In this code snippet we make sure that we use the object that this refers to in the _start function inside the function used by the addEventListener function.

Search escape

The user_activity.xml file in the PAS application includes the following search definition:

<search id="base_search">
        tag=pas tag=change tag=audit  user=$user|s$ | bucket span=5m _time | stats count by _time event_name event_target

In this search we append the search escape filter |s to the user token in case the user token contains any unusual characters that might mess up our search string.

What did we learn?

This section summarizes some of the key lessons learned while we were developing our JavaScript code and Splunk searches.

  • A data model can contain complex search expressions that perform lookups and calculations.
  • If you need to combine results from a pivot and a regular search, it's best to either convert everything to pivots or everything to regular searches.
  • We can import complex third-party JavaScript code and visualization libraries into our app.
  • It's a good idea to make any custom JavaScript components you create "require compatible."
  • You can share code between dashboards by placing it in the dashboard.js file.
  • You can use the JavaScript SDK to interrogate other apps.

More information

More information

Download the Splunk > Quick Reference Guide at:

For a primer into SPL and a collection of recipes, see "Exploring Splunk" at:

For more information about defining a lookup in a static file, see "Configure field lookups" at:

For information about using eval expressions, see:

For more information on the tsats command, see the "Splunk Search Command Reference" at:

For more information on RequireJS, see:

You can learn more about the open source Context.js script at

For more information on Tags Manager, an open source jQuery plugin for displaying a list of tags, letting a user edit the tag list, and that offers an API for managing the tag list programmatically, see:

For more information on the Asynchronous Module Definition, see "WHY AMD?" at:

For information about calendar heatmap visualization, see:

For information about the app key value store feature, see:

You can find a useful introduction to unit testing JavaScript code with Mocha in this blog post: "Using Mocha JS, Chai JS and Sinon JS to Test your Frontend JavaScript Code" at: