UI and visualizations: what the apps look like

In this chapter we focus on the UI of the apps and the processes we used to design and develop the various dashboards, forms, and visualizations that make up the PAS and Auth0 apps.

The primary skills our developers use when they work on the UI and visualizations are:

  • XML
  • HTML
  • CSS
  • JavaScript

Our developers have varying degrees of familiarity with the Splunk® Simple XML model. Simply re-using third-party JavaScript visualization libraries requires basic JavaScript skills; heavy customization of third-party JavaScript visualization libraries requires more in-depth JavaScript skills. Familiarity with the Splunk search processing language (SPL™) is useful in understanding how to format and manipulate data ready to display in a visualization.

A brief introduction to tokens in the Auth0 app

One of the refinements we make to the Auth0 app is to let a user of the app to filter the information on the Simple XML dashboard by tenant. We do this in a Simple XML dashboard by using tokens to pass values from one control to another. The first step in this example is to define a dropdown to display on the dashboard that lets a user select from the list of available data sources. The following screenshot shows this dropdown:

This code snippet from the logins_dashboard.xml file shows how we define this dropdown in Simple XML.

<fieldset submitButton="false" autoRun="true">
  <input type="dropdown" token="auth0_tenant" searchWhenChanged="true">
    <label>Data Input</label>
    <selectFirstChoice>true</selectFirstChoice>
    <populatingSearch fieldForLabel="source" fieldForValue="source">| metadata type=sources | search totalCount &gt; 0 | table source</populatingSearch>
    <default>auth0://contoso</default>
  </input>
</fieldset>

Notice how we save the user's choice to the token named auth0_tenant, how we use the searchWhenChanged attribute to trigger a search when the user selects a new option, and how we use the SelectFirstChoice element to define a default choice.

Later, in the same dashboard file we use the auth0_tenant token when we construct a search string for one of the charts we display. Notice how the token name is delimited using the "$" character in the search string:

<chart>
  <title>Most frequent users</title>
  <searchString>source=$auth0_tenant$ | top limit=10 user_name</searchString>
  ...
</chart>
DEVIn the Auth0 app, we spent more time fine-tuning our UI than we did developing the back-end code for the modular input discussed in the chapter "Working with data: where it comes from and how we manage it." Tweaking and refining the UI can be addictive!

An introduction to the PAS app

The first use case the team has decided to tackle in the PAS app is creating a dashboard to view all user activity over time, including any modification to a document, the database, or file system. This part of the system lets investigators respond reactively by examining the history of a user's interaction with the system the PAS app is monitoring. This dashboard forms a part of the overall reactive analysis function envisioned for the system that will give an investigator the ability to determine which activities one or more users may have performed from any geographic location, and correlate that activity across different users, documents, or data sets.

Walkthrough of an early version of the app

The following screenshots, from an early version of the app lets you see the functionality we implemented for this story. We will then describe some of the issues we encountered and the solutions we implemented. Remember that this shows an early version of the app, and that it does change in subsequent iterations.

The screenshot shows a standard dashboard with panels, controls, visualizations, and tables. The user can specify a time range to view in the Search Criteria panel and this determines the data that the chart shows. For each interval in the time range, the chart shows the number of operations of different types (such as create, delete, and read) that took place. A user can view detailed information by hovering the mouse over the bars in the chart, and highlight operations by hovering the mouse over the chart legend. The tables show the most active users and documents over the time range, and the user can filter this information by clicking on a bar on the chart or on the legend.

Clicking on a user name in the list of top users takes the user to the following dashboard to display detailed information about that user. Similarly, clicking a document in the list of top documents takes the user to another with detailed information about that document.

ARCHRemember the screenshots we show are from earlier versions of the PAS app to illustrate some of the issues we encountered as we started the journey and how we iterated on the implementation.
 

Issues discovered and proposed solutions

So far, this is a relatively simple Splunk app containing a set of dashboards that use standard Splunk Enterprise controls and visualizations, but we identified several interesting topics during the design and implementation of this story.

Using the Search element

Each of these dashboards displays three different sets of information: the chart and the two tables. We could define three separate search elements each with a complete standalone query, but this would mean the page would run three separate searches each returning the same data formatted slightly differently. The following snippets from the summary.xml file shows the solution we adopted to avoid this inefficiency:

<search id="base_search">
    <query>
        | pivot ri_pas_datamodel Root_Event count(Root_Event) AS Count SPLITROW _time AS _time PERIOD auto SPLITROW user AS user
 SPLITROW command AS command SPLITROW object AS object 
 FILTER command isNotNull $filter$ $exclude$ ROWSUMMARY 0 COLSUMMARY 0 NUMCOLS 0 SHOWOTHER 1
    </query>
</search>
...
<search id="top_users_search" base="base_search">
    <query>stats count by user | sort - count</query>
</search>
...
<search id="top_documents_search" base="base_search">
    <query>stats count by object | sort - count</query>
</search>
ARCHAt this point, do not worry about how the query is composed. We'll discuss the Splunk search language (SPL), data models and acceleration in the next chapter "Working with data."

This shows a search element called base-search that returns the data we need and two additional search elements that define post-process searches based on the data the base-search query returns. This means that the page only runs a single query, making it more efficient, while the post-process search elements manipulate the data into a suitable format and structure for the chart and the two tables to consume. Typically we don't recommend using post-process search instances for two reasons: first, they can only handle up to 10,000 events if they are filtering data, and second they do not provide any significant performance benefits in a distributed Splunk Enterprise deployment. However, in the scenario here, we are not using the post-process search instances to perform any additional filtering, they are just sorting and grouping the data from the base search instance. For more information about the search element, see the section "Search element" in Simple XML Reference.

DEVAnother benefit to using post-process searches is that they let you factor out the common prefixes of complex searches. This reduces duplication and makes the code more maintainable.
 

Navigation between dashboards

Clicking on a user name or document in the Summary dashboard does not perform the default drilldown action, but navigates to the User Details or Document Details dashboard and passes information about the selected user or document to enable the target dashboard to filter the data it displays. Although Simple XML enables drilldown functionality, we create some custom code to implement the feature on this page because we also want to include some custom filtering at the same location in the UI. At this point we are moving from using just Simple XML, to using Simple XML with JavaScript extensions to let us add this custom behavior (the chapter "Adding code: using JavaScript and Search Processing Language" includes more examples from the PAS app that show how to use JavaScript with Simple XML). The following code snippet from the summary.js file shows the click event handler for one of the tables on the dashboard:

document_table.on("click:row", function(e) {
     e.preventDefault();
     document_name = e.data["row.event_target"];
     earliest_time = tokens.get("earliest_time");
     latest_time = tokens.get("latest_time");
     window.location.href = "../document_details?document=" + document_name + "&earliest_time=" + earliest_time + "&latest_time=" + latest_time; 
});

Notice how this builds a URL with the appropriate parameter values from the tokens in order to pass the information to the Document Details dashboard. A much better approach is to use a standard JavaScript plugin to build the URL and this is something we implement later in the journey, replacing the code in the previous snippet with this approach using the jQuery plugin:

var queryParams = {
    "form.time.earliest": earliest,
    "form.time.latest": latest
};
queryParams["form." + field_name] = field_value;
window.open(page + "?" + $.param(queryParams), "_blank");

To read the parameter values on the target page we can use the jQuery $.deparam method. However, later in the journey we simplify the code that retrieves the querystring parameters and sets the token values on the page. The following code snippet comes from the user_activity.js file in the final version of the PAS app:

var submittedTokens = mvc.Components.get("submitted");

The get method of the mvc.Components class can retrieve the querystring parameters submitted to the dashboard by using the value submitted as a parameter. For more information about how we use this technique in the PAS app, see the section "Working with tokens in a custom component" in the chapter "Adding code: using JavaScript and Search Processing Language."

Outstanding issues

At this point, there are still some outstanding issues to address including:                

  • The chart type we used for the visualization is lacking some features.
  • We are still querying the raw data in the log files and have not abstracted the data in any way.
  • There are still some minor UI issues to address (such as the double backslashes you can see in the Document Details dashboard).
  • We need to validate that our decision to use post-process search instances is a valid approach for optimizing performance.

How we work #1: UI design prototyping

It's important that the UI design is appropriate for the compliance officers and investigators who will use the PAS app. A key part of the design will be to select visualizations for the dashboards that are easy to understand and intuitive to manipulate for these users. The requirements for the visualizations for the first use case include showing:

  • The overall volume of activity for specific users and documents over time.
  • The types and volumes of the different types of activity that were performed such as reads, writes, deletes, and downloads.
  • A time scale that enables the user to zoom in and out. The range might be several days or just a few minutes.

During our meetings with business subject matter experts (SMEs), we discussed a number of options such as line charts, swim lanes, and stacked bar charts.

We considered line charts that can show a composite view of all activities, combined with a timeline and overall activity volume levels. For example:

We considered swim lane charts that can show each activity separately but stacked, combined with a timeline and overall activity volume level.

We also considered stacked bar charts that let the user see the relative volume of each activity combined with the overall activity volume and a scalable timeline in a single view. Also important for these charts is the ability to toggle the display of each activity type to let the user increase the visibility of the most important activities.

UXWhen the specific events you are looking for are relatively few in number compared to the majority of events in your data, it can be hard to spot them. It's a good idea to give the user the ability to toggle the display of certain categories of events on or off. For example, if the vast majority of events are read events, but you are interested in the infrequent delete or update events, you want to able to toggle the display of the read events off.

For example, we considered the following stacked bar charts:

Our final choice was for a stacked bar chart with a scalable timeline below for zooming and panning because it provides the clearest way to see spikes in activity and makes it easy to identify suspicious or abnormal activity patterns. It also lets us use a single visualization to display the data and should therefore perform better than a solution that requires multiple visualizations to display the same information:

Options for implementing visualizations

If you are using Simple XML or the SplunkJS Stack you can chose from the many available built-in visualizations.

ARCHThe Splunk 6.x Dashboard Examples app has numerous examples of how to visualize your data on a Simple XML dashboard. You can download the app from http://dev.splunk.com/goto/dashboardexamples.

If you are using the Simple XML JavaScript extensions, you also have the option to integrate a third-party JavaScript visualization into your dashboard to expand your available choices. Not surprisingly, this is likely to involve more coding than using the standard Splunk Enterprise visualizations.

Refining the visualizations

During the initial stages of our journey, along with business subject matter experts at Conducive, we settled on using a stacked bar chart to enable a compliance officer to visualize the activities on a document repository over time. However when we began to implement this in the app, we discovered an issue with the way the visualization behaves when a user zooms in and out on the data. The following screenshot shows the visualization zoomed out to show the full range of sample data:

The narrowness of each individual bar makes it very hard to see the data, and the visualization does not aggregate the data into thicker vertical bars that summarize the activities over a wider time range. Note that the gaps in the chart are periods of time when there was no access to the document repository and therefore no log data. When a user zooms in by dragging with the mouse on the summary chart at the bottom to highlight a time range, the visualization looks like the following screenshot:

In this zoomed in version, a user can easily see the activities that took place every hour. We also implemented a small pop-up that can provide additional information when the user hovers the mouse over an area of the chart as shown in the following screenshot:

DEVFor this chart, the team is using the NVD3 chart library from NVD3.org instead of one of the built-in Splunk Enterprise chart types.
 

Using the third-party NVD3 chart visualization library

Attempting to use this third-party visualization in this dashboard raises a number of issues for us:

  • When we implement it in our dashboard we discover that it does not display zoomed out data clearly.
  • It requires complex JavaScript code (over 150 lines) to convert the data retrieved by a search into a format and structure that the visualization can use.
  • There is a steep learning curve for using this third-party visualization.

Resolving the visualization issue

Because of the issue with the way the NVD3 chart renders the vertical bars when it is zoomed out, we revisited the built-in visualizations to see if it was possible to implement the zooming feature without the need to use complex third-party libraries. We discovered some new features in the latest release of Splunk Enterprise (version 6.1) in the form of the Pan and Zoom Chart Controls. The only documentation available to us at the time described how to use these controls in a Simple XML app that does not make use of the JavaScript extensions. However, it is possible to work out how to customize them using JavaScript by studying the rendered JavaScript from a Simple XML application (by viewing the source in a web browser), and by using a JavaScript debugger to examine the arguments passed in to the event handlers.

The following screenshots show an early version of the User Activity dashboard with a fully zoomed out view of the data and a zoomed in view. The line chart at the bottom acts as a zoom control, enabling the user to highlight the time of interest and redraw the remainder of the screen based on that time range selection.

The following code snippet defines the stacked bar chart shown at the top in the screenshot:

<chart id="trend_chart">
    <title>Trend</title>
    <search>
        <query>index=pas sourcetype=ri:pas:application user=$user$ | bucket span=5m _time | stats count by _time event_name event_target | timechart sum(count) by event_name</query>
        <earliest>$trendTime.earliest$</earliest>
        <latest>$trendTime.latest$</latest>
    </search>
    <option name="charting.chart">column</option>
    <option name="charting.axisTitleX.visibility">collapsed</option>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="charting.drilldown">none</option>
</chart>

The following code snippet defines the zoom control chart shown at the bottom in the previous screenshot:

<chart id="zoom_chart">
  <search id="zoom_search">
    <query>index=pas sourcetype=ri:pas:application user=$user$ | bucket span=5m _time | stats count by _time event_name user | timechart sum(count)</query>
    <earliest>$time.earliest$</earliest>
    <latest>$time.latest$</latest>
  </search>
  <option name="charting.chart">line</option>
  <option name="charting.chart.nullValueMode">connect</option>
  <option name="charting.axisTitleX.visibility">collapsed</option>
  <option name="charting.axisTitleY.visibility">collapsed</option>
  <option name="charting.drilldown">none</option>
  <option name="height">100px</option>
</chart>

The JavaScript code in the following snippet shows how we update the range of data the upper trend chart displays in response to the user changing the selection in the lower zoom chart:

zoomChart.on("selection", function(e) {
    // Prevent the zoom chart from automatically zooming to the selection
    e.preventDefault();
    
    // Update trend chart's time range
    tokens.set({
        "trendTime.earliest": e.startValue,
        "trendTime.latest": e.endValue
    });
});

We are planning to continue using this approach for this particular visualization: the chart displays the information clearly, and it's much easier to implement than the NVD3 chart we looked at previously.

A simple example using the D3 third-party visualization library

The following screenshot of the Customer Monitor dashboard illustrates how we use the D3 dendrogram visualization to show which employees have accessed the records associated with a specific customer (in this example, Branden Morales). The dendrogram enables a user to drill down from the customer node, first to the departments, then to the divisions, and finally to the employees to see which employees have accessed documents relating to the customer. The panels at the top of the dashboard show summary figures: the total number of events that relate to the customer, the number of departments with employees who accessed the customer's documents, and the total number of employees who accessed the customer's documents.

Originally, we did not create any custom JavaScript specifically for this dashboard (customer_monitor.xml), but the form element did reference a utility script named autodiscover.js that enables the autodiscovery and instantiation of components developed using the Splunk Django framework (see "The recommended approach" below for more information about how to avoid using this script). The implementation of the dendrogram visualization relies on the code in two folders:

  • The D3 library elements are in the folder appserver/static/components/d3. You can learn more about D3 at http://d3js.org/.
  • The Splunk Enterprise specific wrapper code is in the folder appserver/static/components/dendrogram. We discuss this wrapper code in more detail below.
ARCHNotice how we place these script files in their own folders in appserver/static/components to indicate that this is library code rather than a script we created specifically for this app.
 

The customer_monitor.xml file includes two div elements that define the dendrogram visualization. The first specifies the search that retrieves the customer data to display:

Note: This section describes our first pass at adding this visualization and does not show the recommended approach. The section "Tidying Up" below shows the recommended approach.
<div id="dendrogram_search" class="splunk-manager" data-require="splunkjs/mvc/searchmanager" data-options='{
    "search": {
        "type": "token_safe",
        "value": "index=pas customer_name=$$customer$$| stats count by department department_group user "
    },
    "preview": true,
    "earliest_time": {
        "type": "token_safe",
        "value": "$$earliest$$"
    },
    "latest_time": {
        "type": "token_safe",
        "value": "$$latest$$"
    }
}'></div>
DEVThe class, data-require, and data-options indicate that we are working with a component originally developed using the Django framework and that we must add the autodiscover.js file to ensure that autodiscovery works correctly. See "The recommended approach" below for more information about the recommended way to avoid using these Django-related attributes.

The search returns data in the correct shape for the visualization. The columns are in the order that we drill down through the levels: department, division (department group), user. Notice how the token that holds the customer name uses $$ as a delimiter; this is because the search string is embedded within an HTML attribute value.

The second div element on the dashboard specifies how to render the dendrogram visualization:

<div id="dendrogram" class="splunk-view" data-require="app/pas_ref_app/components/dendrogram/dendrogram" data-options='{
    "managerid": "dendrogram_search",
    "root_label": {
        "type": "token_safe",
        "value":"$$customer$$"
    },
    "right": 600,
    "height": 600,
    "initial_open_lavel": 2,
    "node_outline_color": "#415e70",
    "node_close_color": "#b9d8eb"
}'></div>

This element references the dendrogram wrapper code and the search definition in the previous div element. It then sets various configuration options such as root_label and initial_open_level that control the appearance and behavior of the visualization.

The wrapper code

The wrapper code that enables us to use the D3 dendrogram visualization is in the folder appserver/static/components/dendrogram. It consists of a JavaScript file (dendrogram.js) and a CSS file (dendrogram.css), and these files illustrate several important points about how to import and use a third-party visualization.

We create dendrogram.js from scratch by starting with the following template code:

define(function(require, exports, module) {
    var _ = require("underscore");
    var d3 = require("../d3/d3");
    var SimpleSplunkView = require("splunkjs/mvc/simplesplunkview");
    require("css!./dendrogram_basic.css");
    var Dendrogram = SimpleSplunkView.extend({
        className: "splunk-toolkit-chord-chart",
        options: {
            "managerid": null,
            "data": "preview"
        },
        output_mode: "json",
        initialize: function() {
            SimpleSplunkView.prototype.initialize.apply(this, arguments);
        },
        createView: function() {
            return true;
        },
        // Making the data look how we want it to for updateView to do its job
        formatData: function(data) {
            return formatted_data; // this is passed into updateView as 'data'
        },
        updateView: function(viz, data) {
        }
    });
    return Dendrogram;
});

Next we add the D3 code that creates the visualization to the updateView method. For this example, we can take the sample D3 code from http://bl.ocks.org/mbostock/4063570 and remove the lines that load sample data from the flare dataset that we don't need because we are using data from a Splunk Enterprise search:

updateView: function(viz, data) {
    this.$el.html("");
    var node_outline_color = this.settings.get("node_outline_color");
    var node_close_color   = this.settings.get("node_close_color");
    var node_open_color    = this.settings.get("node_open_color");
    var width  = this.$el.width();
    var height = this.settings.get("height_px");
    var m = [20, this.settings.get("margin_right"), 20, this.settings.get("margin_left")],
        w = width - m[1] - m[3],
        h = height - m[0] - m[2],
        i = 0;
    var tree = d3.layout.tree()
        .size([h, w]);
    var diagonal = d3.svg.diagonal()
        .projection(function(d) { return [d.y, d.x]; });
    var vis = d3.select(this.el).append("svg:svg")
        .attr("width", w + m[1] + m[3])
        .attr("height", h + m[0] + m[2])
    .append("svg:g")
        .attr("transform", "translate(" + m[3] + "," + m[0] + ")");
}
DEVThe most difficult part of using a third-party visualization is to get the data from a Splunk Enterprise search into the correct format.
 

Next we need to format the search results in the shape the D3 control expects (often as a JSON object). In this example, we learned about the format of data expected by the visualization by looking at the sample D3 code from http://bl.ocks.org/mbostock/4063570. The following snippet shows the code in the formatData function that formats the results of the search:

formatData: function(data) {
    var height    = this.settings.get("height");
    var height_px = this.settings.get("height_px");
    this.settings.set("height_px", height === "auto" ? Math.max(data.length*30, height_px) : height);
    var nest = function(list) {
        var groups = _(list).groupBy(0);
        return _(groups).map(function(value, key) {
            var children = _(value)
                .chain()
                .map(function(v) {
                    return _(v).rest();
                })
                .compact()
                .value();
            return children.length == 1 && children[0].length === 0 ? {"name": key} : {"name": key, "children": nest(children)};
        });
    };
    return {
        "name": this.settings.get("root_label"),
        "children": nest(data)
    };
},

This function creates a hierarchical dataset that models the hierarchy of data that the visualization displays.

You can learn more about this wrapper code at dev.splunk.com/goto/dendrowrap, and by viewing the recording of the Splunk 2014 .conf session "I Want That Cool Viz in Splunk!."

You can also view another complete example that shows how to use the D3 Bubble Chart if you download and install the Splunk 6.x Dashboard Examples app.

The recommended approach

Our first-pass at implementing this visualization relied on the autodiscover.js file and some parts of the Django development framework. The approach shown here does require a small amount of custom JavaScript code but has fewer dependencies.

To make these changes, we remove the two div elements from the customer_monitor.xml file, we remove the autodiscover.js file, and we no longer have the confusing $$ delimiters in the customer_monitor.xml file. In the final version of this dashboard, the Simple XML file contains the search and a placeholder div element as shown in the following snippet:

<search id="dendrogram_search">
    <query>
          `pas_index` customer_name=$customer$
        | stats count by department department_group user
    </query>
</search>
...
<row>
    <panel>
        <html>
            <h2>Breakdown by departments ...</h2>
            
            <div id="dendrogram"></div>
        </html>
    </panel>
</row>
DEVWe have left the search command in the Simple XML rather than moving it to JavaScript because it's easier to maintain there.
 

We now have the following code in the customer_monitor.js file to render the visualization:

require.config({
    paths: {
        "pas_ref_app": "../app/pas_ref_app"
    }
});
require([
    "splunkjs/ready!",
    "splunkjs/mvc/simplexml/ready!",
    "jquery",
    "pas_ref_app/components/dendrogram/dendrogram"
], function(
    mvc,
    ignored,
    $,
    DendrogramView
) {
    new DendrogramView({
        "managerid": "dendrogram_search",
        "root_label": mvc.tokenSafe("$customer$"),
        "right": 600,
        "height": 600,
        "initial_open_lavel": 2,
        "node_outline_color": "#415e70",
        "node_close_color": "#b9d8eb",
        "el": $("#dendrogram")
    }).render();
});

This code identifies the search and various options to configure the visualization. For more information about the require.config function, see the chapter "Adding code: using JavaScript and Search Processing Language."

Enhancing the visualization

In the final version of the app, we have modified the code in the dendrogam.js file to add count values to each node in the dendrogram visualization to provide more information to the user. All of these changes are in the updateView and formatData functions. The following screenshot shows the final version of the visualization:

A complex example using the D3 third-party visualization library

The previous section describes how we created a dendrogram visualization using the D3 library where we use a small amount of custom JavaScript code to insert the visualization on the page and the visualization is provided as a reusable control. At a later stage in the development of the PAS app we add the following Policy Violations panel to the Summary dashboard and construct the donut visualizations using the D3.js library (http://d3js.org/). That requires considerable more custom JavaScript because we are extensively modifying the basic visualization provided by the D3 library. Our approach is based on some pre-existing sample code using the D3 library to draw a donut that we then modified to meet our own specific requirements. We did consider using the HTML canvass element, but opted instead for an SVG vector graphics based approach over a raster-based approach. The advantages of the vector-based approach are that the graphics scale smoothly without pixilation as the user zooms in the browser, and our UI designers can use CSS to apply styles to the graphics elements independently of the code.

The following screenshot shows our donut visualization:

ARCHYou can mix and match different technologies for rendering views within the same dashboard. Different panels on the Summary dashboard use either built-in Simple XML controls or custom HTML.
 

We derive this visualization from a pie-chart sample found on the D3 web site and we add some automatic scaling to the panel and individual controls. The entire panel is data driven, so we pass as parameters to the panel the number of donuts, the labels, and the values to display. For the image in the previous screenshot, the panel uses the following data, passed as a JavaScript array, to build the display:

centerText data titleText
0 color:gray, size:1 Development
4 color:red, size:1 Human Resources
3 color:orange, size:2 Marketing
13 color:red, size:2
color:orange, size:1
Research

ARCHBehind the scenes, we retrieve the list of departments to display from data the user provides on the Setup dashboard and that is saved in the KV store.
 

In the summary.js file, the following JavaScript code passes this data to the view:

donutSeriesView.setData(donutSeriesData);

In summary.js, we define a class named DonutSeriesView to encapsulate the logic for the view. We instantiate the donutSeriesView instance in the previous code snippet as follows:

var donutSeriesView = new DonutSeriesView(
    d3.select(".donut_series"),
    200);

This code selects the div element on the page and sets its maximum height to be 200 pixels.

The _renderDonutSeriesPanel method in the DonutSeriesView class is responsible for rendering the view. It first calculates some dimensions based on the size of the view and the number of donuts, before adding each individual donut to the view as shown in the following code snippet:

_.each(data, function(donutData) {
    var donutContainer = container.append("span").attr("class", "donut");
    
    var donutSize = widthPerDonut - 2*DONUT_SPACING;
    this._renderDonutChart(
        donutData.data,
        donutData.titleText,
        donutData.centerText,
        donutData.lowerText,
        donutContainer,
        donutSize,
        30/170 * widthPerDonut);
    
    donutContainer.node().style.marginLeft = DONUT_SPACING + "px";
    donutContainer.node().style.marginRight = DONUT_SPACING + "px";
}, this);

This code uses the each function from the Underscore.js library to iterate over the departments in our data and render each individual donut by calling the _renderDonutChart method.

Finally, the _renderDonutChart method uses the D3.js library to draw each donut. The method first defines an arc and a pie chart using the D3.js library as follows:

var TITLE_AREA_HEIGHT = 30;
var TITLE_TEXT_PX_HEIGHT = 14;
var radius = size / 2;
var arc = d3.svg.arc()
    .outerRadius(radius)
    .innerRadius(radius - thickness);
var pie = d3.layout.pie()
    .sort(null)
    .value(function(d) { return d.size; });

Next it defines an SVG container element to hold the SVG code that defines the donut:

var svg = container.append("svg")
    .attr("width", size)
    .attr("height", size + TITLE_AREA_HEIGHT)
    .append("g")
    .attr("transform", "translate(" + size/2 + "," + (size/2 + TITLE_AREA_HEIGHT) + ")");

Next it adds the text elements to the SVG to define the title and center text:

svg.append("text")
    .attr("class", "titleText")
    .attr("dy", (-(size / 2) - (TITLE_AREA_HEIGHT - TITLE_TEXT_PX_HEIGHT)/2) + "px")
    .style("text-anchor", "middle")
    .style("font-size", TITLE_TEXT_PX_HEIGHT + "px")
    .text(titleText);
svg.append("text")
    .attr("class", "centerText")
    .attr("dy", ".35em")
    .style("text-anchor", "middle")
    .style("font-size", size * 0.20 + "px")
    .text(centerText);

Finally, it draws the arcs that make up the donut, with one arc for each color:

svg.selectAll(".arc")
    .data(pie(data))
  .enter().append("g")
    .attr("class", "arc")
    .append("path")
    .attr("d", arc)
    .style("fill", function(d) { return d.data.color; });
DEVNotice how the line starting with .enter() is unindented half a level. This is a D3 specific convention for indicating when the current node being acted upon changes during a sequence of calls.
 

In the final version of the PAS app we extract the code relating to the donut visualization into the files policy_violations.js and policy_violations.css. This makes it easier to maintain because all the code related to the custom panel is located in these two files. You can see how we include this code in the summary.xml file:

<form script="summary.js,policy_violations.js"
      stylesheet="summary.css,policy_violations.css,bootstrap-tagsinput.css">

While we were designing our donut visualization, we looked at the following resources for guidance and inspiration:

Using dummy data during visualization development

We developed the donut control visualization before we developed the search that retrieves the data for the control. To manage this, we created a dummy data source that returns sample data in the same shape that we anticipated the real search would return. The following code snippet from the summary.js file shows how we retrieve the sample data from a CSV file and pass it to the donut visualization:

var dataSearch = new SearchManager({
    search: '| inputlookup example_violation_data.csv | lookup violation_info ViolationType | 
      eval isYellow=if(ViolationColor="Yellow",1,0) | eval isRed=if(ViolationColor="Red",1,0) | 
      stats sum(isYellow) as NumYellows, sum(isRed) as NumReds, sum(ViolationWeight) as TotalWeight by Department | 
      table Department, NumYellows, NumReds, TotalWeight'
});
dataSearch.data("results").on("data", function(resultsModel) {
    var rows = resultsModel.data().rows;
    
    // From the search results, compute what data the donut series
    // chart should display.
    var donutSeriesData = [];
    _.each(rows, function(row) {
        ...
        
        donutSeriesData.push({
            data: colorData,
            titleText: department,
            centerText: (totalWeight == null) ? "0" : totalWeight,
            // TODO: Compute % difference from last period
            lowerText: ""
        });
    });
    
    // Display the donut series chart
    donutSeriesView.setData(donutSeriesData);
DEVNotice the "|" symbol at the start of the search string. It denotes a generating search command, which has no input. Such command is not searching over data from Splunk indexes. Instead, it is generating its own data. You need this to ensure that Splunk Enterprise executes the inputlookup command and does not do a keyword search for inputlookup!
DEVWhen search is launched during onclick events, it may be useful to be able to cancel the search as follows: dataSearch.cancel()
DEVIf you generate a CSV file such as the one we use here (example_violation_data.csv) using Excel on a Mac, you must change the line endings from (CR) to (LF) so that Splunk Enterprise will recognize them correctly.

Later, when we developed the search that returns the real data, we replaced search in our code. see the section "Example: Combining multiple searches" in chapter "Adding code: using JavaScript and Search Processing Language. "

Adding controls in Simple XML manually

When we are editing the Simple XML directly and not using the graphical editing tools in Splunk Web, we place each control in its own div element to make it easy to reference the control from JavaScript. The following example comes from the setup.xml file:

<html>
  <form>
    <input id="_key" type="hidden"></input>
    <div id="departments">
        <h3>Departments to show on Summary dashboard: </h3>
        <div id="departments_dropdown"></div>
        <p>
            <em>If the dropdown above has no choices then no events have been generated yet.</em>
        </p>
    </div>
    <hr/>
    <div id="violation_types">
        <h3>Violation Types:</h3>
    </div>
    <hr/>
    <div id="locations">
        <h3>Locations:</h3>
        <input type="checkbox"/> Enabled<br/>
    </div>
    <hr/>
    <input id="save" type="button" class="btn btn-primary" value="Save"/>
  </form>
</html>

The following JavaScript code populates the content of the departments_dropdown div element:

var departmentsDropdown = new MultiDropdownView({
    managerid: "departments_search",
    labelField: "department",
    valueField: "department",
    el: $("#departments_dropdown")
}).render();

Note that we import the MultiDropdownView class from splunkjs/mvc/multidropdownview.

DEVYou can still use the graphical editing tools in Splunk Enterprise after you have made changes to the source manually.
 

Using custom CSS with Simple XML

We use the CSS extension feature of Simple XML to apply custom layout and styling to our dashboards. For example, in the user_activity.xml file we attach a user_activity.css file as shown in the following code snippet:

<form script="user_activity.js" stylesheet="user_activity.css">

This CSS file contains the following definitions:

#activity_levels_panel { 
    width: 33% !important; 
}
#trend_panel { 
    width: 67% !important;
}

The Simple XML file references these definitions in some of its panel elements. For example:

<panel id="activity_levels_panel">
...
<panel id="trend_panel">

Notice how our naming convention makes it easy to identify all resources associated with a single dashboard: user_activity.xml, user_activity.js, and user_activity.css.

If you want to have a CSS (or JavaScript) file that's shared by multiple pages, you can add it to your Simple XML dashboard along with the page specific resources using the following notation:

<form script="user_activity.js,utility.js" stylesheet="user_activity.css,branding.js">

You can add as many resources as you need by using these comma separated lists for the script and stylesheet attributes.

If you want to automatically attach a JavaScript or CSS resource to every Simple XML page in your app, you can use the special files dashboard.js and dashboard.css. For example, you might want to check whether the app has been configured properly and automatically redirect to a setup screen if it is not or, you implement custom app licensing code that checks to see whether the app is licensed before letting you use the app .You don't need to add these files to the script and stylesheet attributes.

DEVAdding such global checks to dashboard.js and dashboard.css is less error prone because you won't forget to add them when you create new dashboards.
 

In the PAS app we use the dashboard.css file to hide the editing buttons that normally show on all our dashboards by using the following CSS code:

.splunk-dashboard-controls .edit-btn { display: none; }
.splunk-dashboard-controls .more-info-btn { display: none; }
DEVIn practice, using dashboard.js is far more common than using dashboard.css.
 
 

Customizing the heatmap visualization with CSS

An interesting use of CSS in the PAS app is where we hide the display of the next month on the standard heatmap control on the User Activity dashboard. This control normally displays the current and next month, but because we are displaying historical data there is never any data to display for the next month. To hide the next month, we simply set a fixed width for the panel that contains the heatmap control so there is only space for the current month. The following snippet from user_activity.css shows how we do this:

#activity_levels_panel .panel-element-row {
    width: 250px;
    margin-left: auto; margin-right: auto;
}

Managing screen real estate with CSS

We also use CSS to modify the default dashboard layout to maximize the available screen real estate on some of our dashboards. The following screenshot shows the User Activity dashboard before we made the changes. As you can see, there is a lot of unused space next to the User Activity dashboard title text:

We use CSS to adjust the layout as shown in the following screenshot:

To achieve this result, our CSS moves the input fields up next to the title and shifts each row of the dashboard up to fill the empty space. The following snippet shows the relevant CSS code from the file user_activity.css:

.splunk-timerange {
    float: right;
    position: relative;
    left: 150px;
    top:-55px;"
 }
#input1 {
    position: relative;
    left: 250px;
    top:-55px;"
}
#input1 label {
    position:relative; 
    left: -50pt; 
    top: 19pt;
}
.dashboard-row {
    position: relative;
    top:-56px;
 }

We have another example of using CSS to reclaim some space on the User Activity dashboard in the Trend panel. The following screenshot shows the layout before we make the changes:

The next screenshot shows the results of our CSS changes where we hide the footer, the resize handles, and refresh time:

The following code snippet shows the relevant CSS code. Notice how we use a wildcard to identify the refresh time indicator that has a generated id:

div .ui-resizable-handle {
   display:none !important;
}

div .panel-footer {
 display:none !important;
}

div[id*='zoom_chart-refreshtime']{
 display:none !important;
}

Using a custom table cell renderer in Simple XML

On the Summary dashboard, in the Suspicious Activity (Since Midnight) panel we use the CustomCellRenderer class to add the colored dots in the first column:

The following code snippet from the suspicious_activity.js file shows how we render custom content in the cell in the table view:

var CustomIconCellRenderer = TableView.BaseCellRenderer.extend({
    canRender: function(cell) {
        return cell.field === 'C';  // the color column
    },
    
    render: function($td, cell) {
        $td.html('<span style="font-size: 3em; color: ' + cell.value + '">&#x25CF;</span>');
    }
});

Notice how this code uses the TableView.BaseCellRenderer class. In Simple XML, it is not straight forward to get access to a table view and it is necessary to access the view through the table element. The following code shows how we do this and then pass the view our custom cell renderer:

var tableElement = mvc.Components.getInstance('suspicious_activity_table');
tableElement.getVisualization(function(tableView) {
    tableView.table.addCellRenderer(new CustomIconCellRenderer());
    tableView.table.render();
});

For more information about custom cell renderers, see "How to create a custom table row renderer using SplunkJS Stack."

Adding colors and logos

As a final set of changes to the PAS app at this stage in our journey we customize its appearance with a logo and some different colors. On the home page in Splunk Enterprise, the title bar for the PAS app in the list of installed apps now has a custom color:

We make this change by editing the default\data\ui\nav\default.xml file and adding the color attribute to the nav element:

<nav color="205365">
    <view name="summary" default="true"/>
    <view name="user_activity"/>
    <collection label="Suspicious Document Access">
        <view name="offhours_document_access"/>
        <view name="terminated_employee_document_access"/>
        <view name="anomalous_activity"/>
    </collection>
    <view name="setup"/>
</nav>

We use the default.xml file to identify the Summary dashboard as the default dashboard for the PAS app.

We also add a logo to the top right of each dashboard:

To add the logo, we add a static\appLogo.png file to the app.

How much detail should we display?

During the iterations on the design of the User Activity Dashboard for the PAS app, we received conflicting advice from the two subject matter experts (SMEs) we consulted over the level of detail to display on the form. One SME was in favor of including a panel that displays a raw feed of the log data, arguing that an auditor or compliance officer will want to get right into the data as soon as possible. The other SME argued that a raw feed was too detailed at this level and that a user would want some higher-level tools on the dashboard, but with the ability to drilldown when the need arises.

UXIt's common to encounter conflicting points of view when you are designing your dashboards. You need to get feedback from real end users about the usefulness of the alternative approaches.
 

What did we learn?

This section summarizes some of the key lessons learned when we were building the UIs for our apps.

  • Using post-process search instances on a dashboard can reduce the number of searches you need to perform and factor out common prefixes from complex searches in different places on the dashboard.
  • To pass parameters between dashboards, you can implement your own mechanism using the request query string if you need to integrate it with other custom features. Otherwise, you can use the built-in drilldown capabilities and automatic population of token values from URL parameters provided by Simple XML.
  • There is a two-fold risk associated with using third-party visualizations. The first element of risk is simply that it may take time to learn how to use the library effectively and be able to customize it to meet your requirements, many sophisticated visualizations are complex to use. The second element of risk is that third-party visualizations may require you to perform additional manipulations on your search results to get them into a suitable format for use with the visualization. For the next stories we tackle, we will focus first on an end-to-end solution using the built-in Splunk Enterprise visualizations. After we have the end-to-end solution working, we will then explore the options for beautifying our app by using suitable third-party visualizations.
  • Our experience of trying to use the D3 visualization with our app illustrates how you can easily prototype ideas in Splunk Enterprise. In this case by trying to use a third-party visualization, we determined that it did not meet our requirements, so we looked for other another solution. This did not significantly affect the overall design of our app, our data model, or our searches.
  • With minimal overhead, you can reuse visualization components from diverse libraries such as D3. The main effort would be in formatting the search results to meet the input requirements of the visualization component.
  • Customizing dashboard layouts can be challenging due the fact that Splunk Enterprise injects some CSS when it renders the page.