Planning a journey

We're planning a journey, and like any real journey we need to know who's coming with us, have an idea of where we want to get to, and what some of the choices along our way will be. We need to make some plans before we start. This journey is not a journey to a place, but the journey we took to learn how to build Splunk® apps in the real world.

This chapter introduces the companies that will collaborate on this journey, the people from those companies who will come along with us, and most importantly the Splunk apps that we intend to build as the goal of our journey. This chapter outlines some of the key features and functionality that we plan to include in the finished apps and introduces some of the issues we are considering right at the start of our journey. The chapters that follow enable you to follow along as we continue on this journey to build our Splunk apps.

A journey

This guide describes the journey undertaken by the project team and is intended to give you an insight into how to set about developing Splunk apps, to recommend some good and proven practices, and to provide a useful reference when you come to build your own Splunk apps. However, like on any real journey, we make mistakes, have arguments, and change our minds along the way; so in addition to showing you how best to do things, we highlight the pitfalls and issues that we encounter, and the solutions we find. We do not intend this guide to cover everything related to developing Splunk apps, it focuses on the specific issues we faced in building our two sample apps. However, our two sample apps are representative of the typical apps Splunk customers do build.

The primary objective of our journey is to deliver applied engineering guidance along with Splunk apps that act as reference implementations. We hope this guidance will help to lower the entry barrier, reduce the time to market, and decrease the risk for you, our Splunk customers and partners. The following five principles characterize our guidance philosophy:

  • Proven: based on real-world experience, and validated by field engineers and partners.
  • Authoritative: offers the best available advice, in context.
  • Accurate: technically validated and properly tested.
  • Actionable: provides the steps to success.
  • Relevant: addresses a real world problem.

Each chapter of this guide covers a specific theme such as UI design or data onboarding and each follows the basic chronology of the journey as we develop our apps. The chapters also include commentary from various experts, descriptions of some of the team's working practices where they shed light on what happened, and discussions about why we made particular decisions. We hope this guide will mature through a dialog with you, our audience of Splunk app developers.

Our organizations

Three different companies collaborate on this journey: Splunk, Conducive, and Auth0. Conducive already uses Splunk Enterprise and has previously developed Splunk apps; Auth0 is new to Splunk Enterprise. Both Conducive and Auth0 have ideas about how they can use Splunk apps to add value to their own products and services and want to partner with Splunk to pursue those plans further. This means that the journey includes two separate development projects for two separate Splunk apps: one for Auth0 to enable insights into how users are interacting with the Auth0 service, and one for Conducive to build a Splunk app to assist in monitoring secure document repositories (for more information, see Our apps below). We built both of these Splunk apps using Splunk simple XML. The next chapter, Platform and tools: a kitbag for our Journey, discusses why we chose this specific technology in more detail.


Conducive is a system integrator with one of the focus areas being standards compliance. Their user audience includes business managers, compliance officers, auditors and investigators.

Some of the specific goals that the Conducive team has for their involvement in the project are to learn how to:

  • Extend the Common Information Model.
  • Develop highly configurable advanced visualizations and user interactions.
  • Implement auto complete drop downs that can lazy load a large number of options.
  • Perform form validation prior to running a search.
  • Use pivots that enable a user to use drag and drop to manipulate the columns displayed on a dashboard.
  • Detect deviations from the norm across multiple variables.
  • Incorporate predictive and statistical analytics.
  • Handle outliers.
  • Design an app and dashboards that enable non-technical users to create reports and configure alerts.


Auth0 is an Identity-as-a-Service infrastructure provider. Their user audience is primarily developers.

Some of the specific goals Auth0 has for their involvement in this journey are to enhance their monitoring and reporting functionality with a Splunk app.

Their learning objectives include:

  • Understanding a structure of a Splunk app.
  • Using the Splunk'search processing language (SPL™).
  • Building and testing modular inputs.
  • Performing data validation when configuring a modular input.
  • Utilizing Simple XML to build a basic set of panels and dashboards and to refine them iteratively.
  • Packaging and publishing a Splunk app.

Our panel of experts

A panel of experts from a number of roles follow the journey, and the following chapters include their comments on the development effort. The roles these experts perform are:

DEV A developer, expert in using Splunk and developing Splunk apps. He wants to see a well written Splunk app that follows proven practices.
ARCH An architect who wants to see a well-architected solution.
UX A UX designer who wants to see an engaging, intuitive, easy-to-use UI for the app.
TEST A tester who wants to ensure the highest possible quality for the reference implementations.
ADM A system administrator who wants to know how to deploy and monitor the apps, and understand how to configure them.
BUS A representative of the business who knows how the apps will be used and their expected benefit to the business.
SHIP A release manager who wants to ensure that the apps are packaged in a way that makes them easy to deploy and to update in the future.
SEC A security expert who wants to ensure that the apps comply with any security requirements in the target environment.
PERF A performance tester who wants to ensure that the apps performance is acceptable with production loads.

Our apps

During the journey, the team develops two apps, one with Conducive and one with Auth0. This book focuses primarily on the Conducive app but discusses the Auth0 app where it provides a contrast or highlights different issues. The following two sections in this chapter provide overviews of the apps as we envision them at the start of the project.

ADMIf you're planning to list your app on the Splunkbase website, you should follow the naming guidelines at

Pluggable Auditing System (PAS)

The PAS Splunk reference app is intended to enable an organization to monitor a highly secure document repository such as one hosted on Conducive wants the organization to use the app to see who has looked at, updated, modified, deleted, or downloaded documents in the repository. The app should enable the organization to be both reactive and proactive in its approach so that in addition to being able to see what happened, the app notifies the organization of potential security issues when it detects unusual behavior associated with either a user or a document.

Potential users of the app include anyone with responsibility for protecting sensitive data. We expect the end users of the app to be non-technical business users such as:

  • An investigator assigned to protect data or to discover the identity of someone who is stealing data. An investigator is typically from the compliance or auditing side of the business.
  • A manager who needs to know what his or her employees are doing. This is not necessarily to monitor productivity but to identify unusual behavior.

BUSThese end user roles are typically non-technical, and they should be able to use the app without technical help: an intuitive UI is important.

In addition to the end users, there is a technical role for configuring the app with rules for triggering alerts specific to the organization and its requirements. Some organizations may configure these rules once when they deploy the app, others may have the requirement to be able to update these rules in response to changes in the business environment or to emerging threats. This may require some basic technical knowledge of Splunk Enterprise. The app should include a set of basic default rules to get started.

The app should include the following five key elements:

  • An Overview Dashboard to provide summary details of most reports and to highlight areas that require further investigation.
  • A Reactive Analysis capability to determine the activities of one or more users of the document repository and the geographic location where they performed those activities. The app should also be able to correlate those activities across different users, documents, or data sets.
  • A Rules-based Proactive Analysis capability to apply a set of predetermined and custom rules to the audit logs and reference data to search for potentially malicious activity. The organization can add custom rules as required to search for evolving and newly identified undesirable behaviors, or to filter out changes in behaviors that the organization no longer considers to be undesirable.
  • A Statistical Proactive Analysis capability to apply statistical analysis to the audit and reference data to establish various behavioral norms. This capability should include the functionality to apply weighting factors to the different data elements in order to differentiate false positives and noise from normal behavior.
  • A Workflow Solution with capabilities such as triggering incidents for review, categorizing them by type, assigning tasks to the relevant personnel, capturing details relevant to the investigation in an unalterable manner, and escalating incidents.

These were our goals at the start of the journey. The finished apps do not incorporate everything we envisaged, primarily because we ran out of time and resources needed to implement everything in our backlog.


At the start of the project, Auth0 is using Splunk Enterprise to monitor some basic events related to users of the Auth0 services. For example, whenever a new user signs up with Auth0, the Auth0 service sends event data to a Splunk REST endpoint. This event data includes information such as the event type, the target app, the client IP address, and the userid. Auth0 then display this information in a simple dashboard.

Auth0 would like to build a Splunk app that will replace the current solution and have the following features:

  • Identifying and tracking 'slipping away' users. Auth0 assign all users a status (new, active, very active, slipping away, all but deleted) and they send out a boilerplate email to slipping away customers asking why they are leaving. Auth0 would like to use rules in Splunk Enterprise to identify slipping away users.
  • Sending emails to all users of a specific connection type. For example, if Facebook make a change to one of their APIs, Auth0 wants to send a notification to all users of the Auth0 Facebook connector.

Auth0 are also considering making a Splunk app available to their customers to enable them to track business metrics such as:

  • If the customer has a portfolio of applications, who is accessing which ones?
  • Which users are encountering problems?
  • Suspicious log-ins and other anomalies.

ARCHYou check the details of your Splunk Enterprise license if you are planning to provide your customers with reports and data from your Splunk Enterprise installation.

BUSThe app should display this information in way that's attractive to customers.

Beginning our journey

The following chapters describe what happened on our journey, with each chapter focusing on a high level theme. Each chapter covers the issues the team encountered in tackling the various stories and use cases, how they resolved those issues, the decisions they made, and the lessons they learned. You'll also learn more about the specifics of the Splunk apps we have introduced in this chapter. Some chapters will also include sections that describe how the team works or arrives at particular decisions.

The following flowchart illustrates the typical workflow for developing a Splunk app. Not all projects will follow these steps exactly, but they provide a rough outline of the key stages for most Splunk app projects:

  1. To start, look at a sample of the real business data and begin to understand what information is available.

  2. Then you must work out how to get that data into Splunk. Options include, using file monitors, open ports, scripted inputs, or modular inputs. The chapter "Working with data: where it comes from and how we manage it" discusses these options for the PAS and Auth0 apps.

  3. During the development process use simulated data so you are not reliant on a third-party system while you are building the app. Typically, this helps you develop and test faster. The chapter "Platform and tools: a kitbag for our journey" discusses this.

  4. Determine the basic business scenarios your app will address by involving your business end users. Create some basic searches to support these scenarios. The chapter "Adding code: using JavaScript and Search Processing Language" describes an example of this.

  5. Use Splunk Enterprise features such as transforms to manipulate the incoming data to make it easier to search. The chapter "Working with data: where it comes from and how we manage it" discusses these options.

  6. Create a skeleton app using the Splunk Enterprise web UI or the command-line tools. You can then add dashboards and visualizations to this app. This is discussed in the chapter "Platform and tools: a kitbag for our journey."

  7. Run the basic searches and add them as panels to the dashboards in your skeleton app. This is discussed in the chapter "Platform and tools: a kitbag for our journey."

  8. Optimize the searches (see the chapter "Adding code: using JavaScript and Search Processing Language") and consider using a data model (see the chapter "Working with data: where it comes from and how we manage it").

  9. Enhance the visualizations and add polish to the UI. These are discussed in the chapters "UI and visualizations: what the apps look like" and "Adding code: using JavaScript and Search Processing Language"

  10. Prepare your app for deployment by adding features such as user roles and permissions. The chapter "Packaging and deployment: reaching our destination" discusses these options.

Our team has a variety of technical skills relevant to the project that include:

  • JavaScript, XML and knowledge of Model-View-Controller web development. These are all relevant to building our apps.
  • C# and Python. These are relevant to building our tests and connectors/modular inputs.

Some of the team also have experience of building Splunk apps using technologies such as Splunk SPL and the simple XML development model for apps.

Production notes

Note that code samples and screenshots shown in the following chapters do not necessarily match the code and UI in the completed apps. The samples show the code and UI at that particular point in the journey, and in some cases they may be modified at a later stage in our journey. In particular, some screenshots and code sample refer to the warum app; this was our original internal code name for the app that was changed near the end of the project to pas.

The majority of the screenshots in this guide were taken on a Windows machine, in some cases where there is a significant difference we show a *nix screenshot as well.