Parts of a Splunk Enterprise App

The fundamental purpose of a Splunk app is to get data into Splunk, extract the parts of the data that are meaningful to the app, and render the data in a form that facilitates easy interpretation of the data. This chapter gives you an overview of a Splunk app architecture, draws parallels to the more traditional web app architecture, and introduces the key Splunk technologies.

App architecture

The easiest way to introduce Splunk app architecture is to think about a typical 3-tier web application:

In the Splunk app, the front end corresponds to splunkweb and the back end corresponds to splunkd. The data tier is fused with splunkd. The diagram below depicts a typical Splunk app architecture.

The dotted lines link files in the Splunk app directory structure to the conceptual piece that they define.

On the traditional front end, you have HTML, CSS, and JavaScript to compose your web pages. Similarly, in splunkweb you have dashboards composed of HTML panels, CSS extensions, and JS extensions.


The SplunkJS stack includes a number of tools for web developers:

  • Libraries for Splunk Enterprise views and search managers for working with searches and interacting with Splunk data.
  • Backbone.js provides an MVC framework as a structure for your code.
  • RequireJS manages dependencies.
  • jQuery helps manage the HTML DOM.

ARCHFor apps that require more specialized capabilities not provided by splunkweb and Simple XML, developers can implement standalone HTML, CSS, and JavaScript together with the SplunkJS Stack. This means the app would run on a web server of your choice, not splunkweb. This gives you additional flexibility in terms of both implementation and deployment.

The back end is a thin layer on top of the Splunk Enterprise data engine responsible for ingesting and storing data. For now just treat it as a black box.

ARCHSome apps only introduce data ingestion facilities but no UI/front end.

In a Splunk app the various pieces comprising the app are represented as files in the file system that are installed on the Splunk server. For example:

  • A dashboard is stored in $SPLUNK_HOME/etc/apps/$APP_NAME/data/ui/views/$DASHBOARD_NAME.xml.
  • A JS extension is stored in $SPLUNK_HOME/etc/apps/$APP_NAME/appserver/static/$DASHBOARD_NAME.js.
  • modular input, which is script that pulls data from a data source (depicted as ACME Service on the diagram), is stored in $SPLUNK_HOME/etc/apps/$APP_NAME/bin/$

The Splunk data engine is configured by a large number of .conf files, each of which has different effects. See Appendix C for the details.

Splunk REST API and SDKs

The REST API provides a well-defined, programmatic interface to server resources, such as configuration files, processing states, and core Splunk Enterprise functionality. You can interact with a Splunk Enterprise instance and do most of the same things that you can using splunkweb -- including authentication, creating and running searches, managing search jobs, creating and managing indexes and inputs, and configuring Splunk Enterprise. 

DEVThe REST API is extremely flexible with over 170 endpoints. It's fully documented and supported. See the REST API Reference Manual:

As a convenience, Splunk SDKs provide you with broad coverage of the REST API in a language-specific fashion to ease your access to the Splunk engine. Currently, the following SDKs are available, for PythonJava, C#, and JavaScript.

DEVSplunk encourages and accepts community contributions to its SDKs. For more details see

These SDKs are mainly wrappers around the REST API that do a lot of the work for you, such as: 

  • Handling HTTP access.
  • Authenticating: When you log in using a username and password, Splunk returns a session key. The SDKs automatically remember and append this session key to subsequent requests.
  • Parsing the XML responses from REST API requests.<
  • Managing namespaces: A namespace is the user/app context for accessing a resource, which is specified by a Splunk username, a Splunk app (such as the default Search app), and a sharing mode. The SDKs send requests based on the namespace that was used for logging in, or you can specify a namespace to access a specific resource. For example, you can list all apps, or only the apps that a specific user has access to.
  • Simplifying access to REST endpoints: The SDKs provide access to the REST API in the native style of different programming languages.
  • Building the correct URL for an endpoint: The SDKs build out the complete REST URLs in the correct format, with the namespace and any additional parameters you specify.
  • Displaying simplified output for searches: The REST API returns search results (events) in XML, JSON, or CSV, but in a raw format. The SDKs provide results readers (helper classes for Python and Java, a code example for JavaScript) that parse these events and return them in a simplified structure with clear key-value pairs.
DEVKeep in mind, SDKs do not necessarily offer full feature-parity with the REST API.

SDKs also offer support for extensibility, for example, modular input support for Java, Python, C#, and JavaScript as well as custom search commands for Python.

App Directory Structure

All of the packaged components for a given app must reside in a directory under $SPLUNK_HOME/etc/apps (as shown in the diagram below).

Each app is further structured as follows:

App configuration

Splunk Enterprise uses configuration files to control almost all data processing. Appendix C provides a cheat sheet of various configuration files, including their meaning and usage.

Splunk Enterprise decides how to evaluate a configuration file and other artifacts (for example, dashboard XML files) based on the various contexts that the file operates within. As an app developer, you really only need to worry about the local vs. default distinction. The configuration in local is considered modified by the user, and the default is the one shipped with the app.

DEVWhen a user modifies a configuration setting in splunkweb (through the UI), it changes a copy of the configuration file in $APP_HOME/local. The value of the default attribute of the configuration file is left unchanged in $APP_HOME/default.

When you develop or modify an app using the Splunk Web interface (edit source), you are actually editing the local file. It's ok to do for a quick test. You should move the configuration to default to make it final. When developing in an IDE or standalone text editor, you should directly modify the configuration in default

DEVChanges to your dashboards (in local) through the web interface take effect immediately. Changes in default require you to hit the Splunk Refresh View endpoint. Some changes still require a Splunk Enterprise restart.

SHIPNever ship your app with files in the $APP_HOME/local directory. Otherwise, it could clobber configurations that a customer made to their copy of the app. Besides, during an app upgrade the default configuration will be overridden while the local is preserved.

Splunk extensibility surface

The extensibility surface area of Splunk Enterprise is vast. It includes the following types of knowledge objects grouped by different phases of Splunk processing pipeline: 

Concepts are italicized, artifacts are not.

When reproducing this diagram, instead of distinguishing italic vs non-italic, you may try using different colors. Also, to avoid retyping the terms, see the pptx file to copy-paste in the Essentials images on Box.

ARCHIn the design of your app, you might consider partitioning your app into parts that are likely to be reusable by other apps and parts that might be candidates for extension at some future date. Add-ons facilitate reuse by multiple apps. Reuse is considerably enhanced if your app is Common Information Model (CIM)-compliant and new add-ons map to the CIM definition.

The following topics summarize the areas that need to be considered by almost all Splunk Enterprise apps. Each of these are covered in more detail in the remainder of the Essentials chapters.

Get your data into your app

Before your app can do anything related to data, you first need to get the data into your app. This is the first topic we address because understanding the structure and content of your data is usually the first step in designing your app. Understanding the source, quantity, performance, fields, and overall structure of your data influences how you decide to acquire the data and how to construct your search commands in the next step. Practically, you might consider the search commands you'll need and even how the data maps to the user interface in parallel with developing your data acquisition strategy.

In the data onboarding topics, we discuss what kind of data is significant to the way Splunk Enterprise handles data, propose optimum data structures (where that option is available to you), recommend programming mechanisms and ways to make your data reusable, and how to simulate your data input during early app development phases.

Efficiently search your data

The goal of search is to find the exact information you need in what's likely a huge amount of data. It's often the proverbial "searching for a needle in a haystack." Splunk Enterprise provides sophisticated search capabilities for finding the needle, supported by a powerful search language, and it's the function of your app to render the data in a useful and meaningful way, usually as a visualization.

In the search topics, we propose a processing model that favors efficient searches and introduce you to fundamental search patterns that apply to most search problems. We go into some detail on things you can do with your data in transforming it from the raw format to meaningful information. More advanced topics present ways to make your searches more efficient. Finally, we suggest guidelines to help troubleshoot your search implementation.

Understand your data

The ultimate objective of your app is to provide meaningful information about your data set. This almost always involves rendering a transformation of the raw data to a visualization. If you look at the app architecture diagram above, this is the front end programming of your app, and Splunk Enterprise provides a number of web programming tools and utilities to facilitate data visualization. In Visualizing your data, we'll enumerate the various mechanisms available, like Simple XML, HTML, CSS, and JavaScript combinations, and illuminate the tradeoffs for different use cases.

In designing your app, this might be the place you start if you know a priori what information you want to display and have some flexibility in specifying the format of the incoming data. Usually, however, this is not the case and you'll need to do some level of concurrent design, considering what data are rendered along with the raw data format and search optimization.

Act on your data

Splunk Enterprise is always monitoring your data, and it gives you numerous different ways to observe trends and visualize your data at any time. But you don't have to monitor your data constantly to be able to identify when you need to act on it. With alerting, you can tell Splunk Enterprise when to inform the appro­priate stakeholders to take action, or even tell Splunk Enterprise to initiate those actions itself. In Acting on data, we describe the types of alerts available and how to manage them as well as how to create a custom alert action.

Package your application

Once your app is installed on a site, you'll most likely need the flexibility of being able to parameterize it for the environment and capabilities of a particular installation. These include tasks like setting remote IP addresses, user roles and credentials, and many other domain- and use case-specific properties.

In the packaging section and topics we describe the Splunk Enterprise mechanisms and conventions for setting up your app and how to package your app for distribution and installation on a Splunk Enterprise instance.