Packaging and deployment: reaching our destination

During this stage of our journey, we explore the options for packaging and deploying the PAS apps. We investigate how we can package the multiple apps that make up our solution in a way that makes it easy for a customer to deploy them in their own environment, and how to facilitate updates to the apps after the customer has deployed them.

Packaging and deploying the PAS app

The following screenshots show the result of the deployment process for a user in Splunk® Enterprise. The first screenshot shows the home page in Splunk Enterprise with the main Splunk Reference App - PAS in the list of installed apps. The second screenshot shows the details of all the installed apps, which includes the main Splunk Reference App and the various add-ons (Splunk Reference App - HR info, Splunk Reference App - Application Add-on, Splunk Reference App - Database Add-on, Splunk Reference App - Files Add-on, and Google Drive Data Provider Add-on). Notice how the add-ons are invisible and that not all the PAS apps share the same version number.


The remainder of this chapter describes how we packaged the apps to achieve this result.


The most complex part of packaging the apps was deciding how to handle the multiple Splunk apps that make up the PAS solution. Typically, a Splunk app is packaged into an .spl file that a customer can either download and install manuallyor discover on the Splunkbase website. From Splunkbase, Splunk Enterprise downloads and installs it automatically. Unfortunately, a single .spl file only enables installing a single Splunk app, and the PAS solution is made up of multiple Splunk apps.

SHIPAn .spl file is a zipped tar archive that contains all the required files for the app.

We considered the following options:

  • Creating an installer inside a Splunk app
    Taking this approach, we could package all the dependencies inside the main app and include logic that installs them as and when they are required. The Splunk App for VMware plans to use this approach in the future.
    A variation on this approach is to have a dedicated installer app that sets up the main app and all of its dependencies, and then redirects requests to the main app it installed. The Splunk App for Enterprise Security uses this approach.
  • Using an external installer
    Using this approach, we could create a script that automatically installs an app and its dependencies. Alternatively, we could use a third-party installer tool to build the installation scripts.
  • Ask the customer to install the app and its dependencies manually from a single ZIP file
    Taking this approach, we could package the main app and all the add-on apps in a single ZIP archive. We would then ask the customer to unpack the archive into the etc/apps/ folder. The Splunk App for VMware currently uses this approach.
  • Ask the customer to install the main app and each dependent add-on individually
    Taking this approach, we would add detailed installation instructions to the documentation explaining how the customer should obtain and install each of the apps individually. Optionally, we could add logic to the main app that detects missing dependencies and then prompts the customer to install them. The Splunk App for Microsoft Exchange uses this approach.

The following chart plots each of these options showing how ease of implementation maps onto ease of use for the customer.

While it may be possible to check programmatically for missing dependencies in the same Splunk Enterprise instance, it is more difficult to check for missing dependencies when you expect that dependency to be installed on another machine such as a Splunk Enterprise forwarder. In this scenario, you can programmatically run some searches in your main app that look for data that the dependency provides. If you don't find the data, the main app can then prompt the customer to install and configure the missing dependency. The Splunk App for Microsoft Exchange uses this approach.

Overview of the solution

The approach we decided on for the PAS app is a hybrid approach. We package the main app as a standard Splunk app in an .spl file. In the splunk_pas_ref_app.spl file, we also include the other add-on apps in a subfolder along with a set of scripts that the customer runs to install them. The advantages of this approach for us are:

  • It is relatively simple for us to build the .spl file and the installation scripts.
  • We can deploy the .spl file to Splunkbase where customers can find it and then install it in their own environments.
  • After the customer has installed the main app, it is relatively simple to run the install script to configure the add-on apps.

The disadvantages of this approach are:

  • Installation is a two-step process. First, install the main PAS app in the standard way from Splunkbase. Then run the appropriate script to install the add-on apps in the local Splunk Enterprise environment.
  • The scripts assume that you are deploying all the apps on the same Splunk Enterprise instance. The user must edit the scripts or install the add-on apps manually in more complex environments that have multiple Splunk instances (such as forwarders) on multiple machines.

Details of the packaging solution

This section describes the important details of how we package and deploy the PAS app. The high-level steps are:

  • Make sure that you organize all the correct files into the correct folder structure.
  • Create an .spl file.
  • Upload the .spl file to Splunkbase.

After you have uploaded the .spl file to Splunkbase, customers can discover your app and install it in their own environments. The following screenshot shows the PAS app ready for a customer to install from the Splunkbase website:

Including the add-on apps and scripts

The main PAS app has the standard folder structure of a Splunk app. For example, it contains these folders: bin, default, metadata, and static. Therefore, the customer can deploy the main PAS app in their environment in the same way as any other Splunk app. We also include all of our add-on apps in the subfolder appserver/addons/ as shown in the following partial directory listing.

| ├───addons
| | ├───eventgen
| | | └───...
| | ├───splunk-add-on-googledrive
| | | └───...
| | ├───splunk-add-on-jira-alert
| | | └───...
| | ├───pas_hr_info
| | | └───...
| | ├───pas_simulated_application_addon
| | | └───...
| | ├───pas_simulated_database_addon
| | | └───...
| | ├───pas_simulated_files_addon
| | | └───...
| | └───pas_simulated_keycard_addon
| | | └───...
| └───static
|   └───components
|     ├───calendarheatmap
|     | └───...
|     ├───d3
|     | └───...
|     ├───dendrogram
|     | └───...
|     └───kvstore_backbone
|       └───...

Splunk Enterprise does not locate and configure these add-on apps in this location. Therefore, in the bin folder in the main PAS app we include two scripts that create symbolic links in the etc/apps/ folder that point to these subfolders containing the add-on apps. The following script creates the symbolic links in OS X, Linux, or *nix operating systems.

echo ===
echo Creating symbolic links to Splunk Reference Solution add-ons in $SPLUNK_HOME
echo ===
cd ../appserver/addons
ln -sv $PWD/* $SPLUNK_HOME/etc/apps
ADMEnsure the SPLUNK_HOME environment variable is properly set.

TESTIf testing an app on different versions of Splunk Enterprise on the same box, install each version in a dedicated folder (perhaps, adding a suffix for the version, for example, splunk_630). Then, it's easy to switch between different versions by resetting SPLUNK_HOME.

The following install-addons.ps1 PowerShell script performs the same task in Windows:

function createSymLinks ($source, $destination)
 Write-Host "Creating symbolic links to Splunk Reference Solution add-ons in" $env:SPLUNK_HOME "\etc\apps" -foregroundcolor black -backgroundcolor white
     $children = Get-ChildItem $source
     for ($i=0; $i -lt $children.Count; $i++) {
      # Write-Host destination="$dest" source = $children[$i].FullName addon= $children[$i]
      # Write-Host Creating... -foregroundcolor green
      cmd /c mklink /d  $dest $children[$i].FullName
cd ..\appserver
createSymLinks -source addons -destination $env:SPLUNK_HOME\etc\apps
SECWe signed this PowerShell script to make it easier for the customer to run the script securely without weakening the execution policies.

The following screenshot shows the details of the symbolic links the script creates from the Terminal window in OS X:

Both scripts fail gracefully if you attempt to run them a second time. They will not override existing apps or symbolic links.

Promoting add-ons to their own packages

The JIRA alert and Google Drive add-ons that we have built in support of the PAS app v1.5 have utility outside of this project. We would like to highlight them and share them with the community, while still using them in their original context within the PAS app. To this end, we have split them out into their own repositories and have linked them into the main PAS repository by use of Git submodules. In this way we can still include the apps in the main PAS package for easy deployment with our script, but we don't need to duplicate any code. We can work on that functionality with a greater focus and remain confident that we are always packaging the latest version of that code.

Since our script only creates links if they don't already exist, our users can also install either of the new standalone modules before installing the PAS app with no ill effects.

We also publish these two add-ons as standalone packages on Splunkbase.

Versions and other metadata

To enable updating the apps at some future time, it's important to add version numbers to both the main PAS app and the add-ons. We do this in the app.conf files for each app. The following example is from the main PAS app:

# Splunk app configuration file
is_configured = 0
check_for_updates = 1
id = pas_ref_app
is_visible = True
label = Splunk Reference App - PAS
author = Splunk Dev
description = Splunk Reference App - PAS, including the main app and the underlying add-ons.
version = 1.5.0
SHIPThe version number can also include text. For example, 0.81 beta.

The package stanza contains metadata related to app upgrades, and the launcher stanza determines how the Launcher page in Splunk Enterprise and the Splunkbase website display information about the app. For more information about the contents of the app.conf file, see "app.conf" in the "Admin Manual."

Tips and useful resources

The page "Step 8: Package your app or add-on" in the "Developing Views and Apps for Splunkweb Manual" contains lots of useful resources related to packaging an app including:

  • What should be in the app.conf file.
  • How to add an icon and screenshots on Splunkbase or in the Launcher. These files must have particular resolutions and you must place them in a particular location.
  • How to tar and zip your app on different platforms. Note that you must manually install the gnutar tool referenced on this page on OS X.
SHIPThere are also Splunk CLI commands that can be used for packaging. The latter will ignore all auxiliary git-related files:
  • splunk package app appname
  • git archive

SHIPWhen you are creating the .spl file, it's a good idea to check out a clean copy of your app from your source control system and use that. This removes the risk of accidently including any temporary file or files created by Splunk Enterprise at runtime. However, you should be sure not to include files related to your source control system such as .git and .gitignore in the .spl package file.

When you tar and zip your app, you must add the folder that contains all your app resources as the root folder in the package.

For the PAS app, we have chosen to include in the .spl file the indexes.conf file that specifies the location of the pas index files. This is a convenience to let a customer downloading and installing the PAS app on a single machine get started quickly with minimal manual configuration steps. In a distributed production environment, an administrator typically needs to configure the indexes on specific machines.

When you upload your .spl file to Splunkbase, it validates the app folder structure and app.conf file. However, the Splunkbase team must approve the app before it becomes visible on Splunkbase. This usually takes about 24 hours to complete.

SHIPWhen publishing your apps/add-ons on Splunkbase, follow these naming conventions:

During the submission process you must provide various pieces of metadata. These include the app name, a short description, license details, and contact details. For more information about the submission process including naming guidelines and approval criteria, see "About submitting content" in the "Working with Splunkbase Manual."

Managing authorization and permissions

We include two roles in the PAS app to let an administrator manage access to the functionality on the Setup dashboard. Any users that an administrator adds to the pasadmin role will be able to save the PAS configuration values on the Setup dashboard. Any users that an administrator adds to the pasuser role will only have permissions to read the configuration data created by a member of pasadmin role. For more information about how the Setup dashboard uses the App Key Value Store (KV Store) to persist global configuration settings for the PAS app, see the chapter "Working with data: where it comes from and how we manage it" in this guide.

You can view the capabilities we assign to these two roles on the Access Controls settings page in Splunk Enterprise. However these pages are large, so here we show just the content of the authorize.conf file where the settings are saved:

srchIndexesDefault = main;pas
importRoles = user
srchIndexesAllowed = *
srchIndexesDefault = main;pas
accelerate_datamodel = enabled
edit_user = enabled
importRoles = pasuser
srchIndexesAllowed = *
srchIndexesDefault = main;pas

In this file, you can see that we add the pas index to the list of indexes searched by default by members of the built-in admin role. The custom pasuser role inherits from the built-in user role and also searches the pas index by default. The custom pasadmin role inherits from the custom pasuser role and also searches the pas index by default. The pasadmin role is also authorized to accelerate data models and edit user settings.

ADMThe administrator can add any existing indexes to the pasuser and pasadmin roles. Events from customer indexes will show up in the PAS app, provided they are tagged correctly.

ADMWhen you define a new role, you should always inherit from an existing role and then add the required capabilities.

To restrict access to the PAS configuration settings, we set the permissions on the KV Store collections (ri_setup and ri-setup-coll) that contain the configuration settings. The following screenshot shows the permissions for the ri-setup-coll KV Store collection, where you can see that we give the pasadmin role read and write permissions, and the pasuser role read permissions only:

Preparing the roles and permissions for packaging

When we defined our custom roles and permissions, we used the Settings pages in Splunk Web UI. To include these settings it's important to copy them from the location where the Splunk Web UI saves them, to the correct files that we include in the .spl package.

When we create a new role from the Settings page, Splunk Enterprise saves the configuration in the file etc/system/local/authorize.conf. We copy the PAS app roles to the file authorize.conf in the default folder in the PAS app. We also remove some unused attributes (such as cumulativeRTSrchJobsQuota and cumulativeSrchJobsQuota) from the role definitions to keep them tidy (see the content of authorize.conf shown above).

When we change the permissions settings through the Settings page, Splunk Enterprise strong the configuration in the file local.meta in the metadata folder in the PAS app. This file is not included when we package the app. We must copy these permissions settings into the default.meta file in the metadata folder that is included when we package the app in the .spl archive file. We modify the contents of this file as shown in the following snippet from the default.meta file:

# Modify access permissions for specific kvstore collections if that level of granularity is needed
version = 6.3.0
version = 6.3.0
# Specified write permissions to admin and pasadmin roles only
access = read : [pasuser,admin,pasadmin], write : [ admin, pasadmin ]
export = none
version = 6.3.0

Here we are using the [ ] stanza notation to apply the permissions to all knowledge objects in the PAS app. It's possible to set more granular permissions by overriding the permissions in a stanza that relates to a specific knowledge object.

Programmatically enabling and disabling an add-on app

The PAS app has an optional dependency on the Eventgen app that can generate sample data for the searches in the app. We are planning to deploy a version of the PAS app to a cloud-hosted sandbox environment to let customers explore the app without installing it in a local Splunk instance. However, to avoid consuming unnecessary processing resources in the cloud we want to disable the Eventgen app by default, but let a customer enable it if they want to view pseudo-live data in the PAS app. To achieve the goal we considered four options:

  • Programmatically enabling/disabling the Eventgen app using the JavaScript SDK
    When we investigated this option, we discovered that the JavaScript SDK does not support enabling and disabling apps from client-side code running in the browser. We would need to run this code server-side, and this would require a major change to the PAS app.
  • Programmatically enabling/disabling the Eventgen app via REST API call
    Due to time constraints, we decided not to investigate this option any further. However, it's possible that this option is not viable using client-side code because it may be prevented by design for security reasons.
  • Programmatically enabling/disabling the underlying modular input of the Eventgen app
    Again, due to time constraints, we decided not to investigate this option any further and opted for the simplest approach possible.
  • Programmatically detect the status of the app and inform the user
    This is the simplest approach and is the one we chose. The next section describes this approach in more detail.

Programmatically detect the status of an app and inform the user

This approach, which is the simplest, is the approach we take in the PAS app. We detect the state of the Eventgen app using the code in the following snippet, and display an appropriate message to the user on the Setup page based on the current state of the Eventgen app. The code retrieves all the installed apps using mvc class, checks that the Eventgen app is installed, and then checks the status of the Eventgen app.

var service = mvc.createService();
    .fetch(function(err, apps) {
        if (err) {
        var eventgenApp = apps.item('eventgen')
        if (eventgenApp) {
            eventgenApp.fetch(function(err, eventgenApp) {
                if (err) {
                if (eventgenApp.state().content.disabled) {
                } else if (!eventgenApp.state().content.disabled) {
        } else {

If we detect that the Eventgen app is not installed, we show the customer a link to download and install it. If we detect that the Eventgen app is installed but disabled, we show the customer a link to the Manage apps page in Splunk Enterprise where the customer can manually enable the app.

Managing app upgrades

We plan to ship enhanced versions of the PAS app in the future and customers should be able to install the new version over the top of the current version. When we package the PAS app, we include two important pieces of information in the app.conf file: a version number such as 0.91 and the check_for_updates configuration value. These enable Splunk Enterprise to detect when a new version of the app is available on Splunkbase and then notify the customer.

SHIP When we upload a new version of the PAS app package to Splunkbase, the web site checks that we are uploading a later version. The site also gives us the option of keeping the new version hidden while we complete our testing. When the tests are complete, we can mark the new version as the default version for customers to install.

Note that from the perspective of the Splunkbase website, the package containing the PAS main app and add-ons only contains a single app (the add-ons are installed by the customer) and therefore instances of Splunk Enterprise will only detect when the version of the main app is incremented. If we want to distribute new versions of the add-on apps independently of the main app then we need to package each add-on as a separate .spl to upload to Splunkbase. For example, we created a separate .spl file for our Google Drive add-on app and uploaded it to Splunkbase to let customers upgrade it independently.

App certification

When submitting an app to Splunkbase, you can request an optional app certification by Splunk. Certified apps recognition constitutes high quality, including a security review, sufficient documentation, and user support. Certified apps will receive special visibility and promotion across Splunkbase to reach more users worldwide.

You can find the list of app certification criteria at:

What did we learn?

This section summarizes some of the key lessons learned when we were packaging and deploying our apps:

  • You package an app into an .spl file. This file is an archive that contains all the resources your app needs to run.
  • If your solution contains multiple apps (a main app and add-on apps) you must choose how you plan to deploy all these apps together.
  • You can upload your .spl file to the Splunkbase web site where customers can discover and install it.
  • Apps uploaded to the Splunkbase web site are validated by Splunk before they become available to customers.
  • You can detect whether or not required dependencies (add-on apps) are available by including custom code in your main app.
  • The Splunkbase web site can notify customers when you make a new version of your app available.

More information

To search for existing apps or find out more information about Splunkbase, see:

To see how other apps handled the installation of a solution containing multiple Splunk apps, see:

For more information about the contents of the app.conf file, see:

For useful resources related to packaging an app, see "Step 8: Package your app or add-on" at:

For more information about the submission process including naming guidelines and approval criteria, see "About submitting content" at: