Overview of the Splunk Packaging Toolkit

Welcome to the Splunk Packaging Toolkit Beta! The Packaging Toolkit is a tool for authoring, packaging, and validating a Splunk app in a way that eases up app management, including installation, configuration, and updating.

The Packaging Toolkit provides support for all phases of Splunk app lifecycle management. For Splunk app developers, that means they can focus on building apps without concerning themselves with different deployment topologies or the details of the deployment process. For Splunk Enterprise administrators, it means that they can easily and reliably install and manage Splunk apps across their entire Splunk deployment from a single point.

The Packaging Toolkit, with minimal additional work by developers during design, disambiguates configuration and code, and partitions, or repackages, relevant pieces of an app into deployment packages. Each deployment package can be based on either physical workloads (forwarders, search heads, indexers, etc.) or logical groups of inputs.

Note: The Splunk Packaging Toolkit is still under development. It is currently available with API and command-line interface (CLI) support only, but development of a user interface is planned.

This topic contains the following sections:

Key concepts

To understand the Packaging Toolkit and what it does, you must understand the following key concepts:

The app manifest

The Packaging Toolkit relies on an app manifest to keep track of what components go where and what dependencies are required. Even when an app manifest is not provided, the Packaging Toolkit can derive one from .conf files and other Splunk app artifacts.

You are not required to do anything with your current apps to take advantage of partitioning across search heads, indexers, and forwarders, but to take advantage of advanced capabilities around partitioning for different forwarder types and input groups, you must provide logical declarations in the app manifest. If not, the Packaging Toolkit can't break the forwarder types further down. Further, if you don't declare dependencies in the app manifest, the Packaging Toolkit can't do dependency resolution.

When you run the Packaging Toolkit's manifest generation command, it auto-generates a list of physical partitioning details, and it stubs out several additional sections. To enable logical partitioning capabilities and dependency resolution, you must augment the app manifest with additional details.

Partitioning

Splunk Enterprise allows you to put any app or add-on you'd like onto any deployment node, even if it's not relevant to or intended for that type of node. For example, modular inputs have no reason to run on a search head, but if components related to modular inputs exist in your Splunk app and you distribute the app to a search head, those modular input components will run on that search head. This is unnecessary, and can lead to problems and potential performance issues in a complex deployment.

The Packaging Toolkit's main capability is partitioning. Partitioning is the act of taking a Splunk app's source package, examining it, and splitting it into a minimum of three deployment packages. Each deployment package contains only the information, configuration, and corresponding code that is relevant to the type of deployment node for which it is intended.

The following diagram illustrates this concept. In it, the Packaging Toolkit generates three deployment packages: one for search heads, one for indexers, and one for forwarders.

Diagram illustrating Packaging Toolkit partitioning: Source package on the left leads to the Packaging Toolkit, which splits it into three deployment packages, each intended for a different deployment node.

This functionality is available in the Packaging Toolkit out of the box, with no additional development necessary for your Splunk app. All existing Splunk apps can be partitioned into these three workloads automatically.

This is partitioning across physical boundaries. Each deployment node is its own workload. The Packaging Toolkit can also partition across logical boundaries for forwarders. Partitioning among logical groups must be done by developers at design time. Later, at deployment time, admins can map logical groups to collections of server classes or nodes.

For example, you might want to have a dedicated deployment package not just for all forwarders, but one that contains special handling for a group of forwarders that deal with Microsoft Exchange data. If a developer has the unique logic and code that apply to that logical group and wants to package it, the Packaging Toolkit enables the developer to declare those logical groupings. It will then, when partitioning, respect those declarations. At install time, admins can either map the logical groups to nodes of server classes or ignore them altogether and collapse them all into one.

Input groups

An input group specifies a named group of data inputs as defined in the inputs.conf file. Each group typically targets a specific forwarding environment. Input groups can be selectively deployed to specific server classes. For example, you could define an input group that targets AD domain servers on specific versions of Windows.

You define input groups in the app.manifest file. To learn more about defining input groups, see the app.manifest schema topic. In particular, see the schema members that begin with inputGroups. You can define the following input group attributes, among others, from within the app.manifest file:

  • Required dependencies
  • List of included data inputs
  • Defined stanzas in inputs.conf
  • Descriptive information

By setting the appropriate app.manifest attributes, developers can enable the Packaging Toolkit to seamlessly integrate an app with the input groups of its dependencies, and Splunk admins can select input groups for deployment without needing to know about the underlying app dependencies and their input groups.

Packaging Toolkit workflows

This section contains descriptions of typical Packaging Toolkit workflows. The first is for developers, and is usually followed by the admin workflow.

Developer workflow

To try out the Packaging Toolkit, you'll want to do the following:

  1. Install the Packaging Toolkit.
  2. If your app includes custom .conf files, add corresponding placement rules. See Creating placement rules for custom .conf files later on this page.
  3. Install your Splunk app to <app-root>.
  4. Generate an app manifest from an existing app package using the generate-manifest command. The Packaging Toolkit parses the package's .conf files to validate the app configuration. The default generated manifest has basic information already filled out, such as the app name, author name, description, and so on.
  5. Augment the rest of the manifest as needed, in particular the [dependencies] and [inputGroups] sections. See app manifest augmentation recommendations later on this page.
  6. Describe the manifest and dependencies using the describe command.
  7. With the app manifest at the root of the app folder, package the app source using the package command. Specify a repository if you want to package your app's dependencies with your main app.
  8. Validate the app package using the validate command.

The Packaging Toolkit looks at the app manifest and performs dependency resolution. It makes sure that any of the packages that you've defined as dependencies don't conflict with each other. The output of the package operation is the app, along with its dependencies if you opted to package them with the app. The package's file name is in the form <group_name>-<app_name>-<version>.tar.gz. For <version>, the Packaging Toolkit uses Semantic Versioning (semver) notation. Note that this is simply the name of the package. The name of the app has not changed.

This completes the Packaging Toolkit developer workflow. You now have a package with an app manifest defined with dependencies and input groups. You can now hand the package off to a Splunk admin, who does partitioning and input group mapping.

Admin workflow

Though a user interface is planned for the admin workflow of the Packaging Toolkit, admins can also use the CLI, which is available now.

  1. Use the partition command to partition the app into deployment packages. If you partition an app without any input group mappings, you generate the three default deployment packages. You can also specify input group mappings on the command line.
  2. Resolve any dependency conflicts. The packaging toolkit checks the dependencies specified in the app that you are partitioning against any installed apps, and might perform one of the following resolutions.
    • Requests authorization if the app that you are partitioning requires any installed apps to be upgraded or downgraded.
    • Fails if the app that you are partitioning or its dependencies are incompatible with any installed apps.
  3. If the packaging toolkit does not find any conflicts, then it generates the deployment packages according to the manifest without requesting authorization. You can now deploy the resulting deployment packages.

Creating placement rules for custom .conf files

If you create a custom .conf file, then you need to create placement rules. Placement rules tell the Packaging Toolkit how to partition the .conf files, or stanzas within the .conf files, into deployment packages. Add placement rules such as @placement indexer, @placement forwarder, or @placement search-head to the corresponding conf.spec file.

You can also combine placement rules. For example, use @placement search-head, indexer to partition the .conf file to both the search head and indexer deployment packages.

Base your placement rules on the following example from props.conf.spec.

@placement forwarder

CHARSET = <string>
* This setting applies at input time, when data is first read by Splunk.
  
@placement indexer
  
LINE_BREAKER = <regular expression>
* Specifies a regex that determines how the raw text stream is broken into initial events, before line merging takes place. (See the SHOULD_LINEMERGE attribute, below)
TRUNCATE = <non-negative integer>
* Changes the default maximum line length (in bytes).
TRANSFORMS-<class> = <transform_stanza_name>, <transform_stanza_name2>,...
* Used for creating indexed fields (index-time field extractions).

App manifest augmentation recommendations

At a minimum, consider augmenting the following within the app manifest:

Dependencies

The "dependencies" section is where you define what other apps your app depends on ("<app-group>:<app-name>"). You can specify version numbers ("version"): specifying "*" indicates any version, or you can specify a specific version or a range of versions. The Packaging Toolkit recognizes Semantic Versioning (semver) notation to indicate version numbers.

You can even specify that you are including the app packages themselves within your app ("package": "<source-package-location>"). These dependencies are located in a hidden .dependencies folder. The Packaging Toolkit uses the .dependencies folder in all commands except for slim generate-manifest. See the following example directory structure for the location of the hidden .dependencies folder:

|-- <app_id>-<version>.tar.gz
|   |-- <app_source>
|   |-- .dependencies
|       |-- <dependency_1>-<version>.tar.gz
|       |-- <dependency_2>-<version>.tar.gz

You only need to know and declare top-level dependencies. That is, if your dependencies have their own dependencies, such as a utility or another app, the Packaging Toolkit will find these "nested" dependencies and validate their version numbers.

Input groups

The "inputGroups" section is where you can define logical groups in which you assign inputs from either your app or your dependencies' apps. The logical groups that you define will eventually map to server classes. Though you as a developer might not know ahead of time which server classes exist, you should know if certain inputs are meant to be grouped together.

Within input groups, you can define which input groups in a dependent app that you want to bubble up into a top-level input group. If, instead, you don't care about specific input groups you want to bubble up, use the "*" notation to indicate you want all input groups to bubble up.

Tasks

By default, the Packaging Toolkit partitions all stanzas in the inputs.conf file to the forwarder deployment package. You might have management tasks or scheduled tasks in the inputs.conf file that need to be partitioned to the search head deployment package instead. You can indicate which stanzas in the inputs.conf file need to be packaged differently in the app manifest file.

For example, if you define [input://task_id] in your inputs.conf file, then add the following section to the app.manifest file.

{
    ...
    "tasks": ["input://task_id"]
}