Add app supportability

Customer Support uses diagnostic files, or diags, to help troubleshoot issues with apps and Splunk installations. Diags are typically gathered by system administrators using the Splunk Diag command-line tool. Splunk Diag collects configuration files, directory listings, system status, and similar information, and then outputs this to a a .tar.gz file, which can then be sent to Support or someone who can assist with troubleshooting.

As a Splunk app developer, you can provide additional data about the troubleshooting information that Splunk Diag collects from your app to support your customers, and use this information yourself if you want. For example, you could allow information from Splunk REST endpoints to be gathered to help troubleshoot your app. Or, you could run a program and collect its output.

 

Determining what Splunk Diag collects

Each app provides a certain amount of information to Splunk Diag by default, including the contents of the app directory.

Splunk Diag also attempts to limit the collection of sensitive data by masking password hashes and skipping files that might contain event data. App developers can assess what is collected by Splunk Diag, and modify the set of data that can be gathered for their apps.

The following command collects only the information that your app provides, along with the baseline of information about memory, storage, CPU availability, and so forth:

splunk diag --collect app:your_app_name

The Splunk Diag command-line tool also provides options to specify what to include or exclude from the diags based on file paths and file categories, further controlling what is gathered.

For more about using Splunk Diag and what is included in diags by default, see Generate a diag in the Splunk Enterprise Troubleshooting Manual.

 

Setting up your app for Splunk Diag output

To set up your app to modify the information that is included in Splunk Diag output, you must:

  • Declare the diag extension in the app.conf configuration file.
  • Implement data collection by creating a Python script.
 

Declare the diag extension

To declare the diag extension in your app:

  1. In a text editor, open your app.conf configuration file, which is located in $SPLUNK_HOME/etc/apps/appname/default by default.
  2. Add a new [diag] stanza with the extension_script setting, which specifies the name of your Python data collection script, as follows:
  3. [diag]
    extension_script = your_script.py
    

    When specifying the script filename:

    • Do not specify a path. The script must be located in your app's /bin directory, and no other path is allowed.
    • Be sure to match the case. Although Mac and Windows have case-insensitive file systems, other environments such as Linux and Solaris are case sensitive.
  4. By default, apps are limited to 100MB of data in a diag, which prevents the app from damaging customer and support workflows in case of unexpected app behavior. If you expect your app to provide much more or much less data in a diag, you can adjust the data limit using the data_limit setting, as follows:
  5. [diag]
    extension_script = your_script.py
    data_limit = 50MB
    
  6. Save your changes.
 

Create the data collection script

The data collection Python script lets you specify additional data that is collected from your app for diags by default. Consider the following guidelines when creating your script:

  • When choosing what to include, consider what a customer might consider to be sensitive information. Generally, customer event data should not be included, even in small amounts.
  • Large amounts of data cause slow collection and logistical problems. Hundreds of megabytes of information for a single app is not good default behavior.
  • Use the splunk diag command-line interface to let the user opt in to send more data when needed.
  • The script should avoid stalling. A special interface is provided to gather REST endpoints with enforced timeouts.
  • Ensure that errors are not hidden. Let exceptions go uncaught or log them.

Your Python module can include an optional setup() function, and must include a collect_diag_info() function, described below. You can also use the Python logging module functions in your script such as logging.warn() and logging.error(). By default, warning and errors are captured to the diag.log file inside the diag. However, the output from logging.debug() is only captured when you run the diag command with the --debug option.

 

setup()

This function is not required to explicitly accept all arguments. However, because the arguments are passed as keywords, the argument names you use must match these names.

setup(parser=parser_obj, app_dir=path, callback=callback_obj, **kwargs)

Arguments

  • parser: An optparse parser object. Short option strings are not supported. App-specific options are placed in their own visual group in the diag --help output.
  • The primary method is parser.add_option(). For example:

    parser.add_option('--my-option', help="This is my option")
    
  • app_dir: An absolute path referencing the installed location of your app. Because apps can be stored in different locations within a Splunk installation, use this path to locate the files and directories used by your app.
  • callback: A callback object with one method, will_need_rest(), which you call if your app gathers REST data. Calling the callback.will_need_rest() function triggers a Splunk login prompt.
  • **kwargs: Required. A container to gracefully accept arguments. This argument accomodates future Splunk releases.
 

collect_diag_info()

This function is not required to explicitly accept all arguments. However, because the arguments are passed as keywords, the argument names you use must match these names.

collect_diag_info(diag=diag_adding_obj, app_dir=path,
                  options=app_options, global_options=global_options, **kwargs)

Arguments

  • diag: An object that adds data to the output and builds a .tar.gz file. This object provides the following ways to add data.
  • To add a single file:

    diag.add_file(filesystem_path, diag_path)
    

    To recursively add all contents inside a directory:

    diag.add_dir(filesystem_path, diag_path)
    

    To add the contents of a Python string as a single file:

    diag.add_string(a_string, diag_path)
    

    To add the output of a splunkd REST endpoint as a single file:

    diag.add_rest_endpoint(endpoint, diag_path)
    
    • An extension using this method must call the callback.will_use_rest() function during setup.
    • The endpoint may include GET parameters, but should not include the host, port, or protocol. For example: "/services/fictional/endpoint?param1=val1&param2=val2".

    The data is stored in the diag in a path prefixed by diag-name/app_ext/your_app_name, followed by the provided diag_path. For example, the following command stores the file as diag-name/app_ext/your_app_name/text/myfile.txt:

    diag.add_file(os.path.join(app_dir, "my_file.txt"), "text/myfile.txt")
    
  • app_dir: An absolute path referencing the installed location of your app. Because apps can be stored in different locations within a Splunk installation, use this path to locate the files and directories used by your app.
  • options: An optparse options object that contains any app-specific options that were declared when setup() was called.
  • global_options: An optparse options object that contains global diag options. These options are provided in case the app wants to honor one of the general directives in an app-interpreted manner. For more about these options, run the diag --help command, or see the server.conf.spec file that is included with Splunk Enterprise.
  • **kwargs: Required. A container to gracefully accept arguments. This argument accomodates future Splunk releases.
 

Script examples

The following example shows a very simple data collection script:

def collect_diag_info(diag, **kwargs):
     diag.add_string("data", "filename.txt")

The following example shows a script that demonstrates the entire Splunk Diag interface:

import os
import logging
 
# Use the **args pattern to ignore options we don't care about.
def setup(parser=None, callback=None, **kwargs):
    logging.debug("test_app's setup() was called!")
 
    # Create an option that shows up in diag --help.
    # This translates into --test_app:grom at the command line.
    # Because this app is called test_app, we can
    # use the value as options.gromvar below
    parser.add_option("--grom", dest="gromvar")
     
    # Declare that we're going to use REST later
    callback.will_need_rest()
 
# The options are out of order, as is possible for keyword invocation
def collect_diag_info(diag, options=None, global_options=None, app_dir=None, **kwargs):
    logging.debug("test_app's collect_diag_info() was called!")
    
    # Collect a file located in the app. In this case the script itself is added
    a_file = os.path.join(app_dir, 'bin', 'diag.py')

    # Rename the file, so that <app>/bin/diag.py appears in the diag as app_ext/<app>/waohoo.txt
    diag.add_file(a_file, 'waohoo.txt')
 
    # Collect a directory from the app
    a_dir = os.path.join(app_dir, 'whatever')
    diag.add_dir(a_dir, 'whatever')
     
    # Try the option we requested above
    if options.gromvar:
        logging.error("gromvar is %s!" % options.gromvar)

    # Collect some REST endpoint data
    diag.add_rest_endpoint("/services/server/info", "server_info.xml")