Create AppInspect custom checks

You can create your own custom checks for AppInspect validation. This topic covers how to create a custom checks directory and groups of custom checks.

If you believe that your custom checks would benefit the Splunk AppInspect CLI tool and its users, the AppInspect team encourages you to submit your checks to help extend the check coverage. E-mail us at appinspect@splunk.com.

Note: You are responsible for any content that you submit to Splunk. Do not submit any content that contains sensitive or personally identifiable information.

Overview

To add custom checks to AppInspect, you first create and specify a custom checks directory. This directory can go anywhere. Inside the custom checks directory you add group files, in which you write the individual checks. For more information about how to create group files and how to write checks, see Custom check implementation, later in this topic.

Once you create your custom checks, run the splunk-appinspect tool and use the --custom-checks-dir option to specify the location of the custom checks directory. Both the list and inspect commands support the --custom-checks-dir option.

--custom-checks-dir option with list command

splunk-appinspect list [groups | checks | tags] --custom-checks-dir path/to/custom/checks/directory/

--custom-checks-dir option with inspect command

splunk-appinspect inspect path/to/splunk_app.tgz --custom-checks-dir path/to/custom/checks/directory

Filter support

When using a custom checks directory, if checks are tagged, they can be filtered by those tags by using the following options:

  • --included-tags
  • --excluded-tags

AppInspect uses tags to provide a mechanism of selection and filtering for validation. Each check is tagged with a keyword that specifies what the check is relevant to. For example, the tag "splunk_appinspect" indicates that the check is relevant to app certification. The "manual" tag means that the check is only run in precert mode. If the --included-tags and --excluded-tags options are not passed any information, all checks are run by default.

For more information about tagging and filtering checks, see the "Filtering on a tag" section in Use the AppInspect tool.

Custom check implementation

When writing a custom check, it is important to be aware of some specifics before getting started.

Custom check limitations

Splunk AppInspect only supports a single custom checks directory for the time being.

Group file conventions

Checks are stored in files called groups, which are Python files. There are several Splunk AppInspect check conventions, some of which are required and some that are optional. These conventions are listed in this section.

Required conventions

  • All group files should be Python (.py) files.
  • Tag all checks appropriately—for example, tag for functionality, for validation goal, and so on.
  • The name of each group file must start with check_. This is how AppInspect identifies a check file. For example: 
    check_tutorial_example.py
  • Check functions must start with the check_ prefix. This is how AppInspect identifies a check function within the group files. For example: 
    check_example(app, reporter)
  • Checks must be tagged. If possible, add a tag that emphasizes functionality or feature support. Add the manual tag if the check expects a manual check returned as a result.
  • Checks must have a Splunk AppInspect version added to them. This is done to enable versioning with the Splunk AppInspect tool. If you are unsure what version number to use, use the current version number for AppInspect.
  • Checks must use the reporter object to report results. To do this, specify reporter in the function signature, as specified in reporter object, later in this topic.

Optional conventions

  • Each group file can have a module level doc string. The doc string is used to describe the goal of the group, and is used in the splunk-appinspect tool for outputting information.
  • Each check function can have a doc string. The doc string is used to describe the functionality and goal of the check. The splunk-appinspect tool uses the doc string to output information.
  • Checks can use the app object to help simplify the inspection of a Splunk app. To learn more, see app object, later in this topic.

Check function dependency injection

A check function has some of its objects populated dynamically when executed. This enables easier integration so that the checks can be focused on validating functionality.

These dynamically populated objects include classes in the source code that can be used to see available functionality. By adding the object name as a parameter to a check function, the object will end up being included "behind the scenes" via dependency injection.

The parameters that can be dynamically injected are:

  • app: An app object
  • reporter: A reporter object

app object

The app object is stored in splunk-appinspect/app.py and represents the Splunk app being validated.

To learn more about the app object and its capabilities, view the source code and read the doc strings. Pasted below are the contents of app.py, but for the most accurate and up-to-date information, see the app.py file in the AppInspect download.

By using the app argument in a check, the check will be able to perform generalized inspection tasks.

def get_config(self, name, dir='default', config_file=None):
  """Returns a parsed config file as a ConfFile object. Note that this 
    does not do any of Splunk's layering- this is just the config file, 
    parsed into a dictionary that is accessed via the ConfFile's helper 
    functions.
    :param name The name of the config file.  For example, 'inputs.conf'
    :param dir The directory in which to look for the config file.  By default, 'default'
    """
def get_spec(self, name, dir='default', config_file=None):
    """Returns a parsed config spec file as a ConfFile object.  
    :param name The name of the config file.  For example, 'inputs.conf.spec'
    :param dir The directory in which to look for the config file.  By default, 'default'
    """
def get_meta(self, name, directory='metadata', meta_file=None):
    """Returns a parsed meta file as a Meta object.  
    :param name The name of the meta file.  For example, 'default.meta'
    :param directory The directory in which to look for the config file.
        By default, 'default'
    """
def get_raw_conf(self, name, dir='default'):
    """Returns a raw version of the config file.
     
    :param name: The name of the config file.  For example 'inputs.conf'
    :param dir The directory in which to look for the config file.  By default, 'default'
    :return: A raw representation of the conf file
    """
def get_filename(self, *path_parts):
    """Given a relative path, return a fully qualified location to that file
    in a format suitable for passing to open, etc.
    example: app.get_filename('default', 'inputs.conf')
    """
def app_info(self):
    """Get app version, hash, title, description, author for the app header."""
def iterate_files(self, basedir='', excluded_dirs=[], types=[], excluded_types=[], recurse_depth=float("inf")):
    """Iterates through each of the files in the app, optionally filtered
    by file extension.
    Example:
    for file in app.iterate_files(types=['.gif', '.jpg']):
        pass
    This should be considered to only be a top down traversal/iteration. 
    This is because the filtering of directories, and logic used to track
    depth are based on the os.walk functionality using the argument of 
    `topdown=True` as a default value. If bottom up traversal is desired 
    then a separate function will need to be created.
    :param basedir The directory to start in
    :param excluded_dirs These are directories to exclude when iterating. 
        Exclusion is done by directory name matching only. This means if you
        exclude the directory 'examples' it would exclude both `examples/`
        and `default/examples`, as well as any path containing a directory
        called `examples`.
    :param types An array of types that the filename should match
    :param excluded_types An array of file extensions that should be
        skipped.
    :param recurse_depth This is used to indicate how deep you want 
        traversal to go. 0 means do no recurse, but return the files at the 
        directory specified.
    """
def file_exists(self, *path_parts):
    """Check for the existence of a file given the relative path.
    Returns True/False
    Example:
    if app.file_exists('default', 'transforms.conf'):
         print "File exists! Validate that~!~"
    """
def directory_exists(self, *path_parts):
    """Check for the existence of a directory given the relative path.
    Returns True/False
    Example:
    if app.file_exists('local'):
         print "Distributed apps shouldn't have a 'local' directory"
    """
def some_files_exist(self, files):
    """Takes an array of relative filenames and returns true if any file
    listed exists.
    """
def some_directories_exist(self, directories):
    """Takes an array of relative paths and returns true if any file
    listed exists.
    """
def all_files_exist(self, files):
    """Takes an array of relative filenames and returns true if all
    listed files exist.
    """
def all_directories_exist(self, directories):
    """Takes an array of relative paths and returns true if all listed 
    directories exists.
    """
def search_for_patterns(self, patterns, basedir='', excluded_dirs=[], types=[], excluded_types=[]):
    """Takes a list of patterns and iterates through all files, running 
    each of the patterns on each line of each of those files.
    Returns a list of tuples- the first element is the file (with line 
    number), the second is the match from the regular expression.
    """
def search_for_pattern(self, pattern, basedir='', excluded_dirs=[], types=[], excluded_types=[]):
    """Takes a pattern and iterates over matching files, testing each line.
    Same as search_for_patterns, but with a single pattern.
    """
def is_executable(self, filename):
    """Checks to see if any of the executable bits are set on a file."""

reporter object

The reporter object is stored in splunk-appinspect/reporter.py, and represents a report for the check. The reporter object is used to provide output regarding the check's success.

To learn more about the reporter object and its capabilities, view the source code and read the doc strings. Pasted below are the contents of reporter.py, but for the most accurate and up-to-date information, see the reporter.py file in the AppInspect download.

By using the reporter argument in a check, the check will be able to return results of a check, and specify information about the test for output.

def warn(self, message):
    """A warn will require that the app be inspected by a real human. Like a
    to-do item
    """
def manual_check(self, message):
    """Declare that this check requires a human to validate"""
def not_applicable(self, message):
    """Report that this check does not apply to the current app"""
def fail(self, message):
    """Failure is when a problem has been found that the app can't be 
    accepted without fixing
    """
def exception(self, exception, category='error'):
    """Error is when there's something wrong with the check script. 
    Don't call this directly- just throw an exception
    """
Note: The warn method will always report success, but it will generate additional messages that you might need to address.

Examples

Suppose you wanted to create a check that validates that your Splunk apps aren't doing "bad things." For demonstration purposes, let's interpret this to mean that the Splunk app being validated shouldn't contain the word "bad." Specifically:

  • If the word "bad" is detected in the Splunk app, the check fails the Splunk app.
  • If the word "bad" is not detected in the Splunk app, the check passes the Splunk app.
  • There is no "not applicable" criteria that fits this check.

To create a new check, go through the following basic procedure:

  1. Create a custom checks directory. Splunk recommends keeping custom checks separate from the built-in AppInspect checks, to avoid confusion.
  2. Create a new Python file with a descriptive name—for example, check_for_bad_files_concerns.py.
  3. Add any required Python imports.
  4. Create the check logic.

The following sections contain example code for our "bad things" example check.

Group file example

Following is a typical group file. Pay close attention to the doc strings to learn about the different parts of a group file.

"""This is a module-level doc string that is used to describe the group."""
@splunk_appinspect.tags("example", "splunk_appinspect")                            # Tags are used to associate features and functionality.
@splunk_appinspect.cert_version(min='1.2.0')                                       # Versions are used to associate integration support.
def check_example_function_one(app, reporter):                                     # The function should start with 'check_' and have app and reporter as part of the function signature.
    """Create a doc-string to describe the check."""                               # This is used to describe a check.
    # Custom check logic goes here.
    # When finished with the check, use the reporter object to return the result. If the reporter is not given a result output it will default to success.
  
#....
  
@splunk_appinspect.tags("example", "splunk_appinspect")                            # Tags are used to associate features and functionality.
@splunk_appinspect.cert_version(min='1.2.0')                                       # Versions are used to associate integration support.
def check_example_function_two(app, reporter):                                     # The function should start with 'check_' and have app and reporter as part of the function signature.
    """Create a doc-string to describe the check."""                               # This is used to describe a check.
    # Custom check logic goes here
    # When finished with the check, use the reporter object to return the result. If the reporter is not given a result output it will default to success

Custom check example

The following is a typical check, which is implemented as a Python function and written inside of a group file. Pay close attention to the doc strings to learn about the different parts of a check.

# This is the file check_for_bad_files_concerns.py
 
# Python imports would go here
"""This doc string is used to describe the group checks as a whole.
check_for_bad_files_concerns makes sure that bad things do not happen
"""
@splunk_appinspect.tags("example", "splunk_appinspect")                            # Tag your check.
@splunk_appinspect.cert_version(min='1.1.20')                                      # Specify the version of Splunk AppInspect you're adding to.
def check_for_bad_files(app, reporter):                                            # Declare the dependencies your check will need; all checks require an app and a reporter object.
    """Look through each file for the word 'bad'. Files are not allowed to contain # Create a doc string for the check; this is used to generate human readable information about a check.
    the word 'bad'. It's bad for the other files' self-esteem.
    """                        
    pattern = "bad"                                                               # A regex pattern to search for.
    matches = app.search_for_pattern(pattern)                                      # The matches that were found.
    for (fileref_output, match) in matches:
        filepath, line_number = fileref_output.split(":")                          # The search_for_pattern function returns a tuple with the file reference and line number, (file_path:line_number), this splits them out to separate variables.
        reporter_output = ("A bad file was detected and that just isn't"
                           "allowed."
                           " File: {}"
                           " Line: {}.").format(filepath,
                                                line_number)
        reporter.fail(reporter_output)                                             # If no reporter output is returned it is assumed to be a success by default.

Advice

When creating custom checks, keep in mind the following information:

  • Try to use group files to aggregate similar checks, or checks oriented towards one common goal. The purpose of groups is to group checks with similar goals into one central area. For example, you might create a group of checks that deal with a specific file type validation, specific feature set validation, and so on.
  • If you're not sure about possible uses or ways to interact with the code base, look at the source code! There are a lot of good examples of checks in the source code for Splunk AppInspect.
  • Make sure to write doc strings, as these are responsible for providing output and clarification to Splunk AppInspect.