How to work with modular inputs in the Splunk SDK for JavaScript

Support for modular inputs in Splunk® Enterprise 5.0 and later enables you to add new types of custom inputs to Splunk Enterprise that are treated as native Splunk Enterprise inputs. Your users interactively create and update your custom inputs using Splunk Web, just as they do for native inputs. If you're not already familiar with modular inputs, see the following topics to get started:

Some examples of modular inputs include:

The Splunk SDK for JavaScript (starting in version 1.4) includes built-in support for creating and using modular inputs with JavaScript. In addition, Node.js offers rich support for handling I/O-bound workloads and streaming data asynchronously. It can scale out across multiple cores and machines, and has a very rich ecosystem of third party modules via npm.

This topic shows you how to create modular inputs using the Splunk SDK for JavaScript, and how to integrate them into your app that incorporates Splunk Enterprise.

This topic contains the following sections:

Why use modular inputs?

Modular inputs are ideal for packaging and sharing technology-specific data sources. One of the primary reasons to use modular inputs is to enable users to interact with key information using the familiar Splunk Web interface, without needing to edit config files. Modular inputs also provide runtime controls and allow the user to specify per-event index-time settings for the input.

Splunk Enterprise treats your custom modular input definition as if it was a native input, allowing users to interactively create and update the input in Splunk Settings just as they would for native inputs. Enabling simple user-level customization in this way differentiates modular inputs from scripted inputs, as does built-in support for validation, multi-platform support, custom REST endpoint access, and more. More information about the differences between modular inputs and traditional scripted inputs is available here: Modular inputs vs. scripted inputs.

What you can do with modular input support in the SDK

With the Splunk SDK for JavaScript you can:

  • Create a modular input.
  • Define the scheme for your modular input.
  • Implement a handler to create and stream back events to Splunk Enterprise.
  • Wire up handlers to handle setup and teardown of the process.
  • Wire up handlers to handle stream start and end.
  • Implement your modular input in an asynchronous, non-blocking manner.
  • Create both single and multiple-instance modular inputs.
  • Log easily back to Splunk from within your modular input.
  • Easily access Splunk capabilities via the SDK from within your modular input.
  • Package your modular input within a Splunk Processing Language (SPL) query to deploy it to other Splunk instances.

Prerequisites

Before you can get started creating modular inputs using the Splunk SDK for JavaScript, ensure you have the following installed on your computer:

  • Node.js v0.8 or later: Splunk Enterprise includes Node.js v0.8 in its standard install, but you should download and install Node.js from http://nodejs.org/download/ if you do not have Splunk Enterprise installed on your JavaScript development computer or if you want the latest version.
  • Splunk SDK for JavaScript v1.4 or later: Install the latest version of the Splunk SDK for JavaScript. The functionality described in this topic is only available in version 1.4 or later of the SDK.

Modular input SDK examples

The Splunk SDK for JavaScript includes two modular input examples in the /splunk-sdk-javascript/examples/modularinputs directory: random numbers and Github commits. To run these examples, you'll need to install them. (They are not installed with the SDK installation process.)

  1. Set the SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.

  2. Copy one or both of the sample directories (github_commits or random_numbers) to the following directory:

    $SPLUNK_HOME/etc/apps
  3. Open a command prompt or terminal window and go to the following directory, where sample_name is either github_commits or random_numbers:

    $SPLUNK_HOME/etc/apps/sample_name/bin/app
  4. Type the following, and then press Enter or Return:

    npm install

    If the install is not successful, first ensure that Node.js is installed. For more information, see Prerequisites. If Node.js is installed, try the following:

    1. Download and extract a ZIP file of the SDK from Github: https://github.com/splunk/splunk-sdk-javascript/archive/master.zip

    2. Copy the extracted splunk-sdk-javascript-master folder to the following path, where sample_name is either github_commits or random_numbers:

      $SPLUNK_HOME/etc/apps/sample_name/bin/app/node_modules
    3. Rename the copied folder to splunk-sdk.

  5. Restart Splunk Enterprise: From Splunk Home, click the Settings menu. Under System, click Server Controls. Click Restart Splunk.

After you have installed the example modular inputs, they appear alongside other data inputs. From Splunk Home, click the Settings menu. Under Data, click Data inputs, and find the names of the modular inputs you just added. Click Add new and fill in the settings that you specified when you created the modular input script:

  • The Random Numbers example has fields for the name of the input and the minimum and maximum numbers to be produced.
  • The Github Commits example has fields for the name of the input, and the owner and name of the repository. There is also an optional access token field to avoid Github's API limits, though it is required if the repository you entered is private. To generate an access token, visit the application settings page, click Generate new token, make sure the repo and public_repo scope checkboxes are enabled, and then click Generate token.

To get a better understanding of how the examples work, take a look at the source code, which is located in the bin/app/ directory within the random_numbers or github_commits example's directory. Within app is a .JS file (named using the name of the app) that does all the work for the modular input.

To create modular inputs programmatically

With the Splunk SDK for JavaScript, you can create modular inputs programmatically using JavaScript. Adding a modular input to Splunk Enterprise is a two-step process: First, write a modular input script, and then package the script with several accompanying files and install it.

To write a modular input script in JavaScript

A modular input script does the following:

  1. Return the introspection scheme to Splunk Enterprise. The introspection scheme defines the behavior and endpoints of the script. When Splunk Enterprise starts, it runs the script to determine the modular input's behavior and configuration.

  2. Validate the script's configuration (optional). Whenever a user creates or edits an input, Splunk Enterprise can call the script to validate the configuration.

  3. Stream data. The script streams event data that can be indexed by Splunk Enterprise. Splunk Enterprise invokes the script and waits for it to stream events.

To create a modular input programmatically using JavaScript, first require "splunk-sdk". In our examples, we've also assigned the classes we'll be using to variables, for convenience. At the very least, we recommend defining a ModularInputs variable as shown here:

    var splunkjs        = require("splunk-sdk");
    var ModularInputs   = splunkjs.ModularInputs;

The preceding three steps are accomplished as follows using the Splunk SDK for JavaScript:

  1. Return the introspection scheme: Define the getScheme method on the exports object.

  2. Validate the script's configuration (optional): Define the validateInput method. This is required if you set the scheme returned by getScheme to use external validation (that is, set Scheme.useExternalValidation to true).

  3. Stream data: Define a function for the streamEvents method.

In addition, you must run the script by calling the ModularInputs.execute method, passing in the exports object you just configured:

ModularInputs.execute(exports, module);

Here's the skeleton of a new modular input script created in JavaScript:

(function() {
    var splunkjs        = require("splunk-sdk");
    var ModularInputs   = splunkjs.ModularInputs;
    var Logger          = ModularInputs.Logger;
    var Event           = ModularInputs.Event;
    var Scheme          = ModularInputs.Scheme;
    var Argument        = ModularInputs.Argument;
    // other global variables here

    // getScheme method returns introspection scheme
    exports.getScheme = function() {
        var scheme = new Scheme("My Modular Input");

        // scheme properties
        scheme.description = "A modular input.";
        scheme.useExternalValidation = true;  // if true, must define validateInput method
        scheme.useSingleInstance = true;      // if true, all instances of mod input passed to
                                              //   a single script instance; if false, user 
                                              //   can set the interval parameter under "more settings"

        // add arguments
        scheme.args = [
            new Argument({
                name: "arg",
                dataType: Argument.dataTypeNumber,
                description: "An argument.",
                requiredOnCreate: true,
                requiredOnEdit: false
            }),
            new Argument({
                name: "count",
                dataType: Argument.dataTypeNumber,
                description: "A counter.",
                requiredOnCreate: true,
                requiredOnEdit: false
            })
            // other arguments here
        ];

        return scheme;
    };

    // validateInput method validates the script's configuration (optional)
    exports.validateInput = function(definition, done) {
        // local variables here
        var arg = parseFloat(definition.parameters.arg);
        var count = parseInt(definition.parameters.count, 10);

        // error checking goes here
	if (count < 0) {
            done(new Error("The count was a negative number."));
        }
        else {
            done();
        }
    };

    // streamEvents streams the events to Splunk Enterprise
    exports.streamEvents = function(name, singleInput, eventWriter, done) {
        // modular input logic goes here
        var getMyArgument = function (arg) {
            return arg;
        };

        // local variables here
        var arg = parseFloat(singleInput.arg);
        var count = parseInt(singleInput.count, 10);
        var errorFound = false;

        // stream as many events as specified by the count parameter
        for (var i = 0; i < count && !errorFound; i++) {            
            var curEvent = new Event({
                stanza: name,
                data: "argument=" + getArgument(arg)
            });

            try {
                eventWriter.writeEvent(curEvent);
            }
            catch (e) {
                errorFound = true; // Make sure we stop streaming if there's an error at any point
                Logger.error(name, e.message);
                done(e);

                // we had an error; die
                return;
            }
        }

        // streaming is done
        done();
    };

    ModularInputs.execute(exports, module);
})();

Using this skeleton as a starting point, we'll now guide you through the creation of the components of a modular input script in JavaScript. This is the same script that is located at /splunk-sdk-javascript/ examples/modularinputs/random_numbers/. It produces a series of random numbers to demonstrate event generation and streaming.

The getScheme method

When Splunk Enterprise starts, it looks for all the modular inputs defined by its configuration, and tries to run them with the argument --scheme. Splunkd expects each modular input to print a description of itself in XML to stdout. The SDK's modular input framework takes care of all the details of formatting the XML and printing it. You only need to implement a getScheme method to return a new Scheme object.

First, create a new Scheme object, providing both a name and description for it:

exports.getScheme = function() {
    var scheme = new Scheme("Random Numbers");
    scheme.description = "Streams events containing a random number.";

In this case, Splunk Enterprise will display "Random Numbers" to users for this input, with the given description.

Next, specify whether you want to use external validation using the useExternalValidation property. External validation is taken care of by implementing the validateInput method. If you set external validation without implementing the validateInput method, the script will accept anything as valid.

scheme.useExternalValidation = true;

If you set useSingleInstance to true, the scheme will pass all the instances of the modular input to a single instance of the script. You're then responsible for handling all of the instances of the modular input.

scheme.useSingleInstance = true;

Generally you only need external validation if there are relationships you must maintain among the parameters, such as requiring one variable to be less than another, or checking whether some resource is reachable or valid. For example, in our random numbers example, we want to ensure the minimum value is always smaller than the maximum value. (See The validateInput method for more information.) If you don't choose external validation, Splunk Enterprise lets you specify a validation string for each argument and runs validation internally using that string.

In the random_numbers modular input example, there are three parameters—min, max, and count—that represent the minimum and maximum values for the generated random numbers, plus a count used to determine how many events to stream per input execution. We'll add them to the scheme's args property using the Argument class and its properties:

    scheme.args = [
        new Argument({
            name: "min",
            dataType: Argument.dataTypeNumber,
            description: "Minimum random number to be produced by this input.",
            requiredOnCreate: true,
            requiredOnEdit: false
        }),
        new Argument({
            name: "max",
            dataType: Argument.dataTypeNumber,
            description: "Maximum random number to be produced by this input.",
            requiredOnCreate: true,
            requiredOnEdit: false
        }),
        new Argument({
            name: "count",
            dataType: Argument.dataTypeNumber,
            description: "Number of events to generate.",
            requiredOnCreate: true,
            requiredOnEdit: false
        })
    ];

After adding arguments to the scheme, return the scheme:

return scheme;
};

The validateInput method

The validateInput method is where the configuration of an input is validated, and is only needed if you've set your modular input to use external validation. If validateInput does not throw an error, the input is assumed to be valid. Otherwise it throws an error when it tells splunkd that the configuration is not valid.

When you use external validation, after splunkd calls the modular input with the --scheme argument to get the scheme, it calls it again with the --validate-arguments argument for each instance of the modular inputs in its configuration files, feeding XML on stdin to the modular input to validate all enabled inputs. Splunkd calls the modular input the same way again whenever the modular input's configuration is changed.

In our random_numbers example, we're using external validation, since we want the max variable to always be greater than the min value, and that count is a non-negative value. Our validateInput method contains basic logic that retrieves the two variables and then compares them to each other:

exports.validateInput = function(definition, done) {
    var min = parseFloat(definition.parameters.min);
    var max = parseFloat(definition.parameters.max);
    var count = parseInt(definition.parameters.count, 10);

    if (min >= max) {
        done(new Error("min must be less than max; found min=" + min + ", max=" + max));
        }
    else if (count < 0) {
        done(new Error("count must be a positive value; found count=" + count));
    }
    else {
        done();
    }
};

The streamEvents method

The streamEvents method is where the event streaming happens. Events are streamed into stdout using an InputDefinition object as input that determines what events are streamed. In the case of the Random Numbers example, for each input, the values are first retrieved and parsed as floats. Then, an Event object is created, its data fields are set, and it's written using the EventWriter.

exports.streamEvents = function(name, singleInput, eventWriter, done) {
    var getRandomFloat = function (min, max) {
        return Math.random() * (max - min) + min;
    };

    var min = parseFloat(singleInput.min);
    var max = parseFloat(singleInput.max);
    var count = parseInt(singleInput.count, 10);

    var errorFound = false;

    for (var i = 0; i < count && !errorFound; i++) {            
        var curEvent = new Event({
            stanza: name,
            data: "number=" + getRandomFloat(min, max)
        });

        try {
            eventWriter.writeEvent(curEvent);
        }
        catch (e) {
            errorFound = true;
            Logger.error(name, e.message);
            done(e);

            return;
        }
    }

    done();
};

Optional: Set up logging

It's best practice for your modular input script to log diagnostic data to splunkd.log. Use a Logger method to write log messages, which include both a standard splunkd.log severity level (such as "DEBUG", "WARN", "ERROR" and so on) and a descriptive message. For instance, the following code is from the Github Commits example, and logs a message if the Github commit has already been indexed:

if (alreadyIndexed > 0) {
    Logger.info(name, "Skipped " + alreadyIndexed.toString() + " already indexed Github commits from " + owner + "/" + repository);
}

To add the modular input to Splunk Enterprise

With your modular input script completed, you're ready to integrate it into Splunk Enterprise. First, package the script, and then install the modular input.

Package the script

To add a modular input that you've created in JavaScript to Splunk Enterprise, you'll need to add the script as a Splunk Enterprise app. First, create the files you'll need to package the script as an app.

Files

Create the following files with the content indicated. Wherever you see modinput_name—whether in the file name or its contents—replace it with the name of your modular input JavaScript file. For example, if your script's name is random_numbers.js, give the file indicated as modinput_name.cmd the name random_numbers.cmd.

modinput_name.cmd

@"%SPLUNK_HOME%"\bin\splunk cmd node "%~dp0\app\modinput_name.js" %*

modinput_name.sh

#!/bin/bash  


current_dir=$(dirname "$0")
"$SPLUNK_HOME/bin/splunk" cmd node "$current_dir/app/modinput_name.js" $@

package.json

When creating this file, replace the values given with the corresponding values for your modular input. All values (except the splunk-sdk dependency, which should stay at ">=1.4.0") can be changed.

{
    "name": "modinput_name",
    "version": "0.0.1",
    "description": "My great modular input",
    "main": "modinput_name.js",
    "dependencies": {
        "splunk-sdk": ">=1.4.0"
    },
    "author": "Me"
}

app.conf

When creating this file, replace the values given with the corresponding values for your modular input:

  • The is_configured value determines whether the modular input is preconfigured on install, or whether the user should configure it.
  • The is_visible value determines whether the modular input is visible to the user in Splunk Web.
[install]
is_configured = 0

[ui]
is_visible = 0
label = My modular input

[launcher]
author=Me
description=My great modular input
version = 1.0

inputs.conf.spec

When creating this file, in addition to replacing modinput_name with the name of your modular input's JavaScript file, do the following:

  • After the asterisk (*), type a description for your modular input.
  • Add any arguments to your modular input as shown. You must list every argument that you define in the getScheme method of your script.
[modinput_name://<name>]
*My great modular input.

arg1 = <value>
arg2 = <value>
count = <value>
Directories

Next, create a directory that corresponds to the name of your modular input script—for instance, "modinput_name"—in a location such as your Documents directory. (It can be anywhere; you'll copy the directory over to your Splunk Enterprise directory at the end of this process.)

  1. Within this directory, create the following directory structure:

    modinput_name/
      bin/
        app/
      default/
      README/
    
  2. Copy your modular input script (modinput_name.js) and the files you created in the previous section so that your directory structure looks like this:

    modinput_name/
      bin/
        modinput_name.cmd
        modinput_name.sh
        app/
          package.json
          modinput_name.js
      default/
        app.conf
      README/
        inputs.conf.spec

Install the modular input

Before using your modular input as a data input for your Splunk Enterprise instance, you must first install it.

  1. Set the $SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.

  2. Copy the directory you created in Package the script to the following directory:

    $SPLUNK_HOME/etc/apps
  3. Open a command prompt or terminal window and go to the following directory, where modinput_name is the name of your modular input script:

    $SPLUNK_HOME/etc/apps/modinput_name/bin/app
  4. Type the following, and then press Enter or Return:

    npm install

    If the install is not successful, first ensure that Node.js is installed. For more information, see Prerequisites. If Node.js is installed, try the following:

    1. Download and extract a ZIP file of the SDK from Github: https://github.com/splunk/splunk-sdk-javascript/archive/master.zip

    2. Copy the extracted splunk-sdk-javascript-master folder to the following path, where modinput_name is the name of your modular input script:

      $SPLUNK_HOME/etc/apps/modinput_name/bin/app/node_modules
    3. Rename the copied folder to splunk-sdk.

  5. Restart Splunk Enterprise: From Splunk Home, click the Settings menu. Under System, click Server Controls. Click Restart Splunk.

After you have installed your modular input, it appears alongside other data inputs. From Splunk Home, click the Settings menu. Under Data, click Data inputs, and find the name of your modular input. Click Add new and fill in the settings that you specified when you created the modular input script. Click Save.

You've now configured an instance of your modular input as a Splunk Enterprise input.