How to create modular inputs in Splunk SDK for C# v2.x

Support for modular inputs in Splunk® Enterprise 5.0 and later enables you to add new types of custom inputs to Splunk Enterprise that are treated as native Splunk Enterprise inputs. Your users interactively create and update instances of modular inputs using Splunk Web, just as they do for native inputs. If you're not already familiar with modular inputs, see the following topics to get started:

Some examples of modular inputs include:

The Splunk SDK for C# includes built-in support for creating and using modular inputs with C#.

This topic shows you how to create modular inputs using the Splunk SDK for C#, and how to integrate them into your app that extends Splunk Enterprise.

This topic contains the following sections:

Why use modular inputs?

Modular inputs are ideal for packaging and sharing technology-specific data sources. One of the primary reasons to use modular inputs is that they enable users to interact with key information using the familiar Splunk Web interface, without needing to edit config files. Modular inputs also provide runtime controls and allow the user to specify per-event index-time settings for the input.

Splunk Enterprise treats your custom modular input definition as if it was one of its native inputs, and users can interactively create and update the input in Splunk Settings just as they would for native inputs. Enabling simple user-level customization in this way—not to mention built-in support for validation, multi-platform support, custom REST endpoint access, and more—differentiates modular inputs from scripted inputs. More information about the differences between modular inputs and traditional scripted inputs is available here: Modular inputs vs. scripted inputs.

What you can do with modular input support in the SDK

With the Splunk SDK for C# you can:

  • Create a modular input.
  • Define the scheme for your modular input.
  • Implement a handler to create and stream back results to Splunk Enterprise.
  • Implement your modular input in an asynchronous, non-blocking manner.
  • Create both single and multiple-instance modular inputs.
  • Log easily back to Splunk Enterprise from within your modular input.
  • Easily access Splunk Enterprise capabilities via the SDK from within your modular input.
  • Package your modular input within a Splunk Processing Language (SPL) query to deploy it to other Splunk Enterprise instances.

Modular input SDK example

The Splunk SDK for C# includes a modular input example in the examples directory: random-numbers. To run these examples, you'll need to install them. (They are not installed with the SDK installation process.)

  1. Set the $SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.

  2. Build the project using your IDE, and then copy the random-numbers directory from:

    ...\splunk-sdk-csharp-pcl\examples\random-numbers\bin\Debug\app
    

    into

    $SPLUNK_HOME\etc\apps\
    
  3. Restart Splunk Enterprise. From Splunk Home, click the Settings menu. Under System, click Server Controls. Click Restart Splunk.

After you have installed the example modular input, it appears alongside other data inputs. To try out the modular input:

  1. From Splunk Home, click the Settings menu.

  2. Under Data, click Data inputs, and find the new "Random numbers" modular input.

  3. Click Random numbers, and then click New.

  4. Give the input some values: Enter a name, a minimum value, and a maximum value.

  5. Click Save.

  6. Go to the Search & Reporting app, and search using the input you just created. For example, to search using an instance of the random-numbers modular input called example, set source as shown here:

    source="random-numbers://example"

To get a better understanding of how the examples work, take a look at the source code, which is located in the Program.cs file within the random-numbers example directory.

To create modular inputs programmatically

With the Splunk SDK for C#, you can create modular inputs programmatically using C#. Adding a modular input to Splunk Enterprise is a two-step process: First write a modular input script, and then package the script with several accompanying files.

To write a modular input script in C#

A modular input script does the following:

  • Return the introspection scheme to Splunk Enterprise. The introspection scheme defines the behavior and endpoints of the script. When Splunk Enterprise starts, it runs the script to determine the modular input's behavior and configuration.
  • Validate the script's configuration (optional). Whenever a user creates or edits an input, Splunk Enterprise can call the script to validate the configuration.
  • Stream data. The script streams event data that can be indexed by Splunk Enterprise. Splunk Enterprise invokes the script and waits for it to stream events.

Splunk Enterprise defines a contract that all implementations of modular inputs must obey. The Splunk SDK for C# provides a class that implements most of the details of that contract, leaving only the details of your particular modular input to be filled in.

A modular input written using the Splunk SDK for C# should inherit from the Splunk.ModularInputs.ModularInput class. The preceding three steps are accomplished as follows using the Splunk SDK for C#:

In addition, you must define a minimal Main method.

Here's the framework of a new modular input script created in C#:

namespace random_numbers 
{
    using Splunk.ModularInputs;
    public class RandomNumbers : ModularInput
    {
        public static int Main(string[] args)
        {
            return Run<RandomNumbers>(args);
        }

        public override Scheme Scheme
        {
            get
            {
                ...
            }
        }

        public override async Task StreamEventsAsync(
            InputDefinition inputDefinition, 
            EventWriter eventWriter)
        {
            ...
        }
    }
}

Splunk Enterprise calls the program a number of times. The first time is when Splunk Enterprise starts; it calls the program to get a scheme for the modular input by invoking the get method of the Scheme class. If the script is using external validation, Splunk Enterprise calls the program—invoking the Validate method—each time an instance of the modular input is created or edited to check whether the instance configuration is valid. If you don't override the Validate method, the validation always passes. Finally, Splunk Enterprise invokes the StreamEventsAsync() method for each modular input instance. If the modular input has been configured to run as a single instance, then all instances are handled by separate invocations of StreamEventsAsync() in the same process. Otherwise, a new process starts for each instance.

Using this framework as a starting point, we'll now guide you through the creation of the components of a modular input script in C#. This is the same script that is located at /splunk-sdk-csharp-pcl/examples/random-numbers/Program.cs. It produces a series of random numbers to demonstrate event generation and streaming.

Return the scheme

The scheme needs to define, at minimum, a human-readable title for the modular input and a set of arguments. All the available fields are defined on the Scheme class, so you can explore them with IntelliSense in Visual Studio.

For example, the get method for the Scheme object in the random-numbers example is as follows:

get {
    return new Scheme
    {
        Title = "Random numbers",
        Description = "Generate random numbers in the specified range",
        Arguments = new List<Argument>
        {
            new Argument
            {
                Name = "min",
                Description = "Generated value should be at least min",
                DataType = DataType.Number,
                RequiredOnCreate = true
            },
            new Argument
            {
                Name = "max",
                Description = "Generated value should be less than max",
                DataType = DataType.Number,
                RequiredOnCreate = true
            }
        }
    };
}

Validate the configuration

You can specify a string to the Validation property of an Argument in a Scheme, and Splunk Enterprise will perform internal validation based on what you specify. For the format of the built-in validation functions you can use, see Validation of arguments in the main modular inputs documentation. If you need more functionality than built-in validation offers, you have two more places you can perform validation.

First, set the UseExternalValidation property of the modular input's Scheme object to true. Then you can set delegates to do validation for each parameter, and define a method to do validation on the whole set of parameters.

To add a delegate to a particular parameter, create an instance of ValidationHandler and set it as the value of the argument's ValidationDelegate property.

ValidationHandler is defined as follows:

public delegate bool ValidationHandler(Parameter parameter, out string errorMessage);

To define validation on the whole collection of parameters, override the Validate method. The Validate method takes a Validation object—which contains some server metadata and the proposed parameter values—and an out parameter errorMessage that returns an error message if there is an error. If Validate returns true, then the parameters are taken to be valid and the error message is ignored. If Validate returns false, then the error message is returned to Splunk Enterprise.

Following is an example implementation of Validate from the random-numbers example:

public override bool Validate(Validation validation, out string errorMessage)
{
    double min = validation.Parameters["min"].ToDouble();
    double max = validation.Parameters["max"].ToDouble();

    if (min >= max) {
        errorMessage = "min must be less than max.";
        return false;
    }
    else
    {
        errorMessage = "";
        return true;
    }
}

Stream events

The StreamEventsAsync method is where the event streaming happens. StreamEventsAsync should be an async function. If a modular input is running in single instance mode, then all the instances of StreamEventsAsync must cooperate. This means that you must:

  • Use only async calls in the implementation.
  • Start anything that must be synchronous in a new Task with the Task.Run method.
  • Use the Task.Delay method instead of the Thread.Sleep method if you need to wait.

The StreamEventsAsync method is passed both an InputDefinition instance that specifies the parameters of this instance of the modular input and an EventWriter instance that is used for all output. You must generate all output through the EventWriter object, as it is engineered to work properly in the presence of multiple instances of modular inputs running at once.

Parameters for an instance are passed in the Parameters property of the InputDefinition object. You can coerce parameters to all the types you would expect—string, int, long, float, double, and bool for single valued parameters; and arrays of the same for multivalued parameters. An invalid cast will result in a runtime error.

Then stream events back to the server using the EventWriter object's QueueEventForWriting method. Events are specified as instances of a structured class Event.

Following is an example implementation of the StreamEventsAsync method:

public override async Task StreamEventsAsync(
    InputDefinition inputDefinition, EventWriter eventWriter)
{
    double min = inputDefinition.Parameters["min"].ToDouble();
    double max = inputDefinition.Parameters["max"].ToDouble();

    while (true)
    {
        await Task.Delay(1000);
        await eventWriter.QueueEventForWriting(new Event
        {
            Stanza = inputDefinition.Name,
            Data = "number=" + (rnd.NextDouble() * (max - min) + min)
        });
    }
}

To add the modular input to Splunk Enterprise

With your modular input script completed, you're ready to integrate it into Splunk Enterprise. First, build the project, then package the script, and then install the modular input.

Build the project

Build your project, taking care of any errors that may appear.

Once you've built the project successfully, an executable with the name of your project will appear within your project's bin directory. Keep track of this file, as well as the .DLL files here; you'll need them later.

Package the script

To add a modular input that you've created in C# to Splunk Enterprise, you'll need to add it as a Splunk Enterprise app. First, create the files you'll need to package the input as an app.

Files

Create the following files with the content indicated. Wherever you see modinput_name—whether in the file name or its contents—replace it with the name of your modular input C# project. For example, if your project's name is "random_numbers," give the directory indicated as modinput_name the name random_numbers.

app.conf

When creating this file, replace the values given with the corresponding values for your modular input:

  • The is_configured value determines whether the modular input is preconfigured on install, or whether the user should configure it.
  • The is_visible value determines whether the modular input is visible to the user in Splunk Web.
[install]
is_configured = 0

[ui]
is_visible = 0
label = My modular input

[launcher]
author=Me
description=My great modular input
version = 1.0

inputs.conf.spec

When creating this file, in addition to replacing modinput_name with the name of your modular input's project, do the following:

  • After the asterisk (*), type a description for your modular input.
  • Add any arguments to your modular input as shown. You must list every argument that you defined in the Scheme method.
[modinput_name://<name>]
*My great modular input.
arg1 = <value>
arg2 = <value>
count = <value>  
Directories

Next, create a directory that corresponds to the name of your modular input script—for instance, "modinput_name"—in a location such as your Documents directory. (It can be anywhere; you'll copy the directory over to your Splunk Enterprise directory at the end of this process.)

  1. Within this directory, create the following directory structure:
    modinput_name/
        bin/
        default/
        README/
  2. Copy your modular input executable (modinput_name.exe), the two .DLL files alongside the executable, and the files you created in the previous sections so that your directory structure looks like this:
    modinput_name/
        bin/
          modinput_name.exe
          Splunk.Client.dll
          Splunk.ModularInputs.dll
        default/
          app.conf
        README
          inputs.conf.spec

Install the modular input

Before using your modular input as a data input for your Splunk Enterprise instance, you must first install it.

  1. Set the SPLUNK_HOME environment variable to the root directory of your Splunk Enterprise instance.

  2. Copy the directory you created in Package the script to the following directory:

    $SPLUNK_HOME/etc/apps
  3. Restart Splunk Enterprise: From Splunk Home, click the Settings menu. Under System, click Server Controls. Click Restart Splunk.

After you have installed your modular input, it appears alongside other data inputs. To try out the modular input:

  1. From Splunk Home, click the Settings menu.

  2. Under Data, click Data inputs, and find the name of your modular input.

  3. Click the name of your modular input, and then click New.

  4. Give the input some values: Fill in the settings that you specified when you created the modular input script.

  5. Click Save.

  6. Go to the Search & Reporting app, and search using the input you just created. For example, to search using an instance of the modinput_name modular input called example, set source as shown here:

source="modinput_name://example"

You've now configured an instance of your modular input as a Splunk Enterprise input.