Tips for writing scripts for modular and scripted inputs in Splunk Cloud Platform or Splunk Enterprise

Here are some tips for writing reliable scripts for modular and scripted inputs.

 Clear environment variables

Clear environment variables that can affect your script's operation. One environment variable that is likely to cause problems is the library path. The library path is most commonly known as LD_LIBRARY_PATH on Linux, Solaris, and FreeBSD. On OSX it is DYLD_LIBRARY_PATH. On AIX, it is LIBPATH

If you are running external python software or using other python interpreters, consider clearing PYTHONPATH.

Changing PYTHONPATH might affect other installations of Python.

On Windows platforms, the SPLUNK_HOME environment variable is set for you. Avoid changing this environment variable. Changing this variable might interfere with the functioning of Splunk platform services.

 Use the same Python version as the Splunk platform installation

For best results, use the version of Python available from your Splunk platform installation. The Splunk platform uses this version to run system scripts. Use this version of Python to test your scripts.

Some Python libraries that your script requires might not be available in the Splunk platform's version of Python. In this case, you can copy the libraries to the same directory as the scripted input.

To run a script using the version of Python available from Splunk Enterprise, use the following command:

Bash

 Be careful with file paths in Python

Be careful when specifying platform-specific paths and relative paths.

 Avoid hard coding platform-specific paths

When writing scripts in Python, avoid hard coding platform-specific file paths. Instead specify file paths that can be interpreted correctly on Windows, UNIX, and Mac platforms. For example, the following Python code launches try.py, which is in the /bin directory of your app, and has been made cross-compatible with Python 2 and Python 3 using Python-Future on https://python-future.org.

from __future__ import print_function
import os
import subprocess
 
# Edit directory names here if appropriate
if os.name == 'nt':
    ## Full path to your Splunk installation
    splunk_home = 'C:\Program Files\Splunk'
...
Show More

 Avoid using relative paths

Avoid using relative paths in scripts. Python scripts do not use the current directory when resolving relative paths. For example, on *nix platforms, relative paths are set relative to the root directory (/). The following example shows how to locate the extract.conf file, which is in the same directory as the script:

import os
import os.path

script_dirpath = os.path.dirname(os.path.join(os.getcwd(), __file__))

config_filepath = os.path.join(script_dirpath, 'extract.conf')

 Format the script output

Format the output of a script so Splunk software can easily parse the data. Consider formatting data so it is more human-readable.

 Use the Common Information Model (CIM) to normalize data

You can break down most log files into three components: fields, event type tags, and host tags. With these components you can set up log files in a way that makes them easy to process and that normalizes non-compliant log files, forcing them to follow a similar scheme. Using the Common Information Model (CIM), you can organize these fields and tags into categories and provides a separate data model for each category. You can use the CIM data models to test your data to ensure that it has been normalized correctly, and then report on it.

Download the Common Information Model Add-on from Splunkbase.

See the Common Information Model Add-on Manual.

 Format timestamps

Format timestamps at the beginning of an event as follows:

• RFC-822 and RFC-3339 formats

  RFC-822 and RFC-3339 are standard timestamp formats for email headers and internet protocols. These formats provide an offset from GMT, and thus are unambiguous and more human-readable. RFC-822 and RFC-3339 formats can be handled with %z in a TIME_FORMAT setting.

• UTC format

  UTC formatting might not be as human-readable as some of the other options. If the timestamp is UNIX time, no configuration is necessary. Otherwise, UTC format requires a configuration in the props.conf configuration file to declare the input as TZ=UTC.

The following table shows examples of these time formats.

 Manage multiline data and field names

For multiline data, separate events using some of the following techniques:

• Write a distinctive initial line for a multiline event.

• Use a special end-of-event string to separate events. For example, use three newline characters to specify an end of an event. The event then includes any single or double newline characters.

• For multiline field values, place the field data inside quotes.

• Use an equals sign, =, or other separator to expose name-value pairs. For example, key=value.

• Configure your Splunk platform instance to use other tokens that might exist in the data.

• Field names are case sensitive. For example, the field names "message" and "Message" represent different fields. Be consistent when naming fields.

 Use checkpoints to save state across invocations of the script

Scripts often need to checkpoint their work so that subsequent invocations can pick up from where they left off. For example, save the last ID read from a database, mark the line and column read from a text file, or otherwise note the last input read.

You can checkpoint either the index or the script. When checkpointing data, keep in mind that the following actions are not connected as a transaction:

• Writing out checkpoint files
• Fully writing data into the pipe between the script and splunkd
• splunkd completely writing out the data into the index

In the case of hard crashes, knowing whether the data that the script has acquired has been properly indexed is difficult. Consider trying the following approaches:

• Search the Splunk index.

  One strategy is to have the scripted input search in the Splunk index to find the last relevant event. This is reasonable in an infrequently-launched script, such as one that is launched every 5 or 10 minutes, or at launch time for a script which launches once and stays running indefinitely.

• Maintain an independent checkpoint.

  Because there is some delay between data being fed to the Splunk platform and the data becoming searchable, a frequently run scripted input must maintain its own checkpoint independent of the index.

• Choose a scenario.

  If the script always believes its own checkpoint, data might not be indexed on splunkd. If the index search is believed, some data might be indexed multiple times on splunkd or system crash. You need to choose which scenario best fits your needs.

 Provide secure access to resources

Use proper security measures for scripts that need credentials to access secured resources. Here are a few suggestions on how to provide secure access. However, no method is foolproof, so think carefully about your use case and design secure access appropriately.

• Restrict which users can access the app or add-on on disk.
• Create and use credentials specific to the script, with the minimum permissions required to access the data.
• Avoid putting literal passwords in scripts or passing the password as a command line argument, making it visible to all local processes with operating system access.
• Encrypt passwords. You can create a setup page that allows users to enter passwords. For an example of a setup page that stores encrypted credentials, see the Module 2: Add a setup page to your app in Splunk Cloud Platform or Splunk Enterprise tutorial.

The Splunk platform assembles a secret using locally-available random seeds to encrypt passwords stored in configuration files. This method provides modest security against disclosure of passwords from administrators with local disk read capability. However, it is not an adequate protection for privileged accounts.

 Consider concurrency issues for scripted inputs

Be careful to avoid scheduling two copies of a script to run at the same time. The Splunk platform detects if another instance of the script is running, and does not launch a new instance if another instance is already running. For example, if you have a script scheduled to run every 60 seconds, and a particular invocation takes 140 seconds, the Splunk platform detects this and does not launch a new instance until after the long-running instance completes.

At times you might want to run multiple copies of a script, for example to poll independent databases. For these cases, design your scripts so they can handle multiple servers. Also, design your script so that multiple copies can exist. For example, use two app directories for the script.

Alternatively, you can have separate scripts using the same source type.

 Review the log file for scheduled scripts

The Splunk platform logs exceptions thrown by scheduled scripts to the splunkd.log file, located in $SPLUNK_HOME/var/log/splunk/splunkd.log. Check splunkd.log first if expected events do not appear in the expected index after scheduling the scripted input.

 Design scripts to handle script shutdowns and restarts

Keep these shutdown and restart issues in mind when designing your scripts:

• Output at least one event at a time.

  This makes it easier to avoid reading a partial event if the script is terminated or crashes. Events are typically completed in a timely manner, so built-in time-outs prevent truncated or incomplete events.

  Configure the pipe fd as line-buffered, or use write() to write full events at once. Be sure the events are flushed using line buffered or unbuffered fflush().

• Output relatively small batches of events.

  Fetching thousands of events over a few minutes and then outputting them all at once increases the risk of losing data due to a restart. Alternatively, outputting small batches of events means your data is searchable sooner and improves script transparency.

 See also

• Create custom data inputs for Splunk Cloud Platform or Splunk Enterprise