Logging for Splunk Extensions

When you are developing a scripted input, lookup, custom command, or similar extension to Splunk, you should set up a custom log file for debugging. Splunk writes to sys.stdout for normal processing, but you should write errors to a log file to ensure that your debugging code doesn't interfere with Splunk's operations. Splunk uses the Python logging module to provide a comprehensive logging system for your scripts. Logs are generated and indexed by Splunk in the Splunk logging directory $SPLUNK_HOME/var/log/splunk.

The following example shows you how to define a log object, which creates a custom log (foo.log), sets up auto-rotation, and automatically creates the correct filepath format for any operating system:

import sys, os
import logging, logging.handlers
import splunk
def setup_logging():
    logger = logging.getLogger('splunk.foo')    
    SPLUNK_HOME = os.environ['SPLUNK_HOME']
    
    LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
    LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
    LOGGING_STANZA_NAME = 'python'
    LOGGING_FILE_NAME = "foo.log"
    BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
    LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
    splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a') 
    splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
    logger.addHandler(splunk_log_handler)
    splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)
    return logger
logger = setup_logging()
logger.info("hello world!")

Splunk indexes the log files into the _internal index, so you could easily access them within Splunk as follows:

index=_internal source="*/var/log/splunk/foo.log"