How to display search results using the Splunk Enterprise SDK for Python

After you run a search, you can retrieve different output from the search job:

• Events: The untransformed events of the search. There may not be any events if you have a transforming search and you did not specify any status buckets.
• Results: The transformed results of the search after processing has been completed. If the search does not have transforming commands, the results are the same as the events. If there are transforming commands, the result count is less than the event count.
• Results preview: A preview of a search that is still in progress, or results from a real-time search. When the search is complete, the preview results are the same as the results. You must enable previews for non-real-time searches. Previews are enabled automatically for real-time searches.
• Summary: Summary information about the fields of a search from the results that have been read so far. Set status_buckets on the search job to a positive value to access this data.
• Timeline: The event distribution over time of the untransformed events that have been read so far. Set status_buckets on the search job to a positive value to access this data.

This output is returned as a stream in XML, JSON, JSON_COLS, JSON_ROWS, CSV, ATOM, or RAW format. For examples, see Sample output in different formats.

You can display the direct results without a parser or make your own parser. For convenience, the SDK includes a results reader for XML that properly parses and formats the results for you. For a comparison of XML output displayed without a reader versus the SDK's XML results reader, see Results reader comparison.

 The search results APIs

Retrieve a search job using the Job class. From the search job, you can retrieve events, results, preview results, the summary, and timeline information:

• The Job.events method returns a streaming handle to the job's events. You can specify additional parameters to the method. See GET search/jobs/{search_id}/events in the REST API documentation for a list of valid parameters.
• The Job.results method returns a streaming handle to the job's search results. You can specify additional parameters to the method. See GET search/jobs/{search_id}/results in the REST API documentation for a list of valid parameters.
• The Job.preview method returns a streaming handle to the job's preview search results. You can specify additional parameters to the method. See GET search/jobs/{search_id}/results_preview in the REST API documentation for a list of valid parameters.
• The Job.summary method returns a streaming handle to the job's summary. You can specify additional parameters to the method. See GET search/jobs/{search_id}/summary in the REST API documentation for a list of valid parameters.
• The Job.timeline method returns a streaming handle to the job's timeline results. You can specify additional parameters to the method. See GET search/jobs/{search_id}/timeline in the REST API documentation for a list of valid parameters.

Splunk Enterprise search results can be returned in a variety of formats including XML, JSON, and CSV. To make it easier to stream search results in XML format, results are returned as a stream of XML fragments, not as a single XML document. The results module supports incrementally reading one result record at a time from such a results stream. This module also provides an iterator-based interface for accessing search results while avoiding buffering the result set, which can be very large.

To use the reader, instantiate JSONResultsReader on a search result json stream, as shown here:

reader = JSONResultsReader(result_stream)
for item in reader:
    print(item)
print "Results are a preview: %s" % reader.is_preview

This method requires that the search result have the query parameter output_mode set to json.

The 'ResultsReader' class, which supported the XML results stream, has been deprecated.

The JSONResultsReader class returns dictionaries and Splunk Enterprise messages from a JSON results stream. If the is_preview field is set to True, the results are a preview from a running search. Otherwise, the results are from a completed search. The Message class represents the informational messages that Splunk interleaves in the results stream. A Message object is comprised of two arguments, a string that gives the message (for instance, "DEBUG"), and a string that gives the message itself.

 Code examples

This section provides examples of how to use the job APIs, assuming you first connect to a Splunk Enterprise instance.

 To display results without a reader

You can read the results as a stream of XML with no parsing, but this method requires you to create an XML parser for the data from scratch:

# A blocking search
job = jobs.create("search * | head 100", **{"exec_mode": "blocking"})
print "...done!\n"

print "Search results:\n"

# Prints raw XML stream to the console
result_stream = job.results()
print result_stream.read()

If you don't want to create your own parser, a better method is described in the next section.

 To display results using a results reader

The Splunk Enterprise SDK for Python contains its own results reader that parses and formats results for you. Use the JSONResultsReader class for JSON format.

# A blocking search
job = jobs.create("search * | head 100", **{"exec_mode": "blocking"})
print "...done!\n"
print "Search results:\n"
# Prints a parsed, formatted XML stream to the console
result_stream = job.results(output_mode='json')
...
Show More

Query parameter output_mode must be set to json for the reader to parse the results.

 To paginate through a large set of results

The maximum number of results you can retrieve at a time from your search results is determined by the maxresultrows field, which is specified in a Splunk Enterprise configuration file. As a best practice, maintain the default value of 50,000. If your job has more results than this limit, retrieve your results in sets (0-49999, then 50000-99999, and so on), using the count and offset parameters to define how many results to retrieve at a time. Set the count (the number of results in a set) to maxresultrows (or a smaller value), and increment the offset by this same value to page through each set.

The following example shows how to determine the maximum number of results your system is configured to return:

# Find out how many results your system is configured to return
maxresultrows = service.confs["limits"]["restapi"]["maxresultrows"]
print "Your system is configured to return a maximum of %s results" % maxresultrows

The following example continues shows how to retrieve JSON search results in sets. This example uses a count of 10, and then displays the results as JSON objects using the JSONResultsReader class:

import splunklib.results as results
...
# A blocking search
job = jobs.create("search * | head 100", **{"exec_mode": "blocking"})
print "...done!\n"
...
Show More

 To display preview results

You can display a preview of the results of an in-progress search by using the Job.preview method. Job.preview returns, in a raw form, any results that have been generated so far from the server. You can then use a JSONResultsReader to read the results, converting them into objects over which you can iterate.

For instance, the following code creates a new job with preview results enabled:

import splunklib.client as client
import splunklib.results as results

service = client.connect(...)

job = service.jobs.create("search * | head 2", preview=True)

To determine whether preview has been successfully enabled, check for the following:

job['isPreviewEnabled'] == '1'

To access the preview results, pass the handle returned by Job.preview with query parameter output_mode='json' to a new JSONResultsReader, and then iterate through the reader:

rr = results.ResultsReader(job.preview(output_mode='json'))
for result in rr:
    if isinstance(result, results.Message):
        # Diagnostic messages may be returned in the results
        print '%s: %s' % (result.type, result.message)
    elif isinstance(result, dict):
        # Normal events are returned as dicts
        print result
...
Show More

You can also enable preview on an existing job by calling Job.enable_preview. Because enable_preview can take some time to propagate in the system, you can wait for it using the following code:

job.enable_preview()
while job['isPreviewEnabled'] == 0:
    sleep(0.2)

 To work with results from an export search

Working with search results from export searches differs from working with regular searches in the following ways:

• A reporting (transforming) search returns a set of previews followed by the final events, each as separate elements.

• A non-reporting (non-transforming) search returns events as they are read from the index, each as separate elements.

• A real-time search returns multiple sets of previews, each preview as a separate element.

• For JSON output, each result set is not returned as a single JSON object, but rather each row is an individual object, where rows are separated by a new line and the last row of the set is indicated by "lastrow":true.

  Here's sample JSON output that shows two results sets, each with five rows:

{"preview":true,"offset":0,"result":{"sourcetype":"eventgen-2","count":"58509"}}
{"preview":true,"offset":1,"result":{"sourcetype":"splunk_web_service","count":"119"}}
{"preview":true,"offset":2,"result":{"sourcetype":"splunkd","count":"4153"}}
{"preview":true,"offset":3,"result":{"sourcetype":"splunkd_access","count":"12"}}
{"preview":true,"offset":4,"lastrow":true,"result":{"sourcetype":"splunkd_stderr","count":"2"}}
{"preview":true,"offset":0,"result":{"sourcetype":"eventgen-2","count":"60886"}}
{"preview":true,"offset":1,"result":{"sourcetype":"splunk_web_service","count":"119"}}
{"preview":true,"offset":2,"result":{"sourcetype":"splunkd","count":"4280"}}
{"preview":true,"offset":3,"result":{"sourcetype":"splunkd_access","count":"12"}}
{"preview":true,"offset":4,"lastrow":true,"result":{"sourcetype":"splunkd_stderr","count":"2"}}

This format allows results to be sent as a continuous stream of JSON data that is easy to parse.

The Splunk Enterprise SDK for Python now includes a JSON parser.

As a best practice, use the SDK's JSON results reader to parse the output. Return the results stream in JSON, and use the JSONResultsReader class to parse and format the results.

 Sample output in different formats

The following is sample output in different formats for the search "search index=_internal | head 1":

***** ATOM *****
<?xml version="1.0" encoding="UTF-8"?>
<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . .-->
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="https://www.w3.org/2005/Atom" xmlns:s="https://dev.splunk.com/ns/rest">
  <title>Search Results</title>
  <id>/services/search/jobs/1378940275.16</id>
...
Show More

 Results reader comparison

Here's a single search result. For comparison, the output is displayed in its raw output form, and then SDK's XML results reader:

***** Output from printing raw results *****
<?xml version='1.0' encoding='UTF-8'?>
<results preview='0'>
<meta>
<fieldOrder>
<field>_bkt</field>
<field>_cd</field>
...
Show More

 Parameters for event, results, and results preview

Set these parameters when calling the following methods:

• Job.events
• Job.results
• Job.preview

For more, see the POST search/jobs endpoint in the Splunk Enterprise REST API Reference Manual.