How to run searches and jobs using the Splunk SDK for C# v2.x

With the Splunk SDK for C#, you can add Splunk Enterprise search capabilities to your program. The Splunk SDK for C# supports multiple search types, some of which create search jobs:

  • Searches that produce search jobs and stream the results:
    • Normal search: A normal search runs asynchronously. It returns a search job immediately, but you retrieve the results when the search has finished. Poll the job to determine its status. You can also preview the results if "preview" is enabled.
    • Blocking search: A blocking search runs synchronously. It does not return a search job until the search has finished, so there is no need to poll for status. Blocking searches don't work with real-time data.
    • Real-time search: A real-time search runs asynchronously and immediately. It returns a search job immediately and searches live events as they stream into Splunk for indexing. The events that are returned match your criteria within a specified time range window.
  • Searches that only stream the results:
    • One-shot search: A one-shot search runs synchronously and immediately. Instead of returning a search job, this mode returns the results of the search once completed.
    • Export search: An export search runs asynchronously and immediately, does not create a job for the search, and starts streaming results right away. This search is useful when you're dealing with large amounts of data, and is ideal for real-time data. Export searches always have preview results enabled.

Normal searches

A normal search is an asynchronous search that returns a search job immediately. (In this context, "asynchronous" describes the search operation on the Splunk Enterprise server, and indicates that search results are returned to your client app as they are obtained.) You have two choices for performing normal searches: a pull model (using the results stream with an IEnumerable) and a push model (subscribing to search results with an Observer). In both cases, you must first create a SearchResultStream object to represent the stream of search result records (SearchResult objects).

The pull model for normal searches doesn't require Rx. This example uses a foreach loop to iterate through the search results it retrieves, and prints each record to the console:

Job job = await service.Jobs.CreateAsync("search index=_internal | head 10");
SearchResultStream stream;

using (stream = await job.GetSearchResultsAsync())
{
    try
    {
        foreach (SearchResult result in stream)
        {
            Console.WriteLine(string.Format("{0:D8}: {1}", stream.ReadCount, result));
        }
                  
        Console.WriteLine("End of search results");
    }
    catch (Exception e)
    {
        Console.WriteLine(string.Format("SearchResults error: {0}", e.Message));
    }
}

The Splunk SDK for C# defines the Observable<T> class to provide push-based notifications to observers. This example uses a subscription to observe the collection of search result records, and prints each record to the console.

job = await service.Jobs.CreateAsync("search index=_internal | head 10");

using (stream = await job.GetSearchResultsAsync())
{
    stream.Subscribe(new Observer<SearchResult>(
        onNext: (result) =>
        {
            Console.WriteLine(string.Format("{0:D8}: {1}", stream.ReadCount, result));
        },
        onError: (e) =>
        {
            Console.WriteLine(string.Format("SearchResults error: {0}", e.Message));
            manualResetEvent.Set();
        },
        onCompleted: () =>
        {
            Console.WriteLine("End of search results");
            manualResetEvent.Set();
        }));

    await stream; // wait for stream to complete
}

Reference

Blocking searches

Running a blocking search creates a search job and runs the search synchronously. The job is returned after the search has finished and all the results are in.

When you create a search job, set the parameters of the job as an argument dictionary of key-value pairs. For a list of all the possible parameters, see Search job parameters.

// log in to Splunk Enterprise
await service.LoginAsync("admin", "changeme");

// create results stream and job objects
SearchResultStream searchResultStream;
Job job;

Console.WriteLine("Blocking search");

// create a job, setting the execution mode to "blocking"
job = await service.Jobs.CreateAsync("search index=_internal | head 10", 
                                   new JobArgs() { ExecutionMode = ExecutionMode.Blocking });

// print search results
using (searchResultStream = await job.GetSearchResultsAsync())
{
    foreach (var result in searchResultStream.ToEnumerable())
    {
        Console.WriteLine(result);
    }
}

Reference

Real-time searches

Real-time searches return live events as they are indexed, and this type of search continues to run as events continue to arrive. So, to view results from a real-time search, you must view the preview results. You can think of the previews as a snapshot of the search results at that moment in time. A few search job parameters are required to run a real-time search, which you can set by using the methods of the JobArgs class:

  • Set the execution mode (ExecutionMode) to Normal.
  • Set the search mode (SearchMode) to RealTime, which also enables previews.
  • If you want to specify a sliding window for your search (for example, everything in the last hour), you can set the earliest and latest times to relative time modifiers. For example, set EarliestTime to rt-1h and LatestTime to rt and the time range is continuously updated based on the current time.

The real-time search continues to run until you either cancel or finalize it. If you cancel the search, the search job is deleted. If you finalize it, the search is stopped and the search job is completed.

The following code sample is taken from the search-realtime example in the examples directory. Two things are of note here:

  • Because the search is real-time, we've built in a mechanism for canceling the search.
  • With real-time searches, you view search preview results, hence our use of the Job.GetSearchPreviewAsync() method.

The following example shows a real-time search of your internal index.

static void Main(string[] args)
{
    // create Service object
    using (var service = new Service(SdkHelper.Splunk.Scheme, SdkHelper.Splunk.Host, SdkHelper.Splunk.Port, new Namespace(user: "nobody", app: "search")))
    {
        Task task = Run(service);

        while (!task.IsCanceled)
        {
            Task.Delay(500).Wait();
        }
    }

    Console.Write("Press return to exit: ");
    Console.ReadLine();
}

static async Task Run(Service service)
{
    // log in to Splunk Enterprise
    await service.LoginAsync(SdkHelper.Splunk.Username, SdkHelper.Splunk.Password);
    Console.WriteLine("Press return to cancel.");

    // search query
    string searchQuery = "search index=_internal | stats count by method";

    // create job based on search query and some additional args
    Job realtimeJob = await service.Jobs.CreateAsync(searchQuery, new JobArgs
    {
        SearchMode = SearchMode.RealTime,
        EarliestTime = "rt-1h",
        LatestTime = "rt",
    });

    // cancellation mechanism
    var tokenSource = new CancellationTokenSource();

    Task.Run(async () =>
    {
        Console.ReadLine();

        await realtimeJob.CancelAsync();
        tokenSource.Cancel();
    });
    // keep displaying results if user hasn't canceled
    while (!tokenSource.IsCancellationRequested)
    {
        // view results previews with real-time searches
        using (SearchResultStream stream = await realtimeJob.GetSearchPreviewAsync())
        {
            Console.WriteLine("fieldnames: " + string.Join(";", stream.FieldNames));
            Console.WriteLine("fieldname count: " + stream.FieldNames.Count);
            Console.WriteLine("final result: " + stream.IsFinal);

            foreach (SearchResult result in stream)
            {
                Console.WriteLine(result);
            }

            Console.WriteLine("");
            await Task.Delay(2000, tokenSource.Token);
        }
    }
}

Reference

One-shot searches

Unlike other searches, the one-shot search does not create a search job, so you can't access it using the job-related classes. Instead, use the Service.SearchOneShotAsync() method. Though this method is asynchronous, be aware that a one-shot search is a synchronous operation on the Splunk Enterprise server. That is, it will only return results once the search has finished.

To set properties for the search (for example, to specify a time range to search), you'll need to create an argument map (using a JobArgs object) with the parameter key-value pairs. Some common parameters are:

  • EarliestTime: Specifies the earliest time in the time range to search. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string.
  • LatestTime: Specifies the latest time in the time range to search. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string.
  • rf: Specifies one or more fields to add to the search.

For a full list of possible properties, see the list of Search job parameters, although most of these parameters don't apply to a one-shot search.

This example runs a one-shot search within a specified time range and displays the results in XML.

using (SearchResultStream searchResultStream = 
    await service.SearchOneshotAsync(
        "search index=_internal | head 10",
        new JobArgs
        {
            EarliestTime = "2014-06-19T12:00:00.000-07:00",
            LatestTime = "2014-06-20T12:00:00.000-07:00",
        }))
{
    foreach (var result in searchResultStream.ToEnumerable())
    {
        Console.WriteLine(result);
    }
}

Reference

Export searches

An export search runs immediately, does not create a job for the search, and starts streaming results right away. This search is useful for exporting large amounts of data from Splunk Enterprise. Export searches have two modes: reporting and non-reporting.

  • A reporting search aggregates events from the index and streams real-time previews followed by a final result set when the search completes.
  • A non-reporting search passes events, potentially with modifications to each one, directly from the index and sends back final results in chunks as it gets them from the index.

For reporting mode export searches, use the ExportSearchPreviewsAsync() method of the Service object. For non-reporting mode export searches, use the ExportSearchResultsAsync() method.

To access preview results, first create a SearchPreviewStream object, which is a streaming XML reader that reads SearchPreview objects. Then use the Service.ExportSearchPreviewsAsync() method to export an observable sequence of search result previews to the reader. In this example, the fields of each preview search event are listed (the SearchPreview.Results property), along with whether each event is final or partial (the SearchPreview.IsFinal property):

// create streaming reader
SearchPreviewStream searchPreviewStream;

// export the search result previews
using (searchPreviewStream = service.ExportSearchPreviewsAsync("search index=_internal | head 100").Result)
{
    int previewNumber = 0;

    // enumerate through each search result preview
    foreach (var searchPreview in searchPreviewStream.ToEnumerable())
    {
        Console.WriteLine("Preview {0:D8}: {1}", ++previewNumber, searchPreview.IsFinal ? "final" : "partial");
        int recordNumber = 0;

        foreach (var result in searchPreview.Results)
        {
            Console.WriteLine(string.Format("{0:D8}: {1}", ++recordNumber, result));
        }
    }
}

To access the final results of an export search, simply use the final results versions of the same APIs, as noted here:

Export Preview Search Results API Export Search Results API Description

Service.ExportSearchPreviewsAsync() method

Service.ExportSearchResultsAsync() method

Asynchronously exports an observable sequence of search result previews or search results.

SearchPreviewStream class

SearchExportStreamclass

A streaming XML reader.

SearchPreview class

SearchResult class

A single record on a stream.

Reference

 

Search job parameters

Properties to set

The parameters you can use for search jobs correspond to the parameters for the search/jobs endpoint in the REST API.

This list summarizes the properties you can set for a search job (click here for properties you can retrieve).

Parameter
Description
searchRequired. A string that contains the search query.
auto_cancelThe number of seconds of inactivity after which to automatically cancel a job. 0 means never auto-cancel.
auto_finalize_ecThe number of events to process after which to auto-finalize the search. 0 means no limit.
auto_pauseThe number of seconds of inactivity after which to automatically pause a job. 0 means never auto-pause.
earliest_timeA time string that specifies the earliest time in the time range to search. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string. For a real-time search, specify "rt".
enable_lookupsA Boolean that indicates whether to apply lookups to events.
exec_modeAn enum value that indicates the search mode ("blocking", "oneshot", or "normal").
force_bundle_replicationA Boolean that indicates whether this search should cause (and wait depending on the value of "sync_bundle_replication") bundle synchronization with all search peers.
idA string that contains a search ID. If unspecified, a random ID is generated.
index_earliestA string that specifies the time for the earliest (inclusive) time bounds for the search, based on the index time bounds. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string.
index_latestA string that specifies the time for the latest (inclusive) time bounds for the search, based on the index time bounds. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string.
latest_timeA time string that specifies the latest time in the time range to search. The time string can be a UTC time (with fractional seconds), a relative time specifier (to now), or a formatted time string. For a real-time search, specify "rt".
max_countThe number of events that can be accessible in any given status bucket.
max_timeThe number of seconds to run this search before finalizing. Specify 0 to never finalize.
namespaceA string that contains the application namespace in which to restrict searches.
nowA time string that sets the absolute time used for any relative time specifier in the search.
reduce_freqThe number of seconds (frequency) to run the MapReduce reduce phase on accumulated map values.
reload_macrosA Boolean that indicates whether to reload macro definitions from the macros.conf configuration file.
remote_server_listA string that contains a comma-separated list of (possibly wildcarded) servers from which to pull raw events. This same server list is used in subsearches.
rfA string that adds one or more required fields to the search.
rt_blockingA Boolean that indicates whether the indexer blocks if the queue for this search is full. For real-time searches.
rt_indexfilterA Boolean that indicates whether the indexer pre-filters events. For real-time searches.
rt_maxblocksecsThe number of seconds indicating the maximum time to block. 0 means no limit. For real-time searches with "rt_blocking" set to "true".
rt_queue_sizeThe number indicating the queue size (in events) that the indexer should use for this search. For real-time searches.
search_listenerA string that registers a search state listener with the search. Use the format: search_state;results_condition;http_method;uri;
search_modeAn enum value that indicates the search mode ("normal" or "realtime"). If set to "realtime", searches live data. A real-time search is also specified by setting "earliest_time" and "latest_time" parameters to "rt", even if the search_mode is normal or is not set.
spawn_processA Boolean that indicates whether to run the search in a separate spawned process. Searches against indexes must run in a separate process.
status_bucketsThe maximum number of status buckets to generate, which corresponds to the size of the data structure used to store timeline information. This value is also used for summaries. A value of 0 means to not generate timeline or summary information.
sync_bundle_replicationA Boolean that indicates whether this search should wait for bundle replication to complete.
time_formatA string that specifies the format to use to convert a formatted time string from {start,end}_time into UTC seconds.
timeoutThe number of seconds to keep this search after processing has stopped.
 
Properties to retrieve

This list summarizes the properties that are available for an existing search job:

Property
Description
cursorTimeThe earliest time from which no events are later scanned.
delegateFor saved searches, specifies jobs that were started by the user.
diskUsageThe total amount of disk space used, in bytes.
dispatchStateThe state of the search. Can be any of QUEUED, PARSING, RUNNING, PAUSED, FINALIZING, FAILED, DONE.
doneProgressA number between 0 and 1.0 that indicates the approximate progress of the search.
dropCountFor real-time searches, the number of possible events that were dropped due to the "rt_queue_size".
eai:aclThe access control list for this job.
eventAvailableCountThe number of events that are available for export.
eventCountThe number of events returned by the search.
eventFieldCountThe number of fields found in the search results.
eventIsStreamingA Boolean that indicates whether the events of this search are being streamed.
eventIsTruncatedA Boolean that indicates whether events of the search have not been stored.
eventSearchSubset of the entire search before any transforming commands.
eventSortingA Boolean that indicates whether the events of this search are sorted, and in which order ("asc" for ascending, "desc" for descending, and "none" for not sorted).
isDoneA Boolean that indicates whether the search has finished.
isFailedA Boolean that indicates whether there was a fatal error executing the search (for example, if the search string syntax was invalid).
isFinalizedA Boolean that indicates whether the search was finalized (stopped before completion).
isPausedA Boolean that indicates whether the search has been paused.
isPreviewEnabledA Boolean that indicates whether previews are enabled.
isRealTimeSearchA Boolean that indicates whether the search is a real time search.
isRemoteTimelineA Boolean that indicates whether the remote timeline feature is enabled.
isSavedA Boolean that indicates whether the search is saved indefinitely.
isSavedSearchA Boolean that indicates whether this is a saved search run using the scheduler.
isZombieA Boolean that indicates whether the process running the search is dead, but with the search not finished.
keywordsAll positive keywords used by this search. A positive keyword is a keyword that is not in a NOT clause.
labelA custom name created for this search.
messagesErrors and debug messages.
numPreviewsNumber of previews that have been generated so far for this search job.
performanceA representation of the execution costs.
priorityAn integer between 0-10 that indicates the search's priority.
remoteSearchThe search string that is sent to every search peer.
reportSearchIf reporting commands are used, the reporting search.
requestGET arguments that the search sends to splunkd.
resultCountThe total number of results returned by the search, after any transforming commands have been applied (such as stats or top).
resultIsStreamingA Boolean that indicates whether the final results of the search are available using streaming (for example, no transforming operations).
resultPreviewCountThe number of result rows in the latest preview results.
runDurationA number specifying the time, in seconds, that the search took to complete.
scanCountThe number of events that are scanned or read off disk.
searchEarliestTimeThe earliest time for a search, as specified in the search command rather than the "earliestTime" parameter. It does not snap to the indexed data time bounds for all-time searches (as "earliestTime" and "latestTime" do).
searchLatestTimeThe latest time for a search, as specified in the search command rather than the "latestTime" parameter. It does not snap to the indexed data time bounds for all-time searches (as "earliestTime" and "latestTime" do).
searchProvidersA list of all the search peers that were contacted.
sidThe search ID number.
ttlThe time to live, or time before the search job expires after it has finished.