REST API Example Application: Social Splunk

Developer: Nick Key

What does my application do?

I am introducing a social element into Splunk. The application is a YouTube client that makes HTTP calls to YouTube REST API server to gather relevant search results based on a given search string. Imagine searching for YouTube videos within Splunk and being able to play the videos.

The search result, in the form of ATOM XML feed, which contains a list of videos with important metadata such as title, author, date uploaded and view count are parsed into key-value pairs using a custom adaptor. The processed metadata are then indexed into Splunk through TCP connection and at the same time rendered in the browser using a custom controller.

What rest end points am I hitting:

I'm hitting three REST endpoints:

  • /services/auth/login to get an auth token
  • /services/search/jobs to create a search job
  • /services/search/jobs/<search_id> to get search status

How am I calling them:

There are basically four basic steps I use to get the search results from Splunk REST API.

The example is as shown below:


        import httplib2
        from xml.dom import minidom

        baseurl = 'https://localhost:8089'
        userName = '<splunk_username>'
        password = '<splunk_password>'

        myhttp = httplib2.Http()

        # Step 1: Get an auth token
        serverContent = myhttp.request(baseurl + '/services/auth/login', 'POST', 
            headers={}, body=urllib.urlencode({'username':userName, 'password':password}))[1]
        sessionKey = minidom.parseString(serverContent).getElementsByTagName('sessionKey')[0].childNodes[0].nodeValue
        #print "============>auth token:  %s  <============" % sessionKey
        
        # Step 2: Create a search job
        searchJob = myhttp.request(baseurl + '/services/search/jobs','POST',
            headers={'Authorization': 'Splunk %s' % sessionKey},body=urllib.urlencode({'search': searchQuery}))[1]
        sid = minidom.parseString(searchJob).getElementsByTagName('sid')[0].childNodes[0].nodeValue
        #print "============>sid:  %s  <============" % sid
        
        # Step 3: Get search status
        myhttp.add_credentials(userName, password)
        services_search_status_str = '/services/search/jobs/%s/' % sid
        isNotDone = True
        while isNotDone:
            searchStatus = myhttp.request(baseurl + services_search_status_str, 'GET')[1]
            isDoneStatus = re.compile('isDone">(0 1)')
            isDoneStatus = isDoneStatus.search(searchStatus).groups()[0]
            if (isDoneStatus == '1'):
                isNotDone = False
        #print "============>search status:  %s  <============" % isDoneStatus
        
        # Step 4: Get search results
        services_search_results_str = '/services/search/jobs/%s/results?output_mode=json&count=0' % sid
        searchResults = myhttp.request(baseurl + services_search_results_str, 'GET')[1]
        #print "============>length: %s  <============" % len(searchResults)
        #print "============>search result:  [%s]  <============" % searchResults

What do I get back:

Each step shown in the code snippet returns some sort of result such as the session key, search ID, search status and search result

What do I use to parse the results I get back:

Splunk returns the search result in three formats =96 xml, csv and json. I specifically choose to have all the results returned in json format using output_mode=json and count=0. The arguments are shown as below:

/services/search/jobs/%s/results?output_mode=json&count=0

Any problems encountered:

It is very simple to utilize Splunk REST API. There are many endpoints available for use but the example with the three REST endpoints are good enough to assist a developer to get started.