Splunk SDK for Python Dashboard application

Customers often ask about using third-party tools and services with Splunk data. So, we came up with this application that uses the Splunk SDK for Python and demonstrates how to use a Leftronic dashboard to show real-time Twitter data that we index using Splunk.

What you need

You'll need a few things before you can get this application running:

• The Splunk SDK for Python (along with Splunk and Python, of course). See the Requirements page for details.
• A Twitter account to get live Twitter data.
• A Leftronic account to build a Leftronic dashboard. You provide an email address, and you get one dashboard for free.

How it works

The basic process components to the Dashboard application are:

• Getting data into Splunk. We use real-time Twitter data as a source and index it by running the Twitted example in the Splunk SDK for Python.
• Pushing data to Leftronic. This Dashboard application runs a handful of different real-time searches on the indexed Twitter data, transforms the results into a format that Leftronic understands, then pushes the output to your Leftronic dashboard.
• Displaying real-time data in a Leftronic dashboard. You'll need to build your own Leftronic dashboard to receive Splunk data and display the real-time results.

About the real-time searches

We use a variety of searches to get Twitter data out of Splunk and into Leftronic. Here are the real-time searches we run.

Top sources of tweets

We get the top sources (such as web client and mobile devices) of tweets from the last five minutes:

def top_sources(service):
    query = "search index=twitter status_source=* | stats count(status_source) as count by status_source | sort -count | head 5"
    created_job = service.jobs.create(query, search_mode="realtime", earliest_time="rt-5m", latest_time="rt")

Geo-location of tweets

We get all the tweets with geo-location enabled and their coordinates:

geo(service):
    query = "search index=twitter coordinates_type=Point coordinates_coordinates=* | fields coordinates_coordinates"
    created_job = service.jobs.create(query, search_mode="realtime", earliest_time="rt-5m", latest_time="rt")

Most recent tweets

We get the most recent tweets from the last five minutes:

def tweets(service):
    query = "search index=twitter | head 15 | fields user_name, user_screen_name, text, user_profile_image_url "
    created_job = service.jobs.create(query, search_mode="realtime", earliest_time="rt-5m", latest_time="rt")

User and tweet counts

We run a single search to get two statistics—the number of distinct users who tweeted in the last five minutes and the total number of tweets:

def counts(service):    
    query = "search index=twitter | stats count by user_id | fields user_id, count | stats count(user_id) as user_count, sum(count) as tweet_count"
    created_job = service.jobs.create(query, search_mode="realtime", earliest_time="rt-5m", latest_time="rt")

Top tags

We get a list of the top hashtags from the last five minutes:

def top_tags(service):
    query = 'search index=twitter text=* | rex field=text max_match=1000 "#(?<tag>\w{1,})" | fields tag | mvexpand tag | top 5 tag'
    created_job = service.jobs.create(query, search_mode="realtime", earliest_time="rt-5m", latest_time="rt")

How we send data to Leftronic

We poll these searches for new results in a loop. Then, we transform the results and submit the output to Leftronic using the Leftronic API. For more about the Leftronic API and the data format it requires, see the Leftronic API page.

def send_data(access_key, stream_name, point = None, command = None):
    data = {
        "accessKey": access_key,
        "streamName": stream_name
    }
    
    if not point is None:
        data["point"] = point
    if not command is None:
        data["command"] = command   
    request = urllib2.Request("https://beta.leftronic.com/customSend/",
        data = json.dumps(data)
    )
    response = urllib2.urlopen(request)

Put it all together and run the Dashboard application

Assuming you have everything that we said you'd need, here's how to run the Dashboard application.

Build your Leftronic dashboard

You'll need to build a Leftronic dashboard by adding a custom data source for each Splunk search you want to display.

1. Go to the Leftronic website and log in.
2. Under Data Sources, click My Custom Data.
3. From the My Custom Data tab, select widgets for your dashboard.
4. For each data source widget, click the Wrench icon to specify a stream name and a title. (The stream names are defined in our Dashboard application's source file, feed.py.)

For example, the Leftronic dashboard in the image above uses these widgets, titles, and streams:

Widget	Title	Stream name
Custom Leaderboard	Top sources	top_sources
Custom Geo	Tweet locations	geo
Custom Text	Tweets	tweets
Custom Text	User count (last 5 min)	users_count_5m
Custom Text	Tweet count (last 5 min)	tweets_count_5m
Custom Leaderboard	Top tags	top_tags

Start indexing Twitter data

Run the Twitted example to index Twitter data:

1. Install the Twitted Splunk app by copying the /splunk-sdk-python/examples/twitted/twitted directory to $SPLUNKHOME/etc/apps.
2. Restart Splunk if it's running (or start it if it's not)—go to $SPLUNKHOME/bin, then enter:

./splunk restart

3. Run the example—open a command prompt in the /splunk-sdk-python/examples/twitted/ directory and enter the following command (you'll be prompted to log into Twitter):

python input.py

You should immediately see incoming Twitter data in the console.

Run the Dashboard application

The Dashboard application is located in the /splunk-sdk-python/examples/dashboard directory. Before you can run it, you need to edit the source file (feed.py) with your Leftronic access key.

1. To get your Leftronic access key, go to https://www.leftronic.com/api/ and log in if you haven't already. Your access code is displayed under Overview—copy it.
2. Open /splunk-sdk-python/examples/dashboard/feed.py, and find this line:

leftronic_access_key = ""

3. Paste your own access key within the quotes, and save the file.
4. Run the application—open a command prompt in the /splunk-sdk-python/examples/dashboard directory and enter the following command:

python feed.py

5. Go back to your Leftronic dashboard—your dashboard should quickly start to display Twitter stats.