Splunk 3.2 introduces a publicly available Python client library that allows external developers to programmatically interact with Splunk by importing a few key modules.

The easiest way to get started with the client library is to get into Splunk’s Python environment. Locate your Splunk install directory (/opt/splunk by default), and start the python interactive shell that comes with Splunk:

# bin/splunk cmd python


This will launch the interactive Python prompt, which starts off looking like this:

Python 2.5.1 (r251:54863, Nov 18 2007, 16:13:41)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>


Starting a search

Import the Splunk modules:

import splunk.auth
import splunk.search as se


If you have installed Splunk with the default settings, then your hostpath is https://localhost:8089. The client library knows this default, so you can authenticate directly by providing a username and password:

key = splunk.auth.getSessionKey('admin','changeme')


The getSessionKey method automatically caches the session key in the current interactive session, so you don’t have to pass it along to subsequent methods. In a production implementation, or if you are connecting to multiple servers, you’ll need to keep track of separate session keys.

If your server is on a different hostname or port, then you need to first update the session defaults:

splunk.mergeHostPath('splunk_hostname:12000', True)
key = splunk.auth.getSessionKey('admin','changeme')


The mergeHostPath method takes host information in many different forms:

• hostname
• hostname:port
• https://hostname
• http://hostname:port

Next, start a search:

job = se.dispatch('search error')


This creates a search job handle object job and start a running search on the server for events that contain the term “error”. If you are connecting to multiple servers, then you’ll also need to provide hostPath and sessionKey parameters as well. This handle is keyed off of the search job ID that is generated by the server, and is available via:

job.id


With this ID, you can always use

XML is a great format for exchanging information because it balances readability, extensibility, and compatibility across heterogeneous environments. However, its flexibility is also a disadvantage because it is far too easy to create a proprietary XML schema, resulting in lots of custom code to interface with various systems. Lots of custom code leads to brittleness, and brittleness leads to frustration. The key to salvation lies in standardization.

Enter the Atom standard: a standards-track schema that defines a generic collection/item container format in XML. Most people equate Atom to an RSS competitor, which is true, but that only covers half of what it does. The Atom Publishing Protocol is a well-defined protocol for performing CRUD (Create, Read, Update, Delete) operations on items over HTTP. The Atom Syndication Format, which is the most commonly used portion, defines the XML schema used to deliver data during a Read operation. Atom was spearheaded by Sam Ruby, and is now back by people like Brad Fitzpatrick, Tim Bray, Jeremy Zawodny, Mark Pilgrim, and is heavily implemented by Google.

Like most software systems, the majority of Splunk’s internal entities can be loosely viewed as a collection of similar items. The requested searches, configuration information, saved searches, users, roles — all just collections. So instead of creating five separate XML schemas for each of these collections that perfectly describe their contents, I chose Atom to serve as a single generic container to describe all of the entities. This kind of reuse is echoed by Pat Helland of Amazon, who gives a great talk on relating the rise of the industrial age to standardization, and Tim Bray (Mr. XML himself), who advocates against creating your own XML unless absolutely necessary.

The benefit of sticking to a standard is that there is a much greater chance that external developers already know exactly

Splunk 3.2 is available for download! This release is one of our biggest so far, representing a tremendous amount of effort by our engineering team, and is a product that I’m proud to stand behind. As I mentioned in my last post about our push for the Splunk Platform, a central tenet is to make a compelling product that developers will not only understand, but also enjoy using. While Dr. LogLogic rambles on about how catering to developers sucks, we know that developers are a huge part of our user base (drop by the #splunk channel on EFNet sometime) and we will continue to make Splunk as flexible and extensible as possible.

With 3.2, we have begun moving some of Splunk’s core services over to a proper REST API. Now, for those of you who have already been using the REST API in 3.1, the new API in 3.2 and beyond is distinctly different, and is intended to replace any older versions. Therefore, the REST API of version 3.1 and before will now be referred to as the UI API, and the term “REST API” will refer to the new API that I’m covering in this post.

Before I dive into the details though, I’d like to clarify the usage of “REST” and what I mean when I speak of it. First of all, REST is not a protocol or standard. There is no RFC, or ISO specification on what constitutes REST; it is a philosophy about the relationship between entities in a software system and the interface to interact with those entities. Roy Fielding’s original thesis named it Representational State Transfer, which when put into practice means that URIs should convey meaning in a durable manner. In essence, REST emphasizes the “what” of a system rather than the “how”. In comparison,

Splunk is on track to become a billion-dollar company and you, the intrepid sysadmin/developer, are going to help us get there. Now, this is not a statement that I’m making as an analyst who “covers” the enterprise software market, and compiles a list of “top software companies to watch”. I’m writing this as Splunk’s Platform Architect, a techie whose goals are to ensure that what comes out of our development group is compelling and exciting to those that are actually working with the product.

It is this developer-centric ethos that sets us apart from so many of the other enterprise software firms and has already paid dividends on community goodwill. Instead of making prospective buyers jump through registration hoops just to view a guided webcast tour, Splunk provides fully functional software downloads to try out on your own data, inside your own network, free from webinar smoke and mirrors.

We don’t just want you to try out the software, we want you to try doing things that aren’t covered in our brochureware, things that sound ludicrous at first but are doable. In fact, in a perverse way, we hope that you do break our product because it reveals new limitations for us to solve, ultimately leading to a product that lets you do your job the way you want, yet easier and faster.

This is where the Splunk Platform comes into play. We want to increase the ubiquity of Splunk by, 1) exposing major components of Splunk as individual services, and, 2) allowing external developers to build on top of Splunk and leverage our award-winning IT search infrastructure. Starting with version 3.2 (you can download the preview version today), there is a new REST API that provides unprecedented access and consistency to every aspect of the Splunk Server. We are

The Splunk cozy has been to a few countries around the world. This month, I took it to the Galapagos, and decided to leave it there at Post Office Bay amongst all the other plaques and memorabilia. I think it’ll be very comfortable for a while. See the rest of my Galapagos photo gallery.

Adventures on a mini-bike amongst the boxes in engineering:

External view:

Internal view:

Dev destroys support in a 4 on 4 boat race.

For all you hardcore Web 2.0 fanboys, I’m giving a talk at AjaxWorld on “High-Performance AJAX Application Design” down in Santa Clara at the end of September. The official blurb is:

Designing an AJAX application that meets enterprise scalability and performance requirements presents technical challenges that aren’t addressed by traditional AJAX frameworks. This session will highlight the techniques used in Splunk to address handling large amounts of data in the browser, persistent multi-panel state management, interface customization and localization, and interactive DOM-accessible graphics support. By leveraging existing, though less common, techniques such as iframe-style AJAX, in-browser XSLT, and contextual CSS, modern browsers can provide a compelling interface without the need for a thick-client installation.

Come by and say hi.

On a daily basis, I pay homage to the wonder that is Blue Bottle Coffee espresso, which flows freely — some would say excessively — from our kitchen. The benefits to productivity that this fine coffee bestows upon the dev team is enormous, easily eclipsing other contenders such as video games or foosball. Of course, there were some hurdles to get to this point, namely somebody pouring M&Ms into the bean grinder of the super-automatic that was previously in service. The result was a pitiful molten mess of chocolate, beans, plastic, and gears. And, of course, the perpetrator was never discovered. So the only recourse was to beef up the machinery and move to a true commercial setup: a La Spaziale, Mazzer Mini, and freshly delivered Blue Bottle. BB even asked us what hardware we were running, and sent us the most compatible beans. Brilliant.

I’m pleased to announce that Splunk 3.0 has been released, and is available for download immediately! It’s been a very long road to GA, but I think it is worth the wait. With 3.0, exploring your unstructured data has never been easier, thanks to the new reporting interface. As always, we love user feedback so try it out and let us know what you like and what you don’t — either to me, or to support@splunk.com. Stop guessing about what’s going on in your datacenter and start getting answers with Splunk.