Hey everyone, just wanted to let you know that a preview release of Splunk just left the docks.

http://www.splunk.com/index.php/preview

I want to introduce to you one the latest features for Windows Splunk - the monitoring of Windows registry in real time for activity/events, and the indexing and searching these events with Splunk.

While working on this we had a few challenges:

First, there aren’t any published win32 APIs that does this in user mode. The best that you can do with win32 API is to poll the registry for certain registry key/hives, and you’ll be notified when if the key or subkey of the hive has been changed. Even when you get a notification for a change, you will not be told which key exactly has changed, you’ll have to figure that out yourself .

Second, scalability. You can’t possibly poll all of the registry in user mode for changes. There are simply too many keys to query.

The solution is to write a device driver that hooks to the kernel and intercepts all registry events. The driver bubbles up the events to the user mode for filtering and tagging, and finally pipe them to Splunk for indexing. Obviously, this driver needs to be very stable and reliable, needs to scale to the point where if you want to monitor all of the events in the registry, and it should be able to handle the load.

With this preview release we launched the first version of the splunk-regmon tool. The tool writes events to standard output, and using Splunk’s ExecProcessor(popen). Splunk is able to get these events and send them through the indexing pipeline. A basic filtering is in place, hard coded for now to only monitor registry events related to changes - i.e. Create, Delete, Set, etc. Create type events

Hey all,

It’s taken longer than we would have liked but our 3rd preview build has been posted.
Get’um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don’t even think of trying it out for real - and there is no guarantee that migration will work from a preview to GA ( we will migrate from 3.1.x to GA but not preview ). Also, don’t run splunk as root - its just not good to do until we run through all our testing.

As always, please send us feedback at splunkpreview@splunk.com or hit us up on IRC (irc.efnet.org #splunk).
The last round of info from Preview #2 was awesome please keep it up!

e.

Happy new year (bit early) all dev.splunk.com readers….
We have just posted our second 3.2 preview release. (build number 30455)

Its packed with holiday goodness, albeit very raw.

First you will notice we have posted a windows build. Its been in the cooker since last Feb and thanks to Mitch, Ledio, Igor and a bit of Amrit we now have a single code base that rocks on linux, mac, solaris, freebsd, aix, AND windows. This was not an easy feat as evidenced by our gift of a pony (soft and electronic) to Mitch for his effort. Its still very raw (the build not the pony), and has a tendency to crash because of a memory fragentation and limited vm space. Which will be fixed by GA… MarkB. will post more on the build so stay tuned for details. Its a big deal for us so be patient and we sure could use feedback on how to make it the best it can be.

Also in this release you will see the UI starts to get some of the async search results. Over the next few releases we will be moving to fully async search in the UI. It will take a few turns but this preview has some of the first cut.

There are a bunch of other improvements; scheduled searches got a bit of a cleanup in the UI and the backend has been improved as well. Performance, bugs, and other tweaks are also spread throughout. I’ll get others to post specifics.

In the mean time, as always its a huge help to us in dev if you can kick the tires before we freeze for GA. Please send feedback to splunkpreview@splunk.com, post comments to this blog, or drop by and tell us in person.

Again, thanks for the help

In the Preview release, we have search-time discovery of simple transactions, with the new transam search command, in honor of one of our developers that hasn’t quite moved on from high school. Transam collapses a set of events that belong to a transaction into a single event. You can specify the parameters as arguments to the transam operator right in the search, or you can refer to a named-transaction definition in transactions.conf. A few simple examples will give you an idea of some things you can do.

• get events with ‘http’, and group any search results into “bursts” of events, grouping any events that occur within two seconds of each other into the same transaction event. [Note: there is an implied “search” command at the head of all searches, so “http” is really “search http”.]

http | transam maxpause=2s

• get events with ‘http’, and collapse those that share the same host and cookie value, that occur within 30 seconds:

http | transam fields=host,cookie maxspan=30s maxpause=30s

• get events with ’sendmail’, and collapse those that have the same userid, between a login and a logout, that occur within 10 minutes:

sendmail | transam fields=uid startswith="eventtype=login" endswith="eventtype=logout" maxspan=10m maxpause=10m

• get events with ‘http’, and then find transactions as defined by email_transaction found in transactions.conf:

http | transam email_transaction

We use Perforce at Splunk, and it’s worked out pretty well for us. I’m a CVS admin at heart, and I know there’s some SVN sentiment, but p4 gives us a nice mix of atomic commits, attractive GUI and command-line tools, and someone to call for help if it ever completely eats itself.

Over time I’ve compiled a small library of scripts for various p4 functions that have been written time and again at different sites…mergetool is one of them. This little tool accepts a merge target (”yours” in p4-speak) and projectile (”theirs” in p4), labels both, performs an integrate, and performs a “safe” resolve -as. It logs any failures for you to resolve by hand, or submits the change set if the resolve completes successfully. It does this with a bunch of logging in a well-organized, date-stamped directory suitable for archiving (or splunking).

Splunk runs on a lot of platforms for a relatively young product and that number is always increasing. The day I started, there were packages for Intel and PowerPC Macintoshes, i686 Linux, Solaris 8 on Sparc, and FreeBSD on x86, all created with BitRock InstallBuilder, run from a simple shell script, usually by Erik. There really wasn’t much control over what went into the installer — if a file was in the installer prep directory and the shell script didn’t know to delete it, out it went.

By the time 2.1 was on its way, we’d decided to switch to native packages, and our list of platforms had expanded to include Solaris on Intel, with several more on the horizon. We also wanted to provide the “rail tarball” distribution we continue to support, in part so that QA could get started before the packaging automation was complete.

What is that packaging automation, you might ask? Obviously writing custom code to package each platform (not to mention spec or pkgmap files in each platform’s native format) was not a very maintainable solution. Instead we use a locally modified version of Easy Software’s EPM package manager. After a little work, EPM lets us use a common set of list files to create relocatable packages using common pre- and post-install scripts across all of the 9 platforms we now build on. We’re able to control every file and permission that goes into the packages, and in most cases we can add packaging for a new OS platform with a minimum of work (for something very different we haven’t previously had in house, like AIX, more time might need to be spent cleaning up EPM’s support for the platform). We’ve piggy-backed creation of the “rail tarball” distributions on the EPM list file structure, so

Hi! My name is Kim, and I’m the release engineer here at Splunk.

Thanks to my acquisition-happy former employer, Symantec, I’ve seen a variety of startup approaches to release engineering. Most frequently it seems some senior developer has a bug up you-know-where about how the build system should work, and some poor junior developer or sysadmin type person dutifully does the drudge work (usually by hand). At other sites, some very diligent and detail-oriented person creates and executes a process with a great deal of record-keeping and attention to detail but often not a lot of automation. Consistency across different build platforms usually isn’t a strong point.

Here at Splunk, things are a bit different. I called myself the plumber in the title of this post because that’s how I see my job: I create and maintain the plumbing that produces consistent, reproducible Splunk builds across all of our platforms, with as much visibility as I can muster. I see my contribution more as enforcing process through tools — ideally, tools that enable process in a way that is more convenient for everyone than “doing it wrong” — rather than personally pushing all the buttons and scribbling in all the logbooks. And I’ve had the good fortune to come into a culture that encourages this approach.

Whew. That’s a mouthful for an introduction. In the near future I hope to write a bit more about how the plumbing works, and some neat tools I’ve found along the way. I’m sure y’all will be waiting with baited breath.