I’m looking for some help.
I’ve built a VMWare app for splunk and in the process of doing the same for Xen. These Apps use the VMWare and Xensource API’s to index everything about the VM environment. When combined with splunk instances running within the guest OS you get a very comprehensive historical picture. I’m curious are there any splunk customers out there using VMWare or Xen? I’m looking for usecases so that i better understand how to configure the apps. I’d be curious to know what types of information would be useful to capture and what types of searches would one want to perform. Both Xen and VMWare have so much data available that configuration could be complicated. I’m trying to narrow it down to several useful out of the box configurations. If your have any thoughts comment here or email me at erik at splunk dot com.

Thanks
e.

The following is copped from a support email by Stephen Sorkin who is the man behind the splunk server curtain … thought it should go broader.

I’m the manager of the search and indexing team at Splunk. We’re still in the process of writing up our findings from storage benchmarks but here are the general details.

High IO/s typically means both faster indexing in general and faster searching of rare, temporally incoherent events. On average, we’ve seen indexing speeds increase by about 66% going from an 7200 RPM SATA RAID to a 15K RPM SCSI RAID. We’ve seen comparable performance from SCSI and SAS RAIDs, provided they’re 15K RPM.

The best best benchmarking tool we’ve found for measuring how Splunk will behave on your disk hardware is bonnie++. If your disk subsystem can sustain 800 IO/s, you’re in good shape.

As far as searching goes, IO/s is the dominant factor for non-coherent, infrequently accessed search results. This means, if you’re just searching for the newest data, or even have to reach back through 1MM events to return 10k, the disk is NOT the bottleneck, since each individual read() will pull many events off disk. However, if you’re searching for a rare term, like a name, that occurs once an hour or once a day, each read() is going to require the drive arm move. If you’re using a 7200 RPM SATA drive, that’s about 100 IO/s and hence on the order of 100 retrieved events per second. If you have a decent RAID, that could be 800 retrieved events per second.

-s

Hey all,

It’s taken longer than we would have liked but our 3rd preview build has been posted.
Get’um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don’t even think of trying it out for real - and there is no guarantee that migration will work from a preview to GA ( we will migrate from 3.1.x to GA but not preview ). Also, don’t run splunk as root - its just not good to do until we run through all our testing.

As always, please send us feedback at splunkpreview@splunk.com or hit us up on IRC (irc.efnet.org #splunk).
The last round of info from Preview #2 was awesome please keep it up!

e.

Happy new year (bit early) all dev.splunk.com readers….
We have just posted our second 3.2 preview release. (build number 30455)

Its packed with holiday goodness, albeit very raw.

First you will notice we have posted a windows build. Its been in the cooker since last Feb and thanks to Mitch, Ledio, Igor and a bit of Amrit we now have a single code base that rocks on linux, mac, solaris, freebsd, aix, AND windows. This was not an easy feat as evidenced by our gift of a pony (soft and electronic) to Mitch for his effort. Its still very raw (the build not the pony), and has a tendency to crash because of a memory fragentation and limited vm space. Which will be fixed by GA… MarkB. will post more on the build so stay tuned for details. Its a big deal for us so be patient and we sure could use feedback on how to make it the best it can be.

Also in this release you will see the UI starts to get some of the async search results. Over the next few releases we will be moving to fully async search in the UI. It will take a few turns but this preview has some of the first cut.

There are a bunch of other improvements; scheduled searches got a bit of a cleanup in the UI and the backend has been improved as well. Performance, bugs, and other tweaks are also spread throughout. I’ll get others to post specifics.

In the mean time, as always its a huge help to us in dev if you can kick the tires before we freeze for GA. Please send feedback to splunkpreview@splunk.com, post comments to this blog, or drop by and tell us in person.

Again, thanks for the help

Splunk fans.

We have posted the our first of many preview releases. You can find them here:
Our hope is that every week or two as new features or API’s become usable that we post builds soliciting feedback.
This first post has a bunch of backend and UI performance improvements as well as some new but hidden features:

• live searching of data
• flexible roles
• scripted authentication
• event decoration ( for the xmas season )
• auditing of splunk server actions
• file system change detection
• improved (proper) sub second support
• transaction search
• new experimental simple search interface
• “where” support in search clause ( you dont need to use the “| where” anymore and can just search for foo=10 )

I’m not going to explain here what these things mean or how to find them or use them ;-)
Instead the product managers and developers will post here with ideas on what to try and what feedback we are looking for.

I’d like to thank in advance those brave few of you that have the few minutes to install these builds and give us your feedback.

e.

Hi all,

Just a heads up that we are moving to a model where we post previews of upcoming releases.

Starting now, we are going into a mode where long before a GA release we will be posting development builds. At first, they may be a few weeks apart but over time our goal is to post builds as soon as new functionality or API’s are ready for comment.

This first Preview #1 will have backend performance and scale improvements as well as some cool new features. The developers and PM’s will be posting to this blog the specifics of what is new, how to try it, and where we are going.

Our hope is that we get early feedback on new features and API’s before we actually ship.

Thanks in advance for helping try out our early wares.

Kinds Regards,

e.

I find this hard to explain even though its an extremely simple concept. It would be nice to get some feedback since I think we want to productize the idea but we are not clear on what makes sense.

If I have a search/report that I want to run faster, I will save that search and have splunk run it over a small timeframe (5,15,30,60 min) taking the results of that search/report and feeding them back into an index i create to hold cached results.

For example, suppose I like to run nightly reports where I show “top users by bandwidth”. Its easy enough to run the report every night, but suppose there are times during the day when I want incrementals, or I want to look at last week, or perhaps get dailies over a month. Every time I run the search/report I need to search and recalculate “top users by bandwidth”, which if over billions of events can take time ;-)

Instead, I’ll just save the search/report and have Splunk run it every 15 minutes with the results being sent to a “cache” index. This way if I ever want to do an adhoc search on “top users” or if I want to do “weekly reports by day” all the data is precalculated.

Think of this as creating “logs” that are the output of a search/report and then having Splunk index those “logs”. To get fast results you can then search/report on the summarized cached data.

If not obvious why it’s faster, suppose you are indexing 500M events a day and 100M of those have bandwidth data. To report on “top bandwidth by users” I need to run a search to get the 100M events then run the report across all 100M.
If instead I were in the

Dont forget to index your config files!
Why?
Because splunk is a great way to track changes and see differences in your configs.
For most troubleshooting and compliance situations having a historical recored of all your configurations just goes hand in hand with the log data. They are two sides of the same coin.

The cool thing is that it takes just a few seconds to get up and running. If you have splunk installed its all but free to index your configs - they are small in size compared to log files. Even if you indexed all configs in a 2000 machine deployment it would not come close to the volume of even a small size proxy log.

30 second refresher:
Just tail /etc you will capture most of the interesting configs on your box.

from the cli:
> splunk add tail /etc

or in UI just add a tail to /etc

Thats it. That is all you need to do.

** note ** you should grab 3.1 ( http://download.splunk.com ) as there were some bugs in 3.0’s config processing.

Splunk does something very interesting/cool with config files ( also source and script files ).
Unlike log files, splunk will treat the entire contents of the config files as a single result. This means that if i were to tail the file syslog.conf I would immediately get one result when searching for a key such as “authpriv” or for source=syslog.conf. This one result will be the contents of the file with the timestamp of the last mod time.

But as soon as the file changes splunk will recognize the change and re-index the entire file as another single result. Now there will be two results for the search “authpriv” or when searching for source=syslog.conf. At this time, splunk will have effectively kept

A standard preamble about life at splunk.

We are always looking for passionate and intelligent engineers regardless of their background, musical preference, grades they got in collage, or ability to prove some esoteric math theorem. We react best to people that are creative and think for themselves - people that are smarter than shit but don’t over think the wrong thing.

Much of our companies identity, the product, its features, how its used, how we talk about the product, our branding, all mainly come from the development staff - and we plan to keep it that way. Our philosophy is based on the idea that 10 diverse smart people in a room are better off than 25 decent yet uninspired engineers - so although we need to grow fast we end up going slow because we are picky.

Couple of other data-points:

• we like to have fun while at work
• we have smart people, challenging problems, and an interesting architecture
• we practice an liberal interpretation of scrum
• our code is cross platform and *resembles* open source development models
• we like nice hardware ( cpu, monitor, etc ) and dont care what OS you use
• we do most stuff in c/c++ with a bunch of python around the edges
• we dont use many 3rd party libraries/code ( xml parsers, ssl libraries, and a few others ) we write almost everything ourselves

We are always looking for great people so it never hurts to drop us a line. Even if your not sure you want to leave your current job and its fine if you dont have a resume ( i hate updating mine ) - just send us email ( j o b s @ s p l u n k . c o m ), hop on irc - #splunk irc.efnet.org, or stop by: 118 king st.

Having lived through the software-as-building-architecture argument every few years i am accustomed to thinking of (refuting) how software and software development is (not) like the traditional field of building design and development. The analogy driven mind needs something to reference and i guess us noobs in software are desperate to find something historical to feel validated.
Every discipline needs a role model and building design seems to be our adopted hero.

This post proposes an analogy that is far less intellectual than a typical comparison between Christoper Alexander and the design of an EventLoop Abstract Factory class.
My analogy here is based more on THIS weekend with MY wife.

Our house is full of stuff.
This stuff; chairs, tables, artwork, rugs,… we have acquired for good reason - and we could use it. The problem is that despite all good intention these hand-me-downs, gifts, rash purchases, sometimes just don’t work.

Yes we need a coffee table in the front room - but not THAT one.
Maybe the design is wrong.
Maybe the size is wrong.
Maybe the idea of a coffee table that is also is a fireplace seemed like a good idea at the time but WTF.

Sometimes you just need to throw stuff away.
In fact, its best if you institutionalize the process so that you actually do it.
For us its a quarterly Sunday where we drink lots of coffee, send the kids out for the day, get worked up, and throw (give) shit away.

So my analogy here is not about refactoring or deleting obsolete/deprecated code - its about the stuff on the surface - Features.

At least at Splunk, features are a bit like the coffee table we bought on sale because we had guests coming and the fact that it was also a fireplace was a double win. Useful features in theory, and perhaps even for competitive reasons we *need*