We use Perforce at Splunk, and it’s worked out pretty well for us. I’m a CVS admin at heart, and I know there’s some SVN sentiment, but p4 gives us a nice mix of atomic commits, attractive GUI and command-line tools, and someone to call for help if it ever completely eats itself.

Over time I’ve compiled a small library of scripts for various p4 functions that have been written time and again at different sites…mergetool is one of them. This little tool accepts a merge target (”yours” in p4-speak) and projectile (”theirs” in p4), labels both, performs an integrate, and performs a “safe” resolve -as. It logs any failures for you to resolve by hand, or submits the change set if the resolve completes successfully. It does this with a bunch of logging in a well-organized, date-stamped directory suitable for archiving (or splunking).

Like a lot of tech companies, Splunk’s development organization isn’t a model of perfect gender balance. For a year and a half now, I’ve been the only woman in the dev organization.

Surprisingly, this is not an uncomfortable place to be. In 11 years in industry I’ve worked in a variety of organizations: the now-bankrupt dot-com best known for putting an ad with a naked guy up during the Super Bowl, 2 major marquee names with vastly differing corporate cultures, a security start-up stocked with emancipated-minor hackers. Aside from that doomed dot-com — which had a surprisingly strong gender balance throughout technical roles and a culture blessedly free of gender-based intimidation at all levels — Splunk may be the most comfortable place I’ve ever worked. There’s no creepy tokenism (unlike stories I’ve heard about certain other bay area employers), That Guy Who’s Never Seen A Girl Before doesn’t work here…and as far as I can tell, no one really gets harassed except Amrit.

Perhaps a better testament for the dev culture than my opinion — because, frankly, I’m pretty weird to start with — is that other women in the company seem to be pretty comfortable visiting the dev area, either on work errands or just to take a break from the sales-focused environment upstairs. Frankly I can’t imagine that happens too often in the bay area…and more’s the pity.

Splunk runs on a lot of platforms for a relatively young product and that number is always increasing. The day I started, there were packages for Intel and PowerPC Macintoshes, i686 Linux, Solaris 8 on Sparc, and FreeBSD on x86, all created with BitRock InstallBuilder, run from a simple shell script, usually by Erik. There really wasn’t much control over what went into the installer — if a file was in the installer prep directory and the shell script didn’t know to delete it, out it went.

By the time 2.1 was on its way, we’d decided to switch to native packages, and our list of platforms had expanded to include Solaris on Intel, with several more on the horizon. We also wanted to provide the “rail tarball” distribution we continue to support, in part so that QA could get started before the packaging automation was complete.

What is that packaging automation, you might ask? Obviously writing custom code to package each platform (not to mention spec or pkgmap files in each platform’s native format) was not a very maintainable solution. Instead we use a locally modified version of Easy Software’s EPM package manager. After a little work, EPM lets us use a common set of list files to create relocatable packages using common pre- and post-install scripts across all of the 9 platforms we now build on. We’re able to control every file and permission that goes into the packages, and in most cases we can add packaging for a new OS platform with a minimum of work (for something very different we haven’t previously had in house, like AIX, more time might need to be spent cleaning up EPM’s support for the platform). We’ve piggy-backed creation of the “rail tarball” distributions on the EPM list file structure, so

Hi! My name is Kim, and I’m the release engineer here at Splunk.

Thanks to my acquisition-happy former employer, Symantec, I’ve seen a variety of startup approaches to release engineering. Most frequently it seems some senior developer has a bug up you-know-where about how the build system should work, and some poor junior developer or sysadmin type person dutifully does the drudge work (usually by hand). At other sites, some very diligent and detail-oriented person creates and executes a process with a great deal of record-keeping and attention to detail but often not a lot of automation. Consistency across different build platforms usually isn’t a strong point.

Here at Splunk, things are a bit different. I called myself the plumber in the title of this post because that’s how I see my job: I create and maintain the plumbing that produces consistent, reproducible Splunk builds across all of our platforms, with as much visibility as I can muster. I see my contribution more as enforcing process through tools — ideally, tools that enable process in a way that is more convenient for everyone than “doing it wrong” — rather than personally pushing all the buttons and scribbling in all the logbooks. And I’ve had the good fortune to come into a culture that encourages this approach.

Whew. That’s a mouthful for an introduction. In the near future I hope to write a bit more about how the plumbing works, and some neat tools I’ve found along the way. I’m sure y’all will be waiting with baited breath.