We highly recommend that you consider the following best practices when developing apps and add-ons for Splunk.
Are you really building an 'app', or are you building an 'add-on'? What is the difference, anyway?
An app is defined in the framework terminology section as "a redistributable package consisting of some or all of the following components":
In general, apps should focus primarily on creating a compelling and enriched user experience for exploring data produced by a certain technology, product, or solution.
An add-on is defined in the framework terminology section as "a redistributable package consisting of some or all of the following components":
In general, add-ons should consist primarily of configurations, knowledge, inputs, outputs, and workflows that classify, transform, or normalize data and are intended to be used by one or more apps.
Setting the appropriate scope for your development project provides the necessary constraints while setting appropriate expectations as to the finished product's functionality.
Who is the target user of the app? Are there multiple actors that might be involved in the day-to-day use of the app?
If the target user is a seasoned system administrator, time spent of self-documenting views and workflows might be better spent on performance tuning and extensibility.
On the other hand, if the target user is a technology neophyte such as level one help desk employee, the app should probably focus on simplicity and usability.
Let us suppose we are building an app to allow an Telecom's first level help desk personnel to troubleshoot basic phone and billing issues. By delivering the app, first level help desk will be able to better meet SLA and reduce the number of issues that must be escalated.
We might define the assumptions for the app as follows:
Defining these assumptions provides us with an informal contract: if any of these assumptions are invalid, the app may not be effective.
In addition to thinking about the audience and defining assumptions, it should also be considered what the user must be enabled to do with the app.
These are the functional requirements, and should be documented and prioritized.
Once you have scoped your project as an app and have considered the target audience, spend some time talking to representative members of your audience to better understand the problem you are trying to solve.
Continuing on from our example above, a preliminary set of functional requirements might look as follows:
Non-functional requirements define how an app should and should not operate and include (but are not limited to) security and quality requirements.
Non-functional requirements are very important to define and cost in the design phase due to the high cost of retrofitting an app with such requirements after the fact.
Continuing on from our example above, a preliminary set of non-functional requirements might look as follows:
The more complex the developer aspires for an app to be, the more important it is to carefully design the first time run experience.
In cases where a simple configuration dialogue will suffice, Splunk's application framework provides a basic tool chain for application setup in setup.xml.
However, there are several drawbacks to this approach:
In cases where setup.xml is sub-optimal, the best alternative is likely to develop your own custom setup experience within your app rather than through the Splunk Manager.
Eventually, every app will require a revision, either to fix bugs or to offer new and improved functionality.
As such, it is important to plan ahead on packaging to ensure that Splunk administrators have as few obstacles in their path as possible to upgrade your app.
If using SplunkBase as the primary repository for apps, the following app.conf requirements must be met for each app iteration:
For example, the first iteration of an app might have the following in default/app.conf:
[ui]
is_visible = 1
label = my_app
[install]
author=araitz
description=This is a cool app
build = 1
[launcher]
version = 1.0
Subsequent iterations of this app that are uploaded to Splunkbase must have an incremented "build" number greater than 1 as well as a different version number (e.g 1.0.1, 1.1, 2.0, etc).
On that note, it is a good practice to decide on a version standard prior to the initial release of an app. In most cases, a major-minor system will suffice:
It cannot be emphasized enough how important the "id" of an app is, and subsequently how important it is to pick the right "id" initially. In the world of Splunk and Splunkbase, changing an app's "id" is tantamount to creating an entirely new app, and comes with several unpleasant headaches when trying to inform or migrate users of the previous app "id". Changing an "id" is thus not recommended.
Finally, keep in mind that any changes made to an app's default configuration file values will not be effective if the values are present in the app's local configuration files.
For example, the following setting exists in default/inputs.conf:
[monitor:///my_app/my_file]
sourcetype=my_special_sourcetype
During the course of normal operations, a Splunk administrator sets the following in local/inputs.conf:
[monitor:///my_app/my_file]
sourcetype=my_special_sourcetype
host=special_host
If the sourcetype definition in default/inputs.conf changes in a subsequent iteration of the app, the change will not be respected due to Splunk's order of precedence.
Following on from the example above, it should be clear that understanding Splunk namespaces is critical to proper app implementation.
The core Splunk documentation has two articles on the subject, "App architecture and object ownership" and "Configuration file precedence", that should be reviewed by any developer that plans to use the Splunk Platform or Splunk App Framework.
Splunk ships on seven platforms! Even though it is not reasonable to test your app on each of those platforms, if your app interacts with the underlying OS in any special way (for example, as part of a third-party integration), note in your app documentation which platforms you have tested on.
Similarly, since the Splunk Web client is a browser, users can choose to interact with your app with almost any browser (okay, probably not Lynx). Do your best to implement JavaScript, HTML, and CSS in the most portable fashion possible, document known browser issues, and also document which browsers you have tested your app on.
The following rules should be applied when implementing any Splunk App functionality that creates, reads, updates, or deletes Splunk configuration files or entries within Splunk configuration files:
As part of simple common sense, good developers practice code reuse rather than wasting time and incurring undue risk by creating redundant code. The Splunk App Framework provides developers with a number of consumable libraries, the same ones that power Splunk Web and as a result undergo rigorous testing and other scrutiny.
Some of the best opportunities for reuse come in the utilities and helpers that the Splunk App framework provides to developers. There are consumable, documented utility tool chains available for JavaScript, Python, and Mako templates.
When possible, use C++ endpoints rather than writing new interpreted code to solve the same problem. For example, the REST API and, by proxy, Splunk Web expose md5, string, and time parsing that in many cases is faster and more convenient than ad hoc Python or JavaScript.
The Splunk App Framework utilizes models that are easy to reuse and nearly as easy to extend for your specific object. Rather than interacting with entities or REST in a one-off fashion, leverage and extend the models to provide a simple, standard, and tested way to interact with the Splunk object model.
Developers often ask: "Why doesn't splunkd perform input validation or output sanitization?" The answer is that splunkd is a general-purpose engine, and as such, there are thousands of conceivable uses for splunkd wherein allowing any input and output is desirable.
Thus, input validation must be built at the application level. The Splunk Python models provide methods for scrubbing or rejecting invalid input, and should be the central point that validation is performed. If the models don't provide sufficient validation for your use case, extend the model's validation methods through overriding or meta classing.
Avoid input sanitization in favor of flat-out rejecting invalid input. When feasible, favor whitelist validation over blacklist validation.
The vast majority of security vulnerabilities in the last few years are of the Cross-Site Scripting variety and almost always stem from unsanitized/unencoded output. To err is human, but to fail to escape or otherwise sanitize output can be a costly error.
In addition to input validation and output sanitization, there are some additional cardinal security rules to be aware of:
Style, by definition, is particular to the author. That said, it is a good practice to produce readable code so that others can understand, support, and extend it.
In most cases, Splunk core developers have done an excellent job of following current style and naming conventions for Python and JavaScript, so when in doubt, use existing code and provided style guides as a compass.
This includes the use of pydoc and JSdoc to supplement in-line comments, which are unfortunately not uniformly implemented across core Splunk code.
Peer reviews, even if lightweight, are proven to improve the security and quality of code. Don't be shy: share your code with peers, early and often.
If you don't have a network of peers to code review your Splunk App, post on the Splunk Community and ask for a hand. We would rather spend 15 minutes reviewing your code now than have you spend 15 hours fixing a deep design flaw.
Verbosity hurts the total footprint of your app, both in terms of size and complexity. In some cases, verbose code is related to poor performing code.
Perhaps more alarming, verbosity can also indicate that you might not be practicing code reuse and are creating ad hoc code rather than leaning on Splunk's existing libraries and utilities.
Hopefully, everyone reading this section will also have read the above section on namespaces, sharing, and order of precedence.
An app's "local" directory and "local.meta" file under the app's "metadata" directory are not safe places to store default configurations.
Always use the app's "default" directory and the app's "default.meta" file for default configurations.
Lookups are a unique Splunk knowledge object in that there is no "default" and "local" namespace.
Therefore, if you ship an app with a mutable lookup, and your app consumer makes changes to this lookup, these changes will be overwritten upon app update!
The best practice is to only ship immutable lookups in apps (for example, a mapping of error codes to plain-text values) and require that mutable lookups be created via a workflow action, setup view, or saved search.
Even then, take care to educate users as to the special nature of lookups so that they don't edit immutable lookups only to lose their changes when they update the app.
It is important to consider the typical deployment scenarios that are likely for your app.
The three most typical Splunk deployments and associated considerations follow:
All that notwithstanding, good documentation can be critical to the success of your app.