Vet apps and add-ons for Splunk Cloud

Cloud vetting is a process that determines whether an app or add-on can be used in Splunk Cloud.

Apps and add-ons must be evaluated to help ensure the security of the Splunk Cloud environment, as well as the security of the data stored in that environment. In addition, Splunk Enterprise and Splunk Cloud have several crucial differences. While most apps and add-ons on Splunkbase are suitable for an on-premises Splunk Enterprise environment, they have not all been evaluated for transmitting and storing data in a cloud environment. Some apps might not be authorized for Splunk Cloud because they are intended for on-premises Splunk Enterprise systems or for installation on Splunk Forwarders.

Vetting for Splunk Cloud is performed when a Splunk Cloud customer requests that an app or add-on in Splunkbase be installed in their Splunk Cloud environment. A Splunk Cloud customer who wants to use an app or add-on can request that cloud vetting be performed by opening a support ticket with Splunk Support and Services. If the app or add-on passes cloud vetting, it is installed on the customer's instance of Splunk Cloud. If the app or add-on is hosted on Splunkbase, Splunk Cloud is added to the list of compatible products.

This rest of this topic describes the cloud vetting process for developers:


The cloud vetting process

Cloud vetting includes an automated and a manual process (if required):

  • Splunk Inc. runs the Splunk AppInspect tool to perform automated cloud vetting.
  • When necessary, a Splunk employee performs a manual cloud vetting process to further evaluate the app or add-on.

Prepare your app or add-on for cloud vetting

To prepare your app or add-on for cloud vetting:

  1. Review the Development requirements for Splunk Cloud below.
  2. Verify that you have fulfilled all of the Splunk Cloud requirements by running the Splunk AppInspect tool with the cloud tag.
  3. For example, use the AppInspect CLI as follows:

    splunk-appinspect inspect app_path/app_filename.tgz 
    --mode precert --included-tags cloud

    Or, use the AppInspect REST API as follows:

    curl -X POST \
    	-H "Authorization: bearer <token>" \
    	-H "Cache-Control: no-cache" \
    	-F "app_package=@\"app_path/app_filename.tgz\"" \
    	-F "included_tags=cloud" \
    	--url "https://appinspect.splunk.com/v1/app/validate"
    

    For more about AppInspect, see Overview of Splunk AppInspect.

  4. Review the inspect command results:
    • One or more failures indicate that your app or add-on has failed cloud vetting, and is therefore not approved for installation on Splunk Cloud. Identify and fix the failures, then run the command again.
    • One or more manual checks indicate that the app or add-on requires manual checking as part of the cloud vetting process. Therefore, when you submit your app, a Splunk employee must manually perform cloud vetting. The manual cloud vetting process will most likely take longer. To increase the chances of passing cloud vetting, follow the Development recommendations for Splunk Cloud below.
    • Apps or add-ons that return zero failures or manual checks are most likely to be quickly approved for installation on Splunk Cloud.

Splunk Cloud criteria and tips

The criteria that Splunk uses to vet an app or add-on for Splunk Cloud are listed below. These criteria are subject to change as new security threats are discovered and the Splunk platform is updated.

Important  Splunk Cloud app developers and users of Splunk Cloud apps assume responsibility for ensuring proper usage of any third-party services that they choose to use in connection with Splunk Cloud, including compliance with any relevant terms and licenses. As a reminder and pursuant to Splunk Cloud Terms of Service, Splunk is not liable for any problems that might arise from sending data to those third-party services (including, without limitation, any disclosure, modification or deletion of data resulting from access to such third-party services) and does not provide any support for those services. For more information, see Splunk Cloud Terms of Service FAQs.

Development requirements for Splunk Cloud

This section lists the requirements for developing an app or add-on for installation in Splunk Cloud.

Do the following:

  • Write all scripts for 64-bit Linux. All scripts must be able to run on Linux because the Splunk Cloud service runs on Linux-based servers.
  • Ensure that all network communication is encrypted and secured with the SSL protocol. Any configurable options that affect network communication must be secure as well.
  • Ensure that all credentials that are used and stored by the app (such as API keys, account passwords, and Base64-encoded private keys) are encrypted, using the storage/passwords REST endpoint. For more information, see Access endpoint descriptions in the REST API Reference Manual and Setup page example with user credentials.
  • Provide source code for review, by providing it with the packaged app or by including a link to a public open-source repository.
  • Package the app according to these guidelines for Splunkbase apps:
    • Package the app as a .tgz, .tar.gz, or .spl file.
    • Remove all hidden files.
    • Remove all executable binary files, unless source code is provided.
    • Remove .pyo and .pyc files.
  • Document all dependencies, installation procedures, and post-installation validation procedures, including a precise list of dependencies with their version numbers. To test your app effectively, Splunk needs to know which SDKs, apps, or add-ons are required to run your app.
  • Test your app's performance. Apps that cause significant performance degradation might be rejected.
  • Ensure that any credentials required for the app to function are entered by the user in a setup or configuration page. For more information, see Create a setup page in the Splunk Add-on Builder User Guide and Splunk Meet the Experts Advanced Visualizations on Github.

The following lists those practices you must avoid when developing apps and add-ons for Splunk Cloud:

  • Do not require privilege elevation with sudo, groupadd, useradd, su, or other similar utilities.
  • Do not use the reverse shell technique. For more information, see ReverseShell on the Ubuntu Wiki site.
  • Do not allow your app or add-on to manipulate files outside of the app directory, except in the following circumstances:
    • When writing to the Splunk software log directory, $SPLUNK_HOME/var/log.
    • When creating modular input checkpoints. For more information, see Data checkpoints in the Developing Views and Apps for Splunk Web manual.
  • Do not allow your app or add-on to manipulate processes outside of the control app, including the Splunk software instance or processes created by other apps.
  • Do not allow your app or add-on to manipulate the operating system.
  • Do not allow your app or add-on to allow file management through the user interface.
  • Do not allow your app or add-on to use any of the reserved ports 443 (inbound only), 8080, 8089, 8443, 9887, or 9997.
  • Do not allow your app or add-on to restart the Splunk Cloud server.
  • Do not allow your app or add-on to monitor the Splunk Cloud infrastructure.
  • Do not allow your app or add-on to send user data from the Splunk Cloud server to a third party without the user's explicit consent.
  • Do not provide automatic update features for scripts, executables, or libraries.

Common failures and preventive actions

This section discusses some common types of failures you might run into when vetting apps and add-ons for Splunk Cloud - what causes these failures and how to prevent them. Avoid these pitfalls to successfully pass Cloud vetting.

Insecure network communication

Appinspect uses check_for_unencrypted_network_communications to check that all network communications are encrypted using SSL/TLS. Vetting fails when data communications between an app and Splunk Cloud are not secured.

Here are some common causes for this type of vetting failures.

Cause for failure Preventive action
The http scheme configured from the Alert Actions, Data Inputs or App configuration pages is not validated, which allows the app to send and receive unencrypted data on the network Always validate user input and only accept https communications
The app uses insecure protocols such as UDP, FTP, and SMTP Use an encrypted SSL/TLS channel for data communications. For example:
smtp = smtplib.SMTP(smtpHost,smtpPort) 
smtp.starttls()         
smtp = smtplib.SMTP_SSL(smtpHost,sslPort)
ftp = ftplib.FTP_TLS()

Secret discloure

AppInspect uses check_for_secret_disclosure to check whether passwords and secrets are protected or are subject to disclosure.

Vetting fails if an app contains credentials as plain text in Splunk Cloud. A secret credential is anything that allows a hacker to impersonate a user and access the user's private information. Secret credentials contain passwords for any account, access token, or key for any platforms such as Splunk session key, and other identification information.

Note: If a field is not a secret credential, do not give it a name that resembles one, such as client_token, access_key, or password

Here are some common causes for this type of vetting failure.

Cause for failure Preventive action
Some secret credentials are exposed in plain text in the source code, README file, lookup table, .conf files, or other files. For example, the password in the README file is not encrypted.
username = admin
password = 23ds_ewwq!
Remove any secret credentials from the source code.
Credentials configured on the UI are stored as plain text, usually in file $APP_HOME/local/xx.conf
For example, setting encrypted to false will cause Cloud vetting to fail.
field.RestField(
    'api_key',
    required=True,
    encrypted=False,   
    default=''
)
Use the storage/passwords endpoint to encrypt secret credentials. Also, make sure you do not have unencrypted secret credentials temporarily stored in a plain text in the file system, which is not allowed. Refer to Setup page example with user credentials and passwords.conf for details.
Credentials are displayed in console or stored as plain text in log files, as is implemented the following code:
send_url = url + "?api_key" + api_key
helper.log_debug(send_url)    
Remove credentials or replace them with dummy credentials in logs or console
The user can set proxy as follows on the UI and save the proxy setting as plain text in a local .conf file.
proxy=http://username:password@ip:port
Provide separate fields - host, port, username, password - on the UI for the user to configure proxy. When the user enter the proxy password, mask the field and store the password in the storage/password endpoint to encrypt it.

System environment issues

Some vetting failures are caused by system environment-related issues.

Cause for failure Preventive action
The app contains code that modifies OS environment variables; for example:

unset LD_LIBRARY_PATH
Remove any code that modifies OS environment variables
The app imports libraries from other non-Splunk apps; for example:
sys.path.append(make_splunkhome_path
(["etc", "apps", "<OTHER_APP_NAME>",
 "lib"]))
Package any required library inside the app to subject it to Cloud vetting
The app contains hard-coded paths that do not work in Splunk Cloud. Use the $SPLUNK_HOME environment variable whenever possible

Out-of-boundary

Some operations outside of an app's boundary will cause the app to fail the vetting process.

Cause for failure Preventive action
The app manipulates files outside of the app's boundary. Make sure all file manipulations are limited inside the app's boundary, which includes the following paths:
$SPLUNK_HOME/etc/<APP_NAME>, 
$SPLUNK_HOME/var/log/<APP_NAME>
The app reads sensitive files external to the app; for example, reading system process information from /etc/shadow, collecting data from the Splunk Cloud infrastructure, or retrieving search results containing keyword "security risks" using system commands such as top (Linux) and btool (Splunk) Remove any function that retrieves sensitive information from outside of the app
The app contains binary files that cannot be vetted Provide source code of the binaries for Cloud Vetting, either by packaging it in the app or including github links in README.md

Shell injection vulnerabilities

If an app is potentially subject to shell injection, it will fail Cloud vetting.

Cause for failure Preventive action
The app behaves as a reverse shell, which is not permitted in Splunk Cloud.
For example, the user configures the Path field on the UI. The value gets passed to a subprocess parameter, which can result in reverse shell injection.
Remove or validate all the functions and paths that allow command line access to prevent reverse shell injection

Development recommendations for Splunk Cloud

The following practices are recommended, but not required, for developing apps or add-ons for Splunk Cloud:

  • Create setup pages that allow users to configure the app or add-on. Splunk Cloud users cannot access the shell or server file system, and cannot manipulate Splunk configuration files directly. For more information, see Create a setup page for a Splunk app.
  • Clearly comment your code to accelerate the review process.
  • Provide sample data with your app or add-on using the Splunk Event Generator utility (Eventgen), along with an eventgen.conf file. Sample data helps demonstrate how your app or add-on functions during the code review. For more information, see the Eventgen app on Splunkbase.
  • Specify when an app or add-on requires multi-threading. Apps or add-ons that require more than one thread per search might be forced to run on their own search head.
  • Ensure that your app or add-on cleans up after itself, including freeing memory, terminating processes, and closing files.
  • Provide version and build numbers in the app configuration file (app.conf). For more, see app.conf in the Admin Manual.
  • Do not use "#!" characters to specify the Python interpreter in scripts. Splunk Cloud uses a customized Python interpreter to invoke all scripts.
  • Do not use the Python command unset LD_LIBRARY_PATH, because it might prevent scripts from properly mapping Splunk software libraries.