Troubleshooting connectivity to the Splunk Enterprise API

There are a number of ways to troubleshoot connectivity to Splunk Enterprise when using a Splunk SDK or logging library. This topic covers a few methods that may help you figure out what is causing problems.

There are two families of problems that are most common:

  • Issues with general connectivity to the API
  • Issues involving certificate validation or the Splunk Enterprise security protocol configuration

General connectivity issues

If you're unable to successfully connect to the Splunk Enterprise API, first try using the command-line tool to log into the Splunk API via cUrl. For example, try the following command, replacing the placeholders with actual values from your setup:

curl -k <server:port>/services/auth/login -d username=<username> -d password=<password>
If you get a response such as the following, you've proven that the API is accessible:
<response>
   <sessionKey>G^b8zfS3YGWFSJTxY7c5fpqso4DZr6_O_z9lcXk^v...</sessionKey>
</response>

If you get a different response, depending on the type of message you receive, there are a few possibilities as to what's going wrong:

  • You receive an error such as the following:

    <msg type="WARN">Remote login disabled by 'allowRemoteLogin' in server.conf</msg>
    

    Check the allowRemoteLogin setting in your Splunk Enterprise instance's server.conf file. If you have it set to never, you've disabled the ability to log in remotely.

  • You receive an error such as the following:

    <msg type="WARN">Remote login has been disabled for 'admin' with the default password. Either set the password, or override by changing the 'allowRemoteLogin' setting in your server.conf file.</msg>
    

    Check the allowRemoteLogin setting in your Splunk Enterprise instance's server.conf file. You have it set to requireSetPassword (the default) and have never changed the admin's password from the default, "changeme."

  • You receive an error such as the following:

    curl: (7) Failed connect to 10.80.9.131:8088; No error 

    Either the URI is wrong, the port number is not correct, or the port is not opened on the firewall.

  • If you receive an empty reply, you're using the wrong scheme. For example, you're using HTTP when you should be using HTTPS.

If you get a valid response using cUrl, but the SDK is still failing, then either the credentials or the URI passed in the code that uses the SDK could be wrong. Check your app's configuration.

Security configuration issues

The second type of issues relates to either certificate validation failing, or to the security protocol configuration in Splunk Enterprise.

Certificate validation

Another kind of error you might see relates to certificate validation. By default, most platforms automatically throw an exception when the HTTPS certificate is not valid. Splunk does not return a valid cert by default, which causes this failure.

This can be disabled within your application's code. Using the Splunk SDK for C#, you can turn off certificate validation using code such as the following:

ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) =>
{
     return true;
};

Security protocol

You can also see an error that indicates the client is unable to negotiate a connection. This usually means that the security protocol the client is using to connect is not within the set that Splunk is configured for in the server.conf file. For example, you might be using Secure Sockets Layer 2.0 (SSL2) while Splunk is configured for SSL3 or Transport Layer Security (TLS) only.

Here is how to configure the security protocol to use TLS using the Splunk SDK for C#:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;

Once the protocols on client and server match, you should be able to successfully connect.