Configure an NGINX load balancer for HTTP Event Collector

When setting up an HTTP Event Collector deployment where you need high availability, throughput, and scale, consider a network traffic load balancer such as NGINX. You can use any load balancer in front of HEC, but this section focuses on how to use NGINX to distribute the load.

Note: This section contains basic information about getting NGINX software up and running to load balance data that is intended for HTTP Event Collector. For detailed information about using Open Source NGINX or NGINX Plus, see the NGINX support site.

This procedure assumes that you are using HTTPS to send data to HEC, and that you are using the default install path, /usr/local/nginx. First, build or install a version of NGINX that enables HTTPS support for an HTTP server. To enable HTTPS, use the following modifier:

./configure --with-http_ssl_module

After you've installed NGINX, complete the following three basic steps:

  1. Configure the SSL certificate.
  2. Configure the upstream servers.
  3. Complete the nginx.conf file.

Configure the SSL certificate

If you're using the default SSL certificate that ships with Splunk Enterprise, copy the $SPLUNK_HOME/etc/auth/server.pem file (%SPLUNK_HOME%\etc\auth\server.pem on Windows Splunk Enterprise hosts) to your load balancer. However, Splunk recommends that you generate your own SSL certificate and use it in place of the default certificate. For more information about configuring Splunk Enterprise to use your own SSL certificate, see Secure Splunk Web with your own certificate in the Securing Splunk Enterprise manual.

The following configuration, which we'll later add to the nginx.conf file, assumes you've copied server.pem to /usr/local/nginx/conf.

    server {
        # Enable SSL for default HEC port 8088
        listen 8088 ssl;

        # Configure default Splunk Enterprise certificate. 
        # Private key is included in server.pem so use it in both settings.
	ssl_certificate     server.pem;
    	ssl_certificate_key server.pem;		

	location / {
            # HEC supports HTTP Keepalive so let's use it
	    # Default is HTTP/1; keepalive is only enabled in HTTP/1.1
  	    proxy_http_version 1.1;

  	    # Remove the Connection header if the client sends it,
  	    # it could be "close" to close a keepalive connection
  	    proxy_set_header Connection "";

            # Proxy requests to HEC
            proxy_pass <scheme>://<host>:<port>/services/collector;
	}
    }

Configure the upstream servers

Next, configure the upstream servers. The upstream servers comprise the group of servers running HTTP Event Collector that are load balancing data before sending it to your indexers. Note that you must use a heavy forwarder here; HEC does not run on a universal forwarder. The following configuration will later be added to the nginx.conf file:

    upstream hec {
        # Update with your list of Splunk servers with HEC enabled
        # server <splunk_server_name>:<hec_port>;
	keepalive 32;

        server splunk1:8088;
        server splunk2:8088;
    }

Complete the nginx.conf file

Now, complete the nginx.conf file to correspond to your NGINX instance. For example, the following complete example nginx.conf file will need to be tuned for your specific NGINX instance and resources:

# Tune this depending on your resources
# See the Nginx docs
worker_processes  auto;

events {
    # Tune this depending on your resources
    # See the Nginx docs
    worker_connections  1024;
}


http {
    upstream hec {
        # Update with your list of Splunk servers with HEC enabled
        # server <splunk_server_name>:<hec_port>;
	keepalive 32;

        server splunk1:8088;
        server splunk2:8088;
    }

    server {
        # Enable SSL for default HEC port 8088
        listen 8088 ssl;

        # Configure default Splunk Enterprise certificate. 
        # Private key is included in server.pem so use it in both settings.
	ssl_certificate     server.pem;
    	ssl_certificate_key server.pem;		

	location / {
            # HEC supports HTTP Keepalive so let's use it
	    # Default is HTTP/1; keepalive is only enabled in HTTP/1.1
  	    proxy_http_version 1.1;

  	    # Remove the Connection header if the client sends it,
  	    # it could be "close" to close a keepalive connection
  	    proxy_set_header Connection "";

            # Proxy requests to HEC
            proxy_pass <scheme>://<host>:<port>/services/collector;
	}
    }
}

Next steps

When you start NGINX, you will be prompted to enter the PEM passphrase for the SSL certificate. The password for the default Splunk Enterprise SSL certificate is password.

From here, update your NGINX server settings to tune the server to handle the data you intend to load balance. For example, consider tweaking settings such as: