Introduction to Splunk HTTP Event Collector

Welcome to Splunk HTTP Event Collector (HEC). HEC is a fast and efficient way to send data to Splunk Enterprise and Splunk Cloud. Notably, HEC enables you to send data over HTTP (or HTTPS) directly to Splunk Enterprise or Splunk Cloud from your application. HEC was created with application developers in mind, so that all it takes is a few lines of code added to an app for the app to send data. Also, HEC is token-based, so you never need to hard-code your Splunk Enterprise or Splunk Cloud credentials in your app or supporting files. HTTP Event Collector provides a new way for developers to send application logging and metrics directly to Splunk Enterprise and Splunk Cloud via HTTP in a highly efficient and secure manner.

How does HTTP Event Collector work?

The basics of Splunk HTTP Event Collector are relatively simple:

  1. Turn on HTTP Event Collector by enabling its endpoint. HEC is not enabled by default.
  2. Generate an HEC token.
  3. On the client that will log to HEC, create a POST request, and set its authentication header to include the HEC token.
  4. POST data to the HEC token receiver.

Note: Non-self service Splunk Cloud customers must open a ticket with Splunk Support to both enable HEC and generate an HEC token.

You can send any kind of data to Splunk Enterprise and Splunk Cloud through HTTP Event Collector. Event data can be raw text or formatted within a JSON object. You can simplify the process by using a logging library, such as Splunk logging for Java or Splunk logging for .NET, which will automatically package and send data to HTTP Event Collector in the correct format. HTTP Event Collector also supports assigning different sourcetypes, indexes, and groups of indexers ("output groups"), so you can fine-tune where and how your data gets consumed by Splunk Enterprise or Splunk Cloud. You can use Deployment Server to deploy HTTP Event Collector configuration files.

Following is a sample event, formatted in JSON according to the HTTP Event Collector protocol. (This event could also have been raw text.) For more information about the contents of each event packet, see Format events for HTTP Event Collector.

{
    "time": 1426279439, 
    "host": "localhost",
    "source": "datasource",
    "sourcetype": "txt",
    "index": "main",
    "event": { "hello": "world" }
}

HTTP Event Collector data flow

Data is logged to HTTP Event Collector as follows:

  1. On the data source, data is packaged using an agent such as:

    • a Splunk logging library (Splunk logging for Java, Splunk logging for .NET, or Splunk logging for JavaScript)
    • a JavaScript request library
    • the Java Apache HTTP client
    • some other client

  2. Each HTTP request is assigned the same unique token in its authorization header, which has been generated with the management endpoint on the Splunk Enterprise or Splunk Cloud instance, using any of the following:

    • the HTTP Event Collector UI
    • a cURL command
    • the Splunk Enterprise command-line interface (CLI)
    • (non-self service Splunk Cloud customers only) a Splunk Support ticket

  3. The HTTP request, each of which includes the token, is sent to the appropriate Splunk Enterprise or Splunk Cloud endpoint.
  4. The token is verified against the list of known good tokens. If it's valid, an affirmative (OK) response is returned to the sender and the data is accepted by Splunk Enterprise or Splunk Cloud.
  5. Splunk Enterprise or Splunk Cloud sends the event data from the HTTP request to indexers to be indexed.

HTTP Event Collector workflow

There are three major workflows in HTTP Event Collector:

End user

An end user of HTTP Event Collector (most often a third-party app developer) simply needs to add a few lines of code to his or her app to enable it to log to HEC in Splunk Enterprise or Splunk Cloud. The easiest way to do this is to integrate the Splunk logging for Java, Splunk logging for JavaScript, or Splunk logging for .NET library into the app. If the user doesn't want to use one of the libraries, he or she must manually configure a mechanism to send event data over HTTP (or HTTPS) to the HTTP Event Collector REST API endpoint on the Splunk server.

Token admin

The token admin can be the Splunk Enterprise or self-service Splunk Cloud admin, or a different person who does not necessarily have experience with Splunk Enterprise or Splunk Cloud. Tokens are required for HTTP Event Collector to accept data that is sent to its port or endpoint. A token admin uses the Splunk Enterprise or Splunk Cloud management UI or Command Line Interface (CLI) to create, edit, disable, enable, and remove tokens. The token admin can also use the REST API token management endpoints to directly edit token configurations, and can enable or disable the HTTP Event Collector endpoints themselves.

Note: Non-self service Splunk Cloud customers must open a support ticket to administer HEC tokens.

Enabling HTTP Event Collector on Splunk Enterprise and self-service Splunk Cloud also adds a new capability: edit_token_http specifically enables roles to create and edit HTTP Event Collector tokens.

Splunk Enterprise or Splunk Cloud admin

On the Splunk Enterprise or self-service Splunk Cloud instance on which HTTP Event Collector is running, the admin can choose what do with and where to send the data that is sent from clients. For example, the admin can specify indexes, sourcetypes, and output groups. To do this, the admin edits the HTTP Event Collector endpoint.

Note: Non-self service Splunk Cloud customers must open a support ticket to edit HEC endpoints.