Building in telemetry with high-performance data collection

This chapter demonstrates a powerful and efficient new way to get data directly into Splunk Enterprise from network sources—the HTTP Event Collector. In addition, it touches on a brand-new visualization scheme: geographic visualizations with choropleth maps.

Consider the two scenarios that our business stakeholders brought to us. Each is very different from the other, but each ended up being a perfect candidate for us to explore.

Scenario 1 (the "keycard scenario"): Our business partners wanted a way to track when and where their employees and contractors were accessing company resources and real estate. Both employees and contractors already have to use their company-issued keycards to gain access. How could the company get a high volume of those access events per minute into Splunk Enterprise, preferably without having to set up a Splunk forwarder, and then view that data in a useful and insightful way, such as on a map?

Scenario 2 (the "telemetry scenario"): Our business stakeholders were developing a web-based app and wanted to use Splunk Enterprise to collect and organize both debug and usage data for the app. Because the data could be coming from anyone, anywhere in the world, they didn't want it to contain any Splunk-specific information, and certainly didn't want it to include any Splunk Enterprise credential information.

Both of these scenarios are prime candidates for a new feature in Splunk Enterprise 6.3: a developer-focused way of getting data in called "HTTP Event Collector." Let's first examine the feature in more detail, and then look at the different ways in which it came in handy in the PAS app.

All about HTTP Event Collector

HTTP Event Collector (or "EC") is a new way to send data to Splunk Enterprise. With EC, the data source (or sources) send data packaged within JSON-formatted data packets through HTTP or HTTPS directly to Splunk Enterprise. No extra software is required on the client side, though there are logging libraries available for several programming languages—including Java, C#, and JavaScript—to help automate the process of properly formatting the data. EC uses specialized tokens, so you don't need to hard-code your Splunk Enterprise credentials in your app or supporting files. It's also scalable. Use Deployment Server to deploy your tokens and configuration files to other Splunk Enterprise instances, and then use a load balancer to distribute incoming HTTP Event Collector data evenly to indexers.

Getting data into Splunk Enterprise using HTTP Event Collector is different from the other data input methods:

  • First, you have to turn on the feature. Doing so opens up the EC port (8088 by default, though you can change that) and causes Splunk Enterprise to start listening on it for incoming requests.
  • Then, generate an HTTP Event Collector authentication token ("EC token"). EC tokens are sent in the headers of incoming data packets to authenticate them with Splunk Enterprise. You generate a new token on your Splunk Enterprise instance, and then give it to the sender of the data.
  • Next, send the data, placing the token in the authorization header of every JSON data packet that is sent. If there is a problem, Splunk Enterprise doesn't accept the data, and sends back a 401 (Unauthorized) status code to the sender.
  • If a data packet has a valid, active token in its auth header, Splunk Enterprise accepts the data packet, sends back a 200 (OK) status code to the sender, and indexes the packet's event data.
DEVProblems that you can encounter when sending data include: an invalid or missing token in the auth header, incorrectly formatted JSON, JSON that doesn't conform to the EC event protocol, an invalid timestamp, an invalid or disallowed index, and so on.

You can send any kind of data to Splunk Enterprise through HTTP Event Collector, but it must be contained within a JSON payload envelope. Here's a sample event in JSON format, created according to the EC event protocol:

{
    "event": { "hello": "world" }
}

This is the simplest kind of event—one that contains just event data. You can also specify values such as a timestamp, a hostname, source and sourcetype values, and a preferred index, but doing so through the event data itself is uncommon and not recommended.

You can also batch events. That is, you are not limited to one event per data packet; you can send multiple events in a single request and Splunk Enterprise will index each event individually.

For all the details about the EC's JSON event format, see "About the JSON event protocol in HTTP Event Collector."

Using HTTP Event Collector

Trying out HTTP Event Collector is easy, and only requires a brief detour from our journey. In this section we'll enable EC, create a new HTTP Event Collector token, and then send some data. You don't even need to have the PAS reference app installed.

First, turn on EC. Unless you've already used it, EC is disabled by default. To enable it, go to Settings > Data inputs > HTTP Event Collector. Then click the Global Settings button in the upper-right corner. This will bring up the following configuration screen for EC:

Event Collector Global settings page

Click the Enable button, and then click Save. You've just turned on HTTP Event Collector.

Now that EC is turned on, let's create a new HTTP Event Collector token. From the HTTP Event Collector page, click the New Token button.

The Select Source screen of the Add Data workflow appears. This is where you name and describe the EC input, specify (if you want) a source field name to give to all data accepted with this input's token, and optionally specify an output group (a named group of Splunk indexers).

Event Collector New token page

Enter at least a name for the input, and then click Next.

The Input Settings screen appears. On this screen, you determine how to assign a sourcetype field value to incoming data (either automatically, by specifying an existing one, or by creating a new one) and what indexes are allowed to index the data accepted with this input's token. You also specify the default index to use to index data bearing this input's token.

Event Collector New token input settings page

On the Input Settings page, leave the Source type as Automatic, and then choose at least one index that is not used for production, or real-world, purposes. Then, click Review. The Review page appears.

Event Collector New token Review page

Review your input settings, and then click Submit. You'll see a message that says, "Token input has been created successfully." A token value is simply a globally-unique identifier (GUID) that Splunk Enterprise has generated to identify data intended for this EC input. You can click the token value from this screen and copy it to your Clipboard for later use, or go back to Settings > Data inputs > HTTP Event Collector, and you'll see that your new input is listed along with its token value and all its other pertinent information.

Select the entire token value, and then copy it to the Clipboard. Now we'll use it to send some data.

Open a command prompt window or terminal. Type the following cURL statement to test out your token. Be sure to replace <host> with your Splunk Enterprise server's hostname, and <token> with the token you just copied to the Clipboard:

curl -k https://<host>:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"Hello, World!"}' 
DEV Because of the way Windows handles single and double quotes, this cURL command will not work on Windows. To get it to work on Windows, you can either replace the single quotation marks (') with double quotation marks (") and then escape the inner double quotation marks, or you can use an app like Postman for Google Chrome to send the request instead. In fact, because of Postman's powerful feature set and its ability to run cross-platform, we highly recommend it: getpostman.com.

If everything is working correctly, you will see the following response:

{"text":"Success","code":0}

This means that Splunk Enterprise has received the data. Let's check it out within the search app.

Back in Splunk Enterprise, on the Apps menu, click Search & Reporting. In the search box, type the following, making sure to replace <input_name> with the name you gave your input:

source="http:<input_name>" 

That's it! Press Return or Enter, and you'll see that Splunk Enterprise has found one event that corresponds to your input's name.

Search window showing the event

The keycard scenario

Recall from earlier in this chapter our keycard scenario:

Our business partners wanted a way to track when and where their employees and contractors were accessing company resources and real estate. Both employees and contractors already have to use their company-issued keycards to gain access. How could the company get a high volume of those access events per minute into Splunk Enterprise, preferably without having to set up a Splunk forwarder, and then view that data in a useful and insightful way, such as on a map?

Sending one event is fine and good, but how do we expand this very basic introductory scenario so that we can leverage HTTP Event Collector for our keycard scenario? What if we wanted to multiply the number of events received by thousands or even tens of thousands? Also, how do we test it to an extent where our business stakeholders will be reasonably confident that it will work as described in the wild? And given that location is a key component to our data, how can we best summarize and display it? These questions were to constitute the next leg in our journey.

BUS HTTP Event Collector is capable of dealing with tens or even hundreds of thousands of events per minute. In our scenario, we didn't need to reach this capacity, but in product testing, the EC has hit very impressive performance testing goals.

Ultimately, this scenario involved not much coding or setup on the Splunk Enterprise side. Most of our time was spent finding an adequate solution to simulate sending as much dynamic event data as we needed. The following is a discussion of what we did and didn't do, and how we could have done things differently.

Hardware and architecture

We decided to start small: a single Splunk Enterprise instance with no load balancers or external indexers. We knew this was feasible because this was a dedicated machine for testing purposes, and based on past experience we were confident that HTTP Event Collector could handle what we would throw at it.

ARCH In a production environment, depending on the number of access points that would be sending data to EC, we would likely need to scale up our Splunk Enterprise infrastructure to include a load balancer and an output group containing several additional indexers. Luckily, EC supports load balancing, and can adapt to fit any size implementation.

Choosing an index

We decided that the index that we had been using for PAS-specific testing operations would suffice as the default index for storing our keycard event data as well. You can set the default index per token when creating a new HTTP Event Collector input.

ADM In our sandbox, the PAS index sufficed as the default index for keycard event data, as it was able to consume what amounted to hundreds of events per minute. However, depending again on the size of your potential data set, you might want to create an index that is dedicated to this type of data. Having EC data living in its own index is the fastest possible way to query it. While the default index may have been fine for this use case, in a production system where data comes in from multiple places, separate indexes allow faster retrieval. Beyond that, there is also the benefit of being able to easily archive separate indexes.

SEC Another reason to create a keycard event data-specific index is security. Consider sending keycard or other similarly sensitive data to its own index. If your keycard data has a separate security policy, this will ensure it doesn't blend with other PAS data. Also, you can't lock down only certain data within an index. Data is locked down at the index level, so it's all or nothing. That is, you can assign a Splunk Enterprise role access to an index (and assign a Splunk Enterprise user that role), but you can't give a role the ability to only search on specific categories of events within an index.

Creating the token

As you saw when we ran through the basic steps to send an event to HTTP Event Collector, creating a token is easy. For our testing purposes, we called our EC input "pas_keycard." We assigned the "pas_keycard" index as the input's default, and decided to assign events a custom source type that we called "ri:pas:keycard." We didn't specify a source to assign to events.

Edit token window

Why should you care what we called these input attributes? Because thinking strategically about what you call the input's attributes can make it easier to interpret the data later. For example, leaving the Source field empty was probably a good call, since we'll likely be receiving data from a number of different places. We could name the source values as events come in and base them on a building name, a server name, and so on. However, we may not need to do this if the data senders' hostnames are descriptive enough. Keep in mind that if the source is empty and the client does not send one, then the source will be http:<token_name>, where <token_name> is the name you gave the EC input when you created it.

DEV Token strategy is worth spending some time thinking over. For example, some developers choose to use different tokens for each different app, operating system, or device form factor from which they receive data. Since tokens can easily be bound to different indexes, you can automatically partition the data for faster retrieval. This kind of thinking is an essential part of planning an EC rollout.

Sending the data

The actual packaging and sending of data is done on the data source. Recall from before that HTTP Event Collector doesn't care what's doing the packaging and sending of data, or what the data consists of, as long as the data is packaged inside of the EC JSON-formatted data packet. Packaging and sending event data can be done in a number of ways:

  • A Splunk logging library (such as Splunk logging for Java, .NET, or JavaScript).
  • A JavaScript request library.
  • The Java Apache HTTP client.
  • Some other client, as long as the data is packaged in JSON according to the event protocol specification.

Our priority was spinning up a way to simulate hundreds of keycard events per minute being generated and sent to EC.

Based on conversations with the Splunk test team, a few possibilities were considered: ApacheBench, Siege, and Apache JMeter. Our developers found that they've all got some strengths and weaknesses:

  • ApacheBench (or ab) is a command-line program that is used to test servers with HTTP. It must be run using a cron-style scheduler, and delegated by a shell script that handles creation and post-execution cleanup of a data set. A shell script would need to be created that generated keycard events based on a range of values kept in a flat file that ApacheBench would consume. The flat file would need to change after every execution of ApacheBench.
  • Joe Dog Software's Siege is a utility that is used to do HTTP load testing and benchmarking. It was considered because of its apparent support for payload templates. That would have allowed our developer to skip the payload generation and deletion steps.
  • Apache JMeter is a Java-based load-testing tool that is used to measure performance for many different types of servers, including HTTP. Though our developers found it to be somewhat unwieldy and not very user friendly, it did allow for the creation of a dynamic payload without the need to write and maintain a separate codebase. Basically, we could configure the tool, run it, and be done. In fact, JMeter allows users to easily run a test indefinitely, rather than a one-and-done script execution cycle.

Ultimately, we went with Apache JMeter. Its ability to dynamically generate event payloads was the deciding factor.

DEV We have included Apache JMeter in the PAS app's test repo in the pas_http_input_keycard_simulator directory. Be sure you have Java installed before installing it. See the README file for more information. When you run JMeter, note that the configuration options are in the left navigation tree. It's been set up to run with the PAS app. The only thing that you will need to change are the hostname and token values. For our testing, we used the values that are in the following four files: HTTPInputTest.jmx, building_names.csv, location.csv, usernames.csv. Of course, you should feel free to adjust values and change the files as you see fit.

Apache JMeter interface

Using JMeter, we generated hundreds of events per minute, getting random values for building name, location (latitude and longitude), and more. JMeter was able to spin up as many threads as we wanted (within reason given hardware constraints).

Apache JMeter HTTP Request details window

The format of the events that JMeter generated was as follows:

{ 
    "user": "<username>", 
    "lat": "<latitude>", 
    "long": "<longitude>", 
    "timestamp": "<unix epoch timestamp>", 
    "type": "<cabinet | door>", 
    "security_level": [0-3], 
    "building_name": "<building_name>", 
    "floor": [0-9] 
}

Here's a sample event, viewed in Splunk Enterprise search:

Splunk Enterprise event detail

Visualizing the data

HTTP Event Collector came through, accepting and indexing the data exactly as it should have. But now that we had the data, what was the best way to transform it? Given how integral location was to our data (Where did the keycard access occur?), how could we best summarize and display it?

Another new feature in Splunk Enterprise 6.3, geographic visualizations using choropleth maps, was ideal for this requirement. In particular, the ability to display data on a map using proportional shading could help enormously in visually tracking keycard access over time, spikes in access events, and so on.

In addition to choropleth maps, available geographic visualizations in Splunk Enterprise now include map visualizations that represent data as points of different sizes on a map to show relative event volumes, plus non-map geographic visualizations such as bar charts that show comparative metrics by city or country.

If your data contains location information for each event, you can create a geographic visualization. Since our data contained latitude and longitude values, we were able to relatively easily create a map visualization.

The options for geographic visualizations are numerous, so we'll refer you to the step-by-step Splunk documentation on how to build a geographic query for the details. To see the dashboard we built into the PAS app, click Keycard Activity in the navigation bar at the top:

Choropleth map example

Inspect each dashboard panel to see the exact search commands we used. In particular, pay attention to how we used the geom search command, which is the piece that actually generates the map.

The telemetry scenario

Recall from earlier in this chapter our telemetry scenario:

Our business stakeholders were developing a web browser-based app and wanted to use Splunk Enterprise to collect and organize both debug and usage data for the app. Because the data could be coming from anyone, anywhere in the world, they didn't want it to contain any Splunk-specific information, and certainly didn't want it to include any Splunk Enterprise credential information.

To summarize, our requirements for this scenario were twofold:

  • Implement browser-based debug logging to Splunk Enterprise. Consider a developer who has created a web app that generates logging statements to the JavaScript console. How can we send these messages with HTTP to a Splunk Enterprise instance?
  • Implement browser-based user interaction telemetry to Splunk Enterprise. Consider a developer who wants to know what a user is doing once he or she logs in to a web app with their browser. What does the user click on first? In what order do users choose options and complete tasks? And so on...

To demonstrate how this can be done, we considered creating a dummy web app that just logs debug statements and user activity and then sends it to HTTP Event Collector. But then we realized that we've already got a prime candidate for this type of feature: the PAS app itself.

Wiring up events

The first thing to do was to ensure that the PAS app's debugging and user interaction events were sufficiently wired to log the data we needed. The bulk of the work was done in the setup.js file that you can find at pas_ref_app/appserver/static/setup.js.

When you take a look at the file, you'll see quite a few calls to either the sendDevLog or sendUxLog JavaScript functions. Let's take a look at the method definitions in code.

The sendDevLog function is attached to events that would be of interest to a developer trying to debug the app. For example, the following code snippet is executed when the setup page's check for the Eventgen tool returns an error. In that case, the app calls sendDevLog with the error message as its parameter:

...
            var eventgenApp = apps.item('eventgen')
            if (eventgenApp) {
                eventgenApp.fetch(function(err, eventgenApp) {
                    if (err) {
                        sendDevLog(err);
                        console.error(err);
                        return;
                    }
...

We also wired pertinent status messages to be logged using the sendDevLog function, such as the following status message for the Google Drive add-in:

...
            // Show the Google Drive app configuration section if the app is present and enabled
            var googleDriveApp = apps.item('googledrive_addon');
            if (googleDriveApp && !googleDriveApp.state().content.disabled) {
                sendDevLog( "Enabling Google Drive Add-on in Setup interface.")
                $('#googleDriveModule').removeClass('hide');
            }
...

The sendUxLog function is attached to events that occur in response to user actions—both status messages and error messages. This includes entering values into fields, choosing values from pop-up menus, opening windows such as the Google authentication window, entering invalid or missing values, and so on. For example, the following code snippet sends a status message when a user enters a weight value in the Violation types section of the Setup page:

...
                sendUxLog("Saving " + viol_title + ": user selected weight: " + viol_weight);
... 

The following snippet shows sendUxLog executing if, after the user clicks Save, the app isn't able to save its configuration successfully. Note that, because this is purely UX logging, the function just sends a message that indicates what the user was trying to do and that the operation was unsuccessful. It does not include any debugging information.

...
                sendUxLog("Unable to save changes!");
...

Sending the data

Now let's take a look at how we wrote these functions. As you can see, both the sendDevLog and sendUxLog functions end up calling the same logging function, sendLog. The difference is that sendDevLog calls it using the developer logging-specific token (dev_debug_key), while sendUxLog calls it using the UX logging-specific token (ux_logging_key):

    function sendUxLog(message) {
        if(dev_debug_key.length>0) {
            sendLog(message, ux_logging_key);
        } else {
            console.log("UX Logging - No HTTP Input Key has been set!  Event logging disabled.");
        }
    }
 
    function sendDevLog(message) {
        if(dev_debug_key.length>0) {
            sendLog(message, dev_debug_key);
        } else {
            console.log("Dev Debug - No HTTP Input Key has been set!  Event logging disabled.");
        }
    }

Here's the definition of the sendLog function from the code:

    function sendLog(message, authCode) {
        var http_request = new XMLHttpRequest();
        var http_input_url = "http://mysplunkserver:8088/services/collector";
        // Using lower-level call to XMLHttpRequest due to issues with how
        // JQuery handles CORS requests
        var xhr = createCORSRequest('POST', http_input_url);
        xhr.setRequestHeader('Content-Type', 'text/plain');
        xhr.setRequestHeader('Authorization', 'Splunk ' + authCode);
 
        var log_message = {
            "event":
                {
                    "username": currentUser,
                    "message": message
                }
            };
 
            xhr.send(JSON.stringify(log_message));
  }

This code is pretty straightforward. First, it specifies the HTTP Event Collector URL (<hostname>:<port>/services/collector). Then, it calls the createCORSRequest function to create a new XMLHttpRequest (XHR) object. The createCORSRequest function was necessary (as opposed to just creating an XHR object by calling XMLHttpRequest directly) because of cross-site scripting issues that we discuss in "Problems encountered," later in this chapter. The function then assembles the event by creating the authorization header and then forming the JSON-format message. It then sends the message.

Viewing the data

Let's get interactive with the last portion of this section and try out HTTP Event Collector logging from the PAS app right now. This procedure assumes that you've installed the PAS app on your local Splunk Enterprise instance, and that you've turned on EC as described earlier in this chapter.

  1. Create two new tokens and name them according to the type of data we'll be receiving. For instance, we named ours "pas:dev:debug" and "pas:ux:logging." Choose the "pas" index as the default index, and choose a source type that accurately describes the data. We chose the existing "ri:pas:application" source type.
  2. Open the setup.js file (at pas_ref_app/appserver/static/setup.js) using a text editor. At the top of the file, paste the corresponding token values between the quotation marks next to the following variable declarations:
    var dev_debug_key = "";
    var ux_logging_key = "";
    
  3. Within the sendLog function near the bottom of the file, replace the value within the quotation marks next to the http_input_url variable with your server's HTTP Event Collector endpoint (<hostname>:<port>/services/collector).
  4. Save the file, and then restart Splunk Enterprise. Alternatively, you can call the debug/refresh endpoint and then clear your browser's cache or force-refresh the page so that your browser retrieves the updated script.
  5. Open the PAS app and then click Setup in the top navigation bar. Then log some events. That is, enter some in-range or out-of-range values in fields, choose options from pop-up menus, and so on. Finally, click Save. PAS app setup page
  6. Click the Search button from the top nav bar, and enter a search that will find this specific data. For instance, since we assigned the logging data a source type of "ri:pas:application," we used the following query:
    index=pas sourcetype="ri:pas:application"

The search results you see should look something like the following.

PAS app setup page

In these results you see somewhat of a play-by-play of what you just did on the Setup page.

Note: If you don't see anything when you perform the search, read ahead to the "Problems encountered" section and make the .conf file additions described there. You are probably running up against cross-site scripting (XSS) restrictions.

Keep in mind that, even though we added this capability to the PAS app, which is a Splunk Enterprise app, you are in no way limited to doing this with just Splunk Enterprise apps. You can add the capability to log data to HTTP Event Collector to any browser-based app.

ADMIf you are familiar with Splunk Enterprise forwarders, you might be asking why you should bother with EC when a forwarder will do the same thing. While a forwarder could certainly be used to forward JavaScript console debug data to Splunk Enterprise, the UX component of this scenario is something that simply can't be accomplished using a forwarder. A forwarder can't capture this type of granular data. Even if you were running a forwarder on the backend to monitor web logs, you wouldn't get this level of detail, because you would only get the requests that users made, not a record of everything the user did in the browser.

Problems encountered

Getting HTTP Event Collector up and running and logging meaningful data was pleasantly quick and easy, but one issue did present some significant challenges we had to overcome: cross-site scripting (XSS) restrictions.

Our developer immediately encountered XSS issues when trying to send data with JavaScript to EC. A domain is the combination of a URI and a port number. The JavaScript that was served by the PAS app through Splunk Enterprise came from port 8000 by default. The default EC port is 8088.

ADMThe EC port is 8088 by default. You can change it by editing its global settings, but be aware of other services that might already be using whatever port number you change it to. For example, it cannot be the same as Splunk Web (8000) or the Splunk Enterprise REST API (8089).

This means that a POST to EC is interpreted as an entirely different domain. You could use a self-signed certificate to authenticate, but while that will work in browsers such as Firefox or Safari, it won't work in Chrome, which requires a valid certificate authority (CA)-signed certificate. To avoid the need to recommend one browser over another, not to mention the hassle and expense of purchasing a signed certificate, we searched for alternatives.

We could have turned off SSL encryption and sent data through HTTP versus HTTPS, but we didn't want to do anything security-related in our reference app that we wouldn't recommend doing in a production environment. We definitely do not recommend sending data to EC without encryption.

The next option was to leverage support built into Splunk Enterprise for cross-origin resource sharing (CORS). To enable support for CORS, we first had to make adjustments in the Splunk Enterprise server.conf and web.conf files:

  • We added a server.conf file to $SPLUNK_HOME/etc/apps/pas_ref_app/local/, and pasted the following two lines into it:
    [httpServer]  
    crossOriginSharingPolicy = * 

  • We added a web.conf file to $SPLUNK_HOME/etc/apps/pas_ref_app/local/, and pasted the following two lines into it:
    [settings]  
    crossOriginSharingPolicy = * 

However, even with CORS functionality enabled in Splunk Enterprise, we ran into issues when attempting to use jQuery to send a JSON payload to EC. It turns out that Splunk Enterprise uses custom JavaScript code that modifies the HTTP headers we were attempting to send and that effectively ruined any attempt to communicate using jQuery.

Because the jQuery library is simply a wrapper around the XMLHttpRequest object, building our own request from scratch was the next obvious step. This is how the createCORSRequest function that we mentioned in the "Sending the data" subsection of the telemetry scenario came about:

      // Create the XHR object.
      function createCORSRequest(method, url) {
            var xhr = new XMLHttpRequest();
            if ("withCredentials" in xhr) {
                  // XHR for Chrome/Firefox/Opera/Safari.
                  xhr.open(method, url, true);
            } else if (typeof XDomainRequest != "undefined") {
                  // XDomainRequest for IE.
                  xhr = new XDomainRequest();
                  xhr.open(method, url);
            } else {
                  // CORS not supported.
                  xhr = null;
            }
            return xhr;
      }

For more information about the server.conf file's format, see dev.splunk.com/goto/serverconf. For more information about the web.conf file's format, see dev.splunk.com/goto/webconf.

SECOvercoming XSS issues is essential to be able to use EC effectively. Hopefully our experiences will help make yours easier.
 

What have we learned?

Here are the key lessons we have learned while working with HTTP Event Collector:

  • HTTP Event Collector (or EC) is a great new way to get data directly into Splunk Enterprise from network sources.
  • EC sends data in JSON format through HTTP or HTTPS to port 8088 on a Splunk Enterprise server.
  • You can also use a load balancer to distribute high volumes of EC data evenly to indexers.
  • EC is disabled by default, but can be enabled in the Data inputs section of Splunk Enterprise.
  • A unique token contained in the auth header of your data is what tells EC on Splunk Enterprise that the data is valid.
  • You can assign field values (such as source, sourcetype, index, and so on) to data based on the token the data contains in its header.
  • EC is great for sending high volumes of data to Splunk Enterprise, such as usage telemetry data.
  • You can also use EC to send debugging and user interaction data about your app directly to Splunk Enterprise.
  • Sending data to EC on Splunk Enterprise opens you up to cross-site scripting (XSS) restrictions.
  • Splunk Enterprise has built-in support for cross-origin resource sharing (CORS), which we leveraged to overcome the XSS restrictions we encountered.