HTTP Event Collector metrics

HTTP Event Collector saves usage data about itself to log files. You can query these usage metrics using Splunk Enterprise or Splunk Cloud to explore usage trends system-wide, per token, per sourcetype, and more, as well as to evaluate HTTP Event Collector performance. Metrics are logged whenever HTTP Event Collector is enabled. HTTP Event Collector is disabled by default, so it will not log data until you enable it.

This topic contains the following sections:

Log file location and management

HTTP Event Collector metrics are written to the http_event_collector_metrics.log file located at the following path:

$SPLUNK_HOME/var/log/introspection/splunk/ 

A new http_event_collector_metrics.log file is created when you start your Splunk Enterprise instance (or log off of and then onto Splunk Cloud). Any existing file with that name is renamed by giving it next higher available numeric extension. For example, if you restart Splunk Enterprise or log off of and onto Splunk Cloud and there exists http_event_collector_metrics.log, http_event_collector_metrics.1, and http_event_collector_metrics.2 files, the http_event_collector_metrics.log file will be renamed http_event_collector_metrics.3 and HTTP Event Collector will begin logging to a new http_event_collector_metrics.log file.

You configure the logging frequency of HTTP Event Collector metrics in the limits.conf file. 60 seconds is the default frequency. HTTP Event Collector continues logging system-level metrics even when there is no data input activity. When there is no activity, you can expect about 200 kilobytes (KB) of metrics log data to be produced every 24 hours. The maximum size of a metrics log file is 25 megabytes (MB). If a log file reaches that limit, the log file is renamed as described in the previous paragraph and a new one is created. Up to five metrics log files can be stored at a time.

The props.conf file defines parameters for reading and indexing the metrics log file.

Querying HTTP Event Collector metrics data

HTTP Event Collector metrics data is indexed to the "_introspection" index. To query the accumulated HTTP Event Collector metrics using Splunk, you can use the following command:

index="_introspection" token

Metrics log data format

HTTP Event Collector metrics data is recorded to the log in JSON format. This means that the log is both easily human-readable and consistent with other Splunk Enterprise or Splunk Cloud log formats. A single entry consists of both input summary metrics and per-token metrics, as shown in the following example:

{
  "datetime": "05-04-2015 14:47:28.922 -0700",
  "log_level": "INFO",
  "component": "TokenInput",
  "data": {
    "token_name": "logger",
    "series": "httpinput_token",
    "transport": "http",
    "format": "json",
    "total_bytes_received": 238,
    "total_bytes_indexed": 20,
    "num_of_requests": 2,
    "num_of_events": 1,
    "num_of_errors": 0,
    "num_of_parser_errors": 1,
    "num_of_requests_to_disabled_token": 0,
    "num_of_requests_in_mint_format": 0
  }
}

{
  "datetime": "05-05-2015 08:21:27.002 -0700",
  "log_level": "INFO",
  "component": "TokenInput",
  "data": {
    "series": "httpinput",
    "transport": "http",
    "format": "json",
    "total_bytes_received": 0,
    "total_bytes_indexed": 0,
    "num_of_requests": 0,
    "num_of_events": 0,
    "num_of_errors": 0,
    "num_of_parser_errors": 0,
    "num_of_auth_failures": 0,
    "num_of_requests_to_disabled_token": 0,
    "num_of_requests_to_incorrect_url": 0,
    "num_of_requests_in_mint_format": 0
  }
}

Input summary metrics

System-wide summary metrics are always accumulated even if there is no input activity. These metrics are identified by the data:series field value of httpinput.

Field

Description

Value

component

HTTP Event Collector metrics data identifier.

TokenInput

data:format

HTTP Event Collector data format.

json

data:num_of_auth_failures

Total number of authentication failures due to invalid token.

unsigned integer

data:num_of_errors

Total number of per-token errors:

  • bad data format
  • no authorization
  • bad authorization
  • connectivity problems

unsigned integer

data:num_of_events

Total number of per-token events received by the HTTP Event Collector endpoint.

unsigned integer

data:num_of_parser_errors

Total number of per-token parser errors due to incorrectly formatted event data.

unsigned integer

data:num_of_requests

Total number of valid per-token individual HTTP(S) requests received by a HTTP Event Collector endpoint. Each request can have one or more data events.

unsigned integer

data:num_of_requests_to_incorrect_url

Total number of requests to an incorrect URL.

unsigned integer

data:num_of_requests_in_mint_format

Total number of requests from Splunk MINT.

unsigned integer

data:num_of_requests_to_disabled_token

Total number of per-token requests to disable token.

unsigned integer

data:series

Metrics data type.

httpinput

data:total_bytes_indexed

Total amount of per-token data sent to the indexer.

unsigned integer

data:total_bytes_received

Total amount of per-token data received by calling the /receive/token endpoint.

unsigned integer

data:transport

Data transport protocol for HTTP Event Collector data.

http

datetime

Date and time associated with the data. Format:
MM-DD-YYYY HH:MM:SS.SSS +/-GMTDELTA

string

log_level

Log severity level.

INFO

Per-token metrics

In contrast to the system-wide summary metrics, per-token metrics are accumulated only when HTTP Event Collector is enabled. These metrics are identified by a value of httpinput_token in the data:series field.

The [http_input] stanza in the limits.conf file defines the logging interval and maximum number of tokens logged for these metrics.

Field

Description

Value

component

HTTP Event Collector metrics data identifier.

TokenInput

data:format

HTTP Event Collector data format. (Always json for metrics logging.)

json

data:num_of_errors

Number of errors:

  • bad data format
  • no authorization
  • bad authorization
  • connectivity problems

unsigned integer

data:num_of_events

Number of events received by the HTTP Event Collector endpoint.

unsigned integer

data:num_of_parser_errors

Number of parser errors due to incorrectly formatted event data.

unsigned integer

data:num_of_requests

Number of valid individual HTTP(S) requests received by a HTTP Event Collector endpoint. Each request can have one or more data events.

unsigned integer

data:num_of_requests_in_mint_format

Total number of requests from Splunk MINT.

unsigned integer

data:num_of_requests_to_disabled_token

Number of requests to a disabled token.

unsigned integer

data:series

Metrics data type.

httpinput_token

data:token_name

Token name.

string

data:total_bytes_indexed

Total amount of data sent to the indexer.

unsigned integer

data:total_bytes_received

Total amount of data received by calling the /receive/token endpoint.

unsigned integer

data:transport

Data transport protocol for HTTP Event Collector data.

http

datetime

Date and time associated with the data. Format:
MM-DD-YYYY HH:MM:SS.SSS +/-GMTDELTA

string

log_level

Log severity level.

INFO


Configuration

The limits.conf and props.conf files control metrics data logging and indexing behavior.

limits.conf

The [http_input] stanza in the $SPLUNK_HOME/etc/system/default/limits.conf file controls HTTP Event Collector metrics data logging.

Note: For information about all HTTP Event Collector-related parameters, including those not related to metrics, see the [http_input] stanza documentation on limits.conf in the Splunk Enterprise Admin Manual.

Example

[http_input]
# the max number of tokens reported by logging input metrics
max_number_of_tokens = 10000
# the interval (in seconds) of logging input metrics report
metrics_report_interval = 60

Parameters

Parameter

Default value

Description

max_number_of_tokens

10000

An unsigned integer that represents the maximum number of tokens reported by HTTP Event Collector metrics.

metrics_report_interval

60

An unsigned integer that represents the number of seconds in an HTTP Event Collector metrics report interval.

props.conf

The [http_event_collector_metrics] stanza in the $SPLUNK_HOME/etc/system/default/props.conf file controls reading and indexing the HTTP Event Collector log files.

Examples

[source::.../http_event_collector_metrics.log(.\d+)?]
sourcetype = http_event_collector_metrics

...

[http_event_collector_metrics]
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = datetime
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%l %z
INDEXED_EXTRACTIONS = json
KV_MODE = none
JSON_TRIM_BRACES_IN_ARRAY_NAMES = true

Parameters

Parameter

Default

Description

SHOULD_LINEMERGE false

Specifies layout of events per line:

  • true = Allow multiple events in the same line.
  • false = Put multiple events in separate lines.

TIMESTAMP_FIELDS

datetime

Log entry time field name.

TIME_FORMAT

%m-%d-%Y %H:%M:%S.%l %z

Log entry time field format.

INDEXED_EXTRACTIONS

json

Metrics log format. (Always json for metrics logging.)

KV_MODE

none

Key-value data indicator:

  • none = No key-value data. (Always none for metrics logging.)

JSON_TRIM_BRACES_IN_ARRAY_NAMES

true

Whether to trim brace characters from JSON array names