Use configuration files to create a KV Store collection

To use KV Store, you must first create a KV Store collection.


Create a KV Store collection

To create a collection, create a collections.conf file in your app's /default or /local directory (for example, $SPLUNK_HOME/etc/apps/yourappname/default/collections.conf), then add a configuration stanza for each collection you want for your app.

This stanza specifies the name of the collection and optionally, the schema of the data. You don't have to explicitly define a schema because the KV Store infers it from the JSON document values. Enforcing data types is also optional, to be used when you want to:

  • Ensure that a given field is always treated as a specific data type.
  • Improve the collection's accelerations, because numbers take less space as integers than as strings, and records will be listed in the correct order.
  • Work with the REST API, because getting data into fields with the right types is important.

When data types are enforced, if the input data does not match the data type, the value is silently dropped.

Add collection stanzas using the following format and settings in collections.conf:

  • [collection_name]: The name of the collection as the stanza name.
  • enforceTypes: Optional. A Boolean that indicates whether to enforce the data type of values when inserting them into the collection. When true, an invalid value causes the record insertion to fail. When false, the field value is ignored but the record is inserted.
  • field.name: Optional. The data type ( number | bool | string | time ) of the specified field. If you don't explicitly set the data type, it is inferred from the JSON type.

After modifying the collections.conf file, you must reload the configuration file or restart Splunk Enterprise to create the new collection. For existing KV Store collections, these actions result in the KV Store verifying that any changes have been made, such as changes to accelerations.

Examples

At a minimum, all you need to create a KV Store collection is the stanza name. The following example shows how to define a "kvstorecoll" collection in collections.conf:

[kvstorecoll]

Let's say you want to enforce data types for the following sample JSON document:

{
    "name" : "Splunk Seattle",
    "id" : 123,
    "address_street" : "1730 Minor Avenue",
    "address_city" : "Seattle",
    "address_state" : "WA",
    "address_zip" : "98101"
}

The following example shows how you might define a "kvstorecoll" collection for these fields (where the field name follows the pattern field.variablename) and data types in collections.conf:

[kvstorecoll]
enforceTypes = true
field.name = string
field.id = number
field.address_street = string
field.address_city = string
field.address_state = string
field.address_zip = string

Work with hierarchical JSON documents

You can populate a KV Store collection with hierarchical JSON documents:

{
    "name" : "Splunk Seattle",
    "id" : 123,
    "address" :
    {
        "street" : "1730 Minor Avenue",
        "city" : "Seattle",
        "state" : "WA",
        "zip" : "98101"
    }
}

Use a dot notation to preserve the structure. For example, use the following collection definition for these fields and data types in collections.conf:

[kvstorecoll]
enforceTypes = true
field.name = string
field.id = number
field.address.street = string
field.address.city = string
field.address.state = string
field.address.zip = string
However, the outputlookup command does not support field names with dots or dollar signs, so you cannot write to this collection using lookups. Instead, use the REST API to add data to this collection. For an example of how to add hierarchical data to a collection, see Use the REST API to manage KV Store collections and data.


Accelerate fields

Accelerations are similar to database indexes on fields in that they help quickly locate data without having to search every event in a collection each time a search is run. You can create accelerations for one or more fields in a collection. However, accelerations slightly decrease insert performance and they impact disk space. The more accelerations you define, the more disk space is used.

To accelerate a field, define the acceleration using the following setting in the collection's stanza:

  • accelerated_fields.name: The name of the acceleration and its definition, which is the field to accelerate and a sorting order, in JSON format.

The following example shows the definition of a "my_accel" acceleration for the "id" field sorted in ascending order in collections.conf:

[kvstorecoll]
accelerated_fields.my_accel = {"id": 1}

A few notes about accelerations:

  • The order of accelerations is important. For example, an acceleration of { "a":1, "b":1 } speeds queries on "a" and "a" + "b", but not on "b" alone.
  • A combined acceleration speeds up queries better than multiple separate accelerations. For example, to speed up queries on "a" + "b", a combined acceleration { "a":1, "b":1 } is better than separate accelerations { "a":1 } and { "b": 1 }.
  • To delete an acceleration, remove the corresponding entry from the stanza and restart Splunk Enterprise.
  • Accelerated fields have a limitation of 1024 bytes per entry, so you cannot use fields that have more than 1024 bytes.

Replicate KV Store collections to indexers

By default, KV Store collections are not bundle-replicated to indexers, and lookups run locally on the search head rather than on remote peers. However, you might want to enable replication for a KV Store collection when:

  • You have large and fairly static KV Store collections with a large number of rows that are returned in lookups.
  • You have a large number of search peers (indexers) in your distributed environment.

To enable replication for KV Store collections and specify where to perform lookups, use the following settings:

  • To enable replication of a particular KV Store collection to indexers, set replicate to "true" in the collection's stanza in collections.conf. By default, the value is "false". (This setting is available in Splunk Enterprise 6.3 and later.)
  • To force a lookup to run on the search head only, set local to "true" in the lookup search command.

The table below summarizes the behavior for different combinations of these settings.

Setting local = true local = false
replicate = true The KV Store collection is replicated to indexers.
The lookup runs only on the search head.
The KV Store collection is replicated to indexers.
The lookup can run on indexers.
replicate = false The KV Store collection is not replicated to indexers.
The lookup runs automatically on the search head.
The KV Store collection is not replicated to indexers.
The lookup command returns an error because there is no KV Store data on the indexers.