Use lookups with KV Store data

To use lookups with KV Store data, you must first define a lookup, and then use search commands.

Define a lookup

To use search lookups with KV Store data, define a lookup for each KV Store collection in the transforms.conf configuration file in your app's /default or /local directory (for example, $SPLUNK_HOME/etc/apps/yourappname/default/transforms.conf).

Define a lookup stanza using the following settings in the transforms.conf file:

  • [lookup_name]: The name of the lookup.
  • external_type: The type of external command, which must be "kvstore".
  • collection: The name of the collection associated with this lookup.
  • fields_list: A comma- and space-delimited list of all fields that you want to make available in the search command.
  • filter: A string that contains a search query with Boolean expressions and/or comparison operators. This query runs whenever you use this lookup, allowing you to retrieve only the data you need from the KV Store collection, which can improve performance when working with large collections.

The following example shows a "kvstorecoll_lookup" lookup for the "kvstorecoll" collection in the transforms.conf file:

[kvstorecoll_lookup]
external_type = kvstore
collection = kvstorecoll
fields_list = _key, CustID, CustName, CustStreet, CustState, CustCity, CustZip
filter = (CustID>500) AND (CustName="P*")
    Note  To be able to view the internal key ID field in lookups, add "_key" as a field in the fields_list. For more, see The key ID field.

Use search commands

Use the following search commands to work with KV Store lookups:

  • Use inputlookup to get search results from a KV Store collection.
  • Use outputlookup to write search results from the search pipeline into a specific KV Store collection.
  • Use lookup to match event data from earlier in the search pipeline to data in a KV Store collection.
    Note  The REST API can access and update any data that belongs to the user ("username") or is shared ("nobody"). Lookups in the search language can access and update shared data only. Lookups cannot access or update user-specific data.

The key ID field

By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. When you use outputlookup to write to the KV Store, a key ID is autogenerated if you don't specify one explicitly. If you want to modify a specific record, you need to provide its key ID.

To view the key ID field in lookups, add "_key" to the fields_list in the lookup definition.

Filters and queries

When working with large KV Store collections, you might want to use a filter to retrieve only the data you need rather than reading the entire collection to improve search performance. You can filter KV Store collections in two ways:

  • Add a filter with a search query to the lookup definition in transforms.conf. This filter runs automatically whenever you use the defined lookup. Here's the syntax:
  • filter = <query>

    For example:

    filter = (CustID>500) AND (CustName="P*")
  • Use the where command with inputlookup with Boolean operators (=, !=, <, >, <=, >=, AND, OR, NOT) and wildcard string searches. Here's the syntax:
  • | inputlookup <lookup-name> where <eval-expression> | ...

    For example:

    | inputlookup kvstorecoll_lookup where (CustID>500) AND (CustName="P*") | ...

Multivalue fields

You can perform lookups that match values in multi-value fields. For example, let's say you have the following two records:

"CustID": 001, "CustName": "Marge Simpson", "CustZip": ["97478", "12345"], ...

"CustID": 002, "CustName": "Herman Munster", "CustZip": ["91101", "12345"], ...

The following lookup counts both instances of the multivalue "CustZip" field that contains "12345", even though the "CustZip" field in both records contains other values:

| stats count | eval CustZip="12345" | lookup kvstorecoll_lookup CustZip | table CustID, CustName, CustZip

Examples

To get data from the "kvstorecoll" collection using the "kvstorecoll_lookup" lookup:

| inputlookup kvstorecoll_lookup where CustState="WA" | stats count

To view the internal key ID ("_key" field), use the eval command as follows:

| inputlookup kvstorecoll_lookup | eval  CustKey = _key | table CustKey, CustName, CustStreet, CustCity, CustState, CustZip

To write data to the "kvstorecoll" collection using the "kvstorecoll_lookup" lookup:

index=sf_food_health sourcetype=sf_food_inspections name="Shalimar Restaurant" | outputlookup kvstorecoll_lookup
    Note  The outputlookup command does not support dots or dollar signs in field names, therefore it does not support dot notation, which is used for hierarchical JSON documents. If you need to write hierarchical data to a KV Store collection, use the REST API rather than a lookup.

To write the contents of a CSV file to the "kvstorecoll_lookup" lookup:

| inputlookup customers.csv | outputlookup kvstorecoll_lookup

To update field values in a particular record (_key ="544948df3ec32d7a4c1d9755"), use the eval command:

| inputlookup csvcoll_lookup | search _key=544948df3ec32d7a4c1d9755 | eval CustName="Marge Simpson" | eval CustCity="Springfield" | outputlookup csvcoll_lookup append=True

To match data to the "kvstorecoll_lookup" lookup:

... | lookup kvstorecoll_lookup CustID as r_id OUTPUT CustName as r_name | ...

For more about using KV Store lookups in Splunk Enterprise, see Configure KV store lookups in the Knowledge Manager Manual.