Define a checkpoint

In this step, we'll configure a checkpoint for the data input we have defined so far.

A checkpoint indicates where to resume gathering data the next time it is collected. Typically, a timestamp is used as a checkpoint. In this example, we will use the pub_date field from JSON response as our checkpoint.

To use checkpointing, the REST definition should do the following:

  • Sort by the checkpoint field, which is pub_date. We can do this by using the sort=latest parameter.
  • Be able to query data starting with the checkpoint pub_date, which is done by using the begin_date REST parameter.

Create Data Input wizard: Define & Test

  1. We've already added the sort parameter to our REST URL, so let's add the begin_date parameter so we can query for this parameter.
    In the REST URL under REST settings, append "&begin_date=20170124". The begin-date parameter will be automatically added to the list of REST URL parameters.
  2. Expand the Checkpoint settings section and select Enable checkpointing, then fill in the fields as follows:
    • Checkpoint parameter name: enter "begin_date" as the name of the checkpoint parameter.
    • Checkpoint field path: enter "response.docs[0].pub_date". This is a JSON path to the checkpoint field in the JSON payload. This option captures the pub_date value within the latest event in the JSON array.

      Note When testing the input, you can verify that the pub_date value that is captured is the latest value by comparing it to the other pub_date fields in the JSON data in the Output pane. This value is highlighted in blue.
    • Checkpoint initial value: enter "20170124". January 24, 2017 is the initial value of the checkpoint, using the format "YYYYMMDD" as specified in the Article Search API documentation.
    • If the formats of the response timestamp and request timestamp are the same, you would leave the following fields empty. However, because the request timestamp calls for a format such as "20170216", and the response timestamp returns values such as "2017-02-16T16:39:51+0000", you must specify the timestamps using the format "%Y-%m-%dT%H:%M:%S.%f+08:00" to normalize them.

    • Response timestamp format: enter "%Y-%m-%dT%H:%M:%S+%f".
    • Request timestamp format: enter "%Y%m%d".

    For more, see the Splunk Add-on Builder User Guide:

  3. Now that we have a specified an initial value for the checkpoint parameter in the Checkpoint settings, let's replace the value of the begin_date parameter in the REST settings.
    In the REST URL under REST settings, change the value of "&begin_date" to "${begin_date}".
  4. To verify the input is working, click Test.
  5. The Output pane displays the result of the request so you know it's working:

  6. Click Save, and then click Finish.

Your data input is complete and you're ready to try it out.

>>  Continue to View the add-on.