Walkthrough: Configure a data input with checkpoints

This walkthrough shows you how to configure a data input that uses checkpointing to enable incremental data collection. The goal of this example is to build an add-on that monitors the latest New York Times articles that correspond to keywords that you pass as arguments when creating data inputs from this configuration.

Depending on the type of REST API you are using, checkpoints can be useful because they mark where data has already been read and indexed. A useful API is one that accepts an argument for the date or record ID, and allows your input to select a few records at a time based on an incremental checkpoint. This checkpoint indicates where to resume reading data the next time it is collected so that the same data is not collected again, duplication is avoided, and queries are smaller.

Prerequisite  This walkthrough example uses the New York Times REST API, which is available on the The New York Times Developer Network. Before you continue, be sure to request an API key. Copy this API key, which you'll need later.

Let's begin with 1. Create an add-on.