Create a Node.js Lambda function for AWS IoT using a blueprint

This topic discusses how to get your AWS IoT data into Splunk Cloud by using AWS Lambda and HTTP Event Collector (HEC).

Note: Sending data from AWS to an on-premises Splunk Enterprise instance is not recommended, and could result in excessive egress costs and unacceptable latency. 

To get your IoT data into Splunk Cloud, you have to start from the end and work your way to the beginning. That is, you must set up the components of the data flow backwards, starting with HEC in Splunk Cloud and finishing with AWS IoT:

  1. Enable HEC in Splunk Cloud and create at least one token. For more information, see Use HTTP Event Collector in the Getting Data In Manual.

  2. Create a Lambda function, and include the HEC hostname and token in your code. Instructions for this step are included later in this topic.

  3. (Optional, for very high data volume) Configure Amazon Kinesis to queue data before sending it to AWS Lambda. For more information, see the Amazon Kinesis documentation.

  4. Create a rule within AWS IoT. AWS IoT rules specify where to send data. For very high data volume, for example, you'd set the rule to send data to Amazon Kinesis. Otherwise, you'd set the rule to send data directly to AWS Lambda. To create a rule in AWS IoT, you must first sign up for the AWS IoT Beta, at https://aws.amazon.com/iot/. Creating rules in AWS IoT is beyond the scope of this documentation; see the AWS Documentation to learn more.


Example scenario: AWS IoT to AWS Lambda to HTTP Event Collector

This example presents the simplest scenario: Data that goes from AWS IoT to AWS Lambda to HEC in Splunk Cloud.

Configure HTTP Event Collector

First, enable HEC and obtain a token. For more information, see Use HTTP Event Collector in the Getting Data In Manual, or complete the walkthrough.

Note: To enable HEC in managed Splunk Cloud, open a support request.

Configure AWS Lambda

Next, configure AWS Lambda to forward data to HEC. The easiest way to do this is by using a Lambda blueprint to create a Lambda function. The Lambda function contains information such as your Splunk Cloud hostname, the HEC token to use, information about how to tag and package your data to send to HEC, and more.

Note: To avoid excess egress costs and latency, run your Splunk Cloud instance in the same AWS Region as your Lambda function. For more information about AWS regions and regional endpoints, see Regions and Endpoints in AWS General Reference.

To use a blueprint to create a new Lambda function, you select the blueprint, and then edit the code. First, however, you'll need to do some prep work, including creating an AWS KMS key and updating the policy for the Lambda role.

Create an AWS KMS key

AWS Lambda uses the KMS to encrypt and decrypt your HEC token. You first use the KMS to encrypt your HEC token. Then you paste the encrypted token into the blueprint code. Finally, every time the Lambda function is invoked, AWS uses the KMS to decrypt the encrypted key and send the HEC token with the data. HEC sees the unencrypted valid token and accepts the data. Be aware that the KMS itself doesn't store your HEC token, just the key to decrypt the token.

  1. Create a new AWS Key Management Store (KMS) key. To learn how to create an AWS KMS key, see Creating Keys in the AWS Key Management Service Developer Guide. Once you've created the key, keep both its name and ARN value handy. You'll need both of them at different points later in this procedure.

  2. Use the AWS command-line interface (CLI) to encrypt the HEC token. For more information about the AWS CLI, see the AWS CLI User Guide. To encrypt the HEC token, use the following command, replacing <your KMS key name> with the name of the KMS key from the previous step (do not use the key value itself) and <splunk event collector token> with the HEC token you want to use:

    aws kms encrypt --key-id alias/<your KMS key name> --plaintext "<splunk event collector token>"
  3. The CLI will display a base-64 encoded encrypted token. Copy this value and keep it handy. You'll need this value when you edit the Lambda blueprint. 

Configure the AWS role

The final bit of prep before selecting a blueprint is ensuring that the AWS role you choose to run the blueprint has permission to decrypt your HEC token using the KMS key:

  1. From the AWS Management Console, under Security & Identity, click Identity & Access Management

  2. From the list on the left side of the window, click Roles.

  3. From the list of roles, click the lambda_basic_execution role.

  4. On the role's page under Permissions > Inline Policies, click Create Role Policy

  5. On the Set Permissions page, click Custom Policy, and then click Select.

  6. On the Review Policy page, give the policy a name, such as "HTTP Event Collector." 

  7. Paste the following JSON into the Policy Document field:

    { 
        "Version": "2012-10-17", 
        "Statement": [ 
        { 
            "Sid": "Stmt1443036478000", 
            "Effect": "Allow", 
            "Action": [ 
                "kms:Decrypt" 
                ], 
            "Resource": [ 
                "<your KMS key ARN>"  
                ] 
        } 
        ] 
    }  

    This JSON defines a policy that allows the role to use the KMS key to decrypt the token. It is also available in comments at the top of the splunk-logger blueprint code.

  8. Replace <your KMS key ARN> in the JSON with your KMS key's ARN value that you saved in Step 1 of "Create an AWS KMS key."

  9. Click Apply Policy.

For more information about roles and policies in AWS, see Creating IAM Roles and Examples of Policies for Delegating Access in the IAM User Guide.

Select the blueprint

Select the Splunk logging blueprint:

  1. Log onto Amazon Web Services, and under Compute, click Lambda.

  2. Click Create a Lambda function.

  3. From the Step 1: Select blueprint page, choose the splunk-logging blueprint. (You may have to look through several pages of blueprints to find splunk-logging.)

    Screen shot of the Step 2: Configure function page.
  4. On the Step 2: Configure function page, name your function and change the description to fit what you'll be doing with the function. The name should follow naming rules for a JavaScript function.

  5. In the Runtime pop-up menu, leave Node.js as the choice.

  6. Next, you're given the choice to either edit the code inline or upload a ZIP file. To use the blueprint, and if your code doesn't require any custom libraries beyond aws-sdk, use the inline editor.

Edit the code

After you choose the Edit code inline option, the editor appears with some pre-filled code. This code is a blueprint that sends some sample events to HEC in Splunk Cloud. You can edit this blueprint to add your own logic. Basic instructions are provided in comments at the top of the code, but if you've followed all the instructions in this topic up until this point, all you need to do is the following:

  • In the blueprint, find the following lines of code:
    // User code
    
    var loggerInfo = {
         splunkHost: 'https://<splunk host:port>/services/collector', // Fill in with your Splunk Cloud host IP/DNS and port (step 1 above)
         base64EncodedEncryptedToken: '', // Fill in with base64-encoded, encrypted HEC token here (step 5 above)
         lambdaFunctionName: '' // Fill in with your function name
    };

    Enter the values for the three variables:

    • For the splunkHost variable, replace <splunk host:port> with the full hostname, including correct protocol (HTTP or HTTPS) and HEC port. For instance, your Splunk Cloud address might look something like the following: https://ec2-xx-xxx-xxx-xx.us-west-2.compute.amazonaws.com:8088. Be sure to include the port you've assigned to HEC (8088 by default), and not the Splunk Web port (8000).
    • For the base64EncodedEncryptedToken variable, paste the base-64 encoded encrypted token that you saved previously (in "Create an AWS KMS key") between the single quotation marks next to the base64EncodedEncryptedToken variable in the code.
    • For the lambdaFunctionName variable, copy the name you typed in the Name field at the top of the page, and then paste it between the single quotes next to the lambdaFunctionName variable in the code.

If you examine the code in more detail, you'll see the following basic functionality. If you choose to create your Lambda function yourself, you'll need these basic components.

  1. We define a token object (initSplunkTokenAsync) to communicate with HEC. This token object takes advantage of AWS KMS to store the HEC token securely.

  2. We define a logger object (Logger) to assemble (using the log function) and send (using the logEvent and flushAsync functions) the events.

  3. In the "User code" section, we define the loggerInfo variable, into which you just entered values that correspond to your setup (that is, the splunkHost, base64EncodedEncryptedToken, and lambdaFunctionName variables). As you can see, this variable is used in numerous places throughout the Lambda function.

  4. At the end of the code, we instantiate two loggers:

    • The first logger (glogger) is a lifecycle logger, and is useful for indicating how often the Lambda function is initialized. Every time the function is first loaded, it logs a simple message ("'Loading function'"). Once the function is loaded, this logger does not run per invocation of the function. 
    • The second logger (logger), which is inside the handler, is invoked every time the handler is invoked. The handler function is passed two parameters: event, which contains the actual event data, and context, which contains contextual information such as the AWS request ID, which will be injected into the events sent to HTTP Event Collector. 

    Note: For more information about the Lambda function programming model in Node.js, see  Programming Model (Node.js) in the AWS Lambda Developer Guide.

    Neither logger actually sends data until the flushAsync function is called. Therefore, the second logger is simply batching the event data as it repeatedly calls the log function. Calling flushAsync sends the batched event data.

    The context object that was passed into the handler function has succeed and fail functions defined on it. You must call succeed when you call flushAsync, as it is expected by AWS Lambda. Otherwise, an error will be logged, even if the data was sent successfully. The succeed function also returns a result. In this case, it's the first event's data.

Save the Lambda function

  1. Under Lambda function handler and role, leave the Handler field set to index.handler

  2. From the Role pop-up menu, choose lambda_basic_execution under Use existing role.

  3. Leave the Advanced settings section as is, and then click Next.

  4. On the Review page, review your settings. If you have to make any changes, click the Edit button. When you're done, click Create function.

Configure AWS IoT

Create a rule within AWS IoT to send data directly to AWS Lambda. To learn how to create AWS IoT rules, first sign up for the AWS IoT Beta, and then see the AWS IoT Documentation. You can't create a rule to send data to a Lambda function that doesn't already exist, which is why this is the last step in the process instead of the first.