.conf25 registration is now open!Register today

 Integrate a webhook with Splunk Observability Cloud

Webhook integrations contain authentication information and a custom payload that Splunk Observability Cloud sends in an HTTP request to the URL specified in the integration’s url property.

Use a Webhook integration when you want to use a notification system for which Splunk Observability Cloud doesn’t provide a specific integration.

To learn more about parameters in the request and response bodies of the Webhook Integration API, see Integrations API.

Info Circle

When you send an API request, you need to use the endpoint specific to your organization's realm. To learn more, see Realms in endpoints.

 Shared secret

The webhook integration uses an encryption key in the form of a string as the shared secret.

If the property is present and non-null, Splunk Observability Cloud does the following:

  1. Calculates a hash-based message authentication code (HMAC) for the notification message using the HMAC SHA256 cryptographic hash algorithm and the sharedSecret value as the cryptographic key.
  2. Encodes the HMAC in base64.
  3. Stores the encoded result in the HTTP header X-SFX-Signature. Splunk Observability Cloud adds this header, so you don’t have to specify it in the headers property.

Do the following to verify notification messages in the webhook target system:

  1. Compute the HMAC SHA256 code for the incoming HTTP request payload using the sharedSecret key.
  2. Encode the computed result in base64.
  3. Compare the encoded value to the value of X-SFX-Signature in the incoming HTTP request. If the values match, the message is valid.

 Default payload

When you don't specify any customization for your webhook payload, the default payload is used. You can use the following template to get similar result.

JSON
{
    "severity": "{{{severity}}}",
    "originatingMetric": "{{{originatingMetric}}}",
    "detectOnCondition": "{{{encodeString detectOnCondition}}}",
    "detectOffCondition": "{{{encodeString detectOffCondition}}}",
    "messageBody": "{{{encodeString messageBody}}}",
    "src": "{{{src}}}",
    "inputs": {{{json inputs}}},
...

 Default payload example

The following example shows the default payload for an alert when memory usage exceeds 90%.

JSON
{
  "sf_schema": 2,
  "detector": "A detector that alerts when memory usage exceeds 90% for 10 minutes",
  "description": "The value of memory.utilization is above 90.",
  "detectorUrl": "https://app.<REALM>.signalfx.com/#/detector/ABCDEFGHIJK/edit",
  "detectorId": "ABCDEFGHIJK",
  "incidentId": "BCDEFGHIJKL",
  "eventType": "exceedMemoryUse",
...

 Custom payload variables

You can customize your payload to make sure responders have the context needed to resolve the issues. See the following table for the full list of variables available for customization.

Info Circle
  • Unless otherwise indicated in the description, all variables are of type string.
  • When a variable is optional, it means it's not always available. If you use an optional variable that isn't available in the present conditions to construct your payload, Handlebars converts it to an empty string.
VariableDescription
detectorName of the detector.
detectorUrlURL of the detector, which includes a parameter to select this specific incident.
detectorIdID of the detector.
description(Optional) Description of the rule for the detector.
dimensionsA map of the dimensions in the input MTS.
imageUrl(Optional) URL of the alert preview image.
incidentIdUnique identifier for this alert notification.
eventTypeUnique identifier for the version of the detector that sent the notification.
ruleName of the detector rule that triggered the alert.
severitySeverity level of the rule determined by Splunk Observability Cloud. Use this variable in your custom payload.
runbookUrlRunbook URL specified in this rule.
tipTip specified in this rule.
messageTitleNotification title for this rule.
messageBodyNotification message for this rule.
detectOnCondition(Optional) Criteria for metric data and detection criteria for the detector rule, written in SignalFlow format.
detectOffCondition(Optional) Clear metric data and detection criteria in this rule, in SignalFlow format.
statusStatus of the incident, kept for backwards compatibility. Use statusExtended for more detailed information. Valid values include:
- anomalous: the alert is firing because the detect conditions are met.
- ok: the alert is cleared because the detect conditions are no longer met or the clear conditions, if any, are met.
statusExtendedNew variable for status of the incident. Use in place of status. Valid values include:
- anomalous: the alert is firing because the detect conditions are met.
- ok: the alert is cleared because the detect conditions are no longer met or the clear conditions, if any, are met.
- manually resolved: a user resolves the alert through the UI or the API.
- stopped: the detector that triggered the alert is edited or deleted.
timestampTime the event occurred, in ISO 8601 format.
inputsMap of the inputs involved in this rule. This variable is of type array. For more information, see inputs array.
sf_schemaThe schema version for this event. The value is always set to 2.
orgIdThe organization ID of the Splunk Observability org where the alert is coming from
originatingMetricName of the metric that triggered the alert.
srcThe source of the alert, as defined by the following dimensions. The first dimension value available is set as the source of the alert.
- k8s.pod.name
- k8s.cluster.name
- k8s.node.name
- k8s.container.name
- host
- host.name
- aws_arn
- gcp_standard_id
- azure_resource_id
- sf_service
- sf_workflow
- sf_operation
- test
sloNameOnly available for SLO alerts. Name of the SLO.
sloIdOnly available for SLO alerts. Unique ID of the SLO.
sloServiceOnly available for APM SLO alerts. An APM service selected as the system health indicator for the SLO.
sloEndpointOnly available for APM SLO alerts. An endpoint for the selected APM service.
sloEnvironmentsOnly available for APM SLO alerts. Environments where you want to track the SLO.
sloAlertRuleOnly available for SLO alerts. Alert rule type for the SLO. Valid values include:
- BREACH
- ERROR_BUDGET_LEFT
- BURN_RATE

 inputs array

Each object in the inputs array is named after the program variable it’s bound to. If an object isn’t bound to a program variable, it uses a name like _S0, _S1, and so on.

Each input object contains the following elements:

ElementDescription
key(Optional) This is a map of the dimensions of the input signal. This element might be empty if there are no dimensions; for example, if the input was a static value and not a comparison against scalar values.
valueValue of the input when the alert triggered or when it cleared.
fragment(Optional) This is the fragment of the SignalFlow program that represents the input. This element might not be present for some detectors or for static, anonymous inputs.

 Helper functions

Apart from using template variables to customize your payload, you can also use the following helper functions to make sure the right value is set on the outgoing alert.

FunctionDescriptionExamples
coalesceReturns the first value that is not empty. You can use this function to specify the desired order of dimensions for the src variable.{{{coalesce dimensions.host dimensions.aws_arn ‘No ID Found!’}}}
This example sets dimensions.host on the alert if it is present, dimensions.aws_arn as a second option, and the string “No ID Found!” if neither host nor AWS ID is present.
severityDecoderReturns 2 if the status variable is ok. Otherwise, the function checks and decodes the severity variable:
- 1: Info
- 3: Warning
- 4: Minor
- 5: Major
- 6: Critical
- empty: Unknown severity
{{{severityDecoder ok='ok' Major='not_ok' default='empty'}}}
This example overrides the default return values and customizes them.
encodeStringEscapes quote and newline characters in a string.{{{encodeString messageTitle}}}
notEmptyAdds text to a payload only if the value specified is not empty map. This function only works with the inputs and dimensions variables.{{#notEmpty dimensions}}
This example only prints if dimensions is not empty.
{{/notEmpty}}
jsonConverts a string to a raw JSON value. Use this function to turn text from a detector into a JSON payload that can be evaluated in third-party integrations.{{{json dimensions}}}
substringReturns a new character sequence that is a subsequence of this sequence. The subsequence starts with the character at the specified index and ends with the character at the second to last index.{{substring var 1}}
{{substring var 1 3}}
abbreviateTruncates a string if it is longer than the specified number of characters. Minimum abbreviation width is 4.{{abbreviate long_str 5}}
replaceReplaces each substring of this string that matches the literal target sequence with the specified literal replacement sequence.{{replace abbreviated '...' ''}}
eqChecks if two elements are equal.{{#eq a b}}yes{{else}}no{{/eq}}
{{#eq a 2}}yes{{else}}no{{/eq}}