Integrate a webhook with Splunk Observability Cloud
Webhook integrations contain authentication information and a custom payload that Splunk Observability Cloud sends in an HTTP request to the URL specified in the integration’s url
property.
Use a Webhook integration when you want to use a notification system for which Splunk Observability Cloud doesn’t provide a specific integration.
To learn more about parameters in the request and response bodies of the Webhook Integration API, see Integrations API.
When you send an API request, you need to use the endpoint specific to your organization's realm. To learn more, see Realms in endpoints.
Shared secret
The webhook integration uses an encryption key in the form of a string as the shared secret.
If the property is present and non-null, Splunk Observability Cloud does the following:
- Calculates a hash-based message authentication code (HMAC) for the notification message using the HMAC SHA256 cryptographic hash algorithm and the
sharedSecret
value as the cryptographic key. - Encodes the HMAC in base64.
- Stores the encoded result in the HTTP header
X-SFX-Signature
. Splunk Observability Cloud adds this header, so you don’t have to specify it in theheaders
property.
Do the following to verify notification messages in the webhook target system:
- Compute the HMAC SHA256 code for the incoming HTTP request payload using the
sharedSecret
key. - Encode the computed result in base64.
- Compare the encoded value to the value of
X-SFX-Signature
in the incoming HTTP request. If the values match, the message is valid.
Default payload
When you don't specify any customization for your webhook payload, the default payload is used. You can use the following template to get similar result.
JSON{ "severity": "{{{severity}}}", "originatingMetric": "{{{originatingMetric}}}", "detectOnCondition": "{{{encodeString detectOnCondition}}}", "detectOffCondition": "{{{encodeString detectOffCondition}}}", "messageBody": "{{{encodeString messageBody}}}", "src": "{{{src}}}", "inputs": {{{json inputs}}},
...
Default payload example
The following example shows the default payload for an alert when memory usage exceeds 90%.
JSON{ "sf_schema": 2, "detector": "A detector that alerts when memory usage exceeds 90% for 10 minutes", "description": "The value of memory.utilization is above 90.", "detectorUrl": "https://app.<REALM>.signalfx.com/#/detector/ABCDEFGHIJK/edit", "detectorId": "ABCDEFGHIJK", "incidentId": "BCDEFGHIJKL", "eventType": "exceedMemoryUse",
...
Custom payload variables
You can customize your payload to make sure responders have the context needed to resolve the issues. See the following table for the full list of variables available for customization.
- Unless otherwise indicated in the description, all variables are of type
string
.- When a variable is optional, it means it's not always available. If you use an optional variable that isn't available in the present conditions to construct your payload, Handlebars converts it to an empty string.
Variable | Description |
---|---|
detector | Name of the detector. |
detectorUrl | URL of the detector, which includes a parameter to select this specific incident. |
detectorId | ID of the detector. |
description | (Optional) Description of the rule for the detector. |
dimensions | A map of the dimensions in the input MTS. |
imageUrl | (Optional) URL of the alert preview image. |
incidentId | Unique identifier for this alert notification. |
eventType | Unique identifier for the version of the detector that sent the notification. |
rule | Name of the detector rule that triggered the alert. |
severity | Severity level of the rule determined by Splunk Observability Cloud. Use this variable in your custom payload. |
runbookUrl | Runbook URL specified in this rule. |
tip | Tip specified in this rule. |
messageTitle | Notification title for this rule. |
messageBody | Notification message for this rule. |
detectOnCondition | (Optional) Criteria for metric data and detection criteria for the detector rule, written in SignalFlow format. |
detectOffCondition | (Optional) Clear metric data and detection criteria in this rule, in SignalFlow format. |
status | Status of the incident, kept for backwards compatibility. Use statusExtended for more detailed information. Valid values include:- anomalous : the alert is firing because the detect conditions are met.- ok : the alert is cleared because the detect conditions are no longer met or the clear conditions, if any, are met. |
statusExtended | New variable for status of the incident. Use in place of status . Valid values include:- anomalous : the alert is firing because the detect conditions are met.- ok : the alert is cleared because the detect conditions are no longer met or the clear conditions, if any, are met.- manually resolved : a user resolves the alert through the UI or the API.- stopped : the detector that triggered the alert is edited or deleted. |
timestamp | Time the event occurred, in ISO 8601 format. |
inputs | Map of the inputs involved in this rule. This variable is of type array. For more information, see inputs array. |
sf_schema | The schema version for this event. The value is always set to 2 . |
orgId | The organization ID of the Splunk Observability org where the alert is coming from |
originatingMetric | Name of the metric that triggered the alert. |
src | The source of the alert, as defined by the following dimensions. The first dimension value available is set as the source of the alert. - k8s.pod.name - k8s.cluster.name - k8s.node.name - k8s.container.name - host - host.name - aws_arn - gcp_standard_id - azure_resource_id - sf_service - sf_workflow - sf_operation - test |
sloName | Only available for SLO alerts. Name of the SLO. |
sloId | Only available for SLO alerts. Unique ID of the SLO. |
sloService | Only available for APM SLO alerts. An APM service selected as the system health indicator for the SLO. |
sloEndpoint | Only available for APM SLO alerts. An endpoint for the selected APM service. |
sloEnvironments | Only available for APM SLO alerts. Environments where you want to track the SLO. |
sloAlertRule | Only available for SLO alerts. Alert rule type for the SLO. Valid values include: - BREACH - ERROR_BUDGET_LEFT - BURN_RATE |
inputs
array
Each object in the inputs
array is named after the program variable it’s bound to. If an object isn’t bound to a program variable, it uses a name like _S0
, _S1
, and so on.
Each input object contains the following elements:
Element | Description |
---|---|
key | (Optional) This is a map of the dimensions of the input signal. This element might be empty if there are no dimensions; for example, if the input was a static value and not a comparison against scalar values. |
value | Value of the input when the alert triggered or when it cleared. |
fragment | (Optional) This is the fragment of the SignalFlow program that represents the input. This element might not be present for some detectors or for static, anonymous inputs. |
Helper functions
Apart from using template variables to customize your payload, you can also use the following helper functions to make sure the right value is set on the outgoing alert.
Function | Description | Examples |
---|---|---|
coalesce | Returns the first value that is not empty. You can use this function to specify the desired order of dimensions for the src variable. | {{{coalesce dimensions.host dimensions.aws_arn ‘No ID Found!’}}} This example sets dimensions.host on the alert if it is present, dimensions.aws_arn as a second option, and the string “No ID Found!” if neither host nor AWS ID is present. |
severityDecoder | Returns 2 if the status variable is ok . Otherwise, the function checks and decodes the severity variable:- 1 : Info- 3 : Warning- 4 : Minor- 5 : Major- 6 : Critical- empty : Unknown severity | {{{severityDecoder ok='ok' Major='not_ok' default='empty'}}} This example overrides the default return values and customizes them. |
encodeString | Escapes quote and newline characters in a string. | {{{encodeString messageTitle}}} |
notEmpty | Adds text to a payload only if the value specified is not empty map. This function only works with the inputs and dimensions variables. | {{#notEmpty dimensions}} This example only prints if dimensions is not empty.{{/notEmpty}} |
json | Converts a string to a raw JSON value. Use this function to turn text from a detector into a JSON payload that can be evaluated in third-party integrations. | {{{json dimensions}}} |
substring | Returns a new character sequence that is a subsequence of this sequence. The subsequence starts with the character at the specified index and ends with the character at the second to last index. | {{substring var 1}} {{substring var 1 3}} |
abbreviate | Truncates a string if it is longer than the specified number of characters. Minimum abbreviation width is 4. | {{abbreviate long_str 5}} |
replace | Replaces each substring of this string that matches the literal target sequence with the specified literal replacement sequence. | {{replace abbreviated '...' ''}} |
eq | Checks if two elements are equal. | {{#eq a b}}yes{{else}}no{{/eq}} {{#eq a 2}}yes{{else}}no{{/eq}} |