Using Forwarder Awareness, Forwarder Group owners can be alerted when Forwarders go missing. This is essential for ensuring that Splunk data streams stay stable and data flow is not interrupted. Using Atlas Forwarder Awareness, Splunk admins can normalize the alerting process for Forwarders, and easily add or drop Forwarders from being watched like never before. Admins can also mute individual Forwarders in a group for maintenance windows or identified instability.
Forwarder Awareness also alerts on a group by group basis, reducing alert fatigue by rolling up reports and adding much needed context to outages.
You must have Splunk admin permissions to follow these steps
The default behavior for alerting in Forwarder Awareness enables Splunk admins to quickly set up their reporting system. Admins need to identify what Forwarders they wish to track, and add them to a Forwarder Group. The admin should also ensure the contact information for the Forwarder Group is accurate. After this group is created, the Admin should then set the Forwarder Group priority 'High' or 'Critical'.
Now, when Forwarder Awareness detects a missing Forwarder in this Forwarder Group, the email listed in the Group configuration will be notified.
Forwarder Awareness checks for missing Forwarders once an hour, and will suppress alerts for 4 hours after it triggers.
Change Default Threshold Behavior
Default behavior requires a Forwarder Group to be labeled as a priority of 'High' or 'Critical' to trigger email alerts. This threshold can be lowered or raised to meet the Splunk team's needs. An admin will need to follow these steps to update the configuration:
In Forwarder Awareness, select Splunk Settings, and the 'Advanced Search' option.
Select 'Search macros'.
Select atlas_forwarder_alert_priority_threshold macro, ensure filters are selected to 'Owner: Any' & 'Created in App' to find this macro.
Change the value of the priority threshold by changing the default '1' to a new number. The priority thresholds are listed below in their numerical format.
0: Low Priority
1: Medium Priority
2: High Priority
3: Critical Priority
For example, if an admin wanted to lower the threshold so all admin created Forwarder Groups triggered alerts for missing Forwarders, then they can change the default '1' to a '-1'. If an admin wanted to raise the threshold so only Forwarder Groups prioritized as 'Critical' triggered, they could change the value to '2'.
The default value for the macro is shown below