Skip to main content
Version: Atlas v3.10

Using Scheduling Inspector

Scheduling Inspector Dashboard

Scheduling Inspector is the sister Element to Scheduling Assistant. While Scheduling Assistant helps you schedule your Scheduled Searches to reduce concurrency risk, Scheduling Inspector inspects your search configurations to ensure they meet best practices. These configurations include your search time spans and ownership.

Using Scheduling Inspector with Splunk Search Peers
If you have configured Distributed Search Groups or Search Peers on your Splunk Environment, then an additional filter will appear under the KPI layer.

Key Metrics

Use this filter to select which Search Head, and the searches scheduled on them, the dashboard should use to populate its visualizations. The default selection 'localhost' will utilize the Search Head the user is currently on.

Scheduled Search Key Metrics

As with many Atlas Elements, Scheduling Inspector is headed by a few KPIs (Key Performance Indicators) that summarize information for quick analysis. Here, KPIs summarize how many searches miss data or overlap, and how much data is missed or over-searched.

Key Metrics

Search Coverage Gaps

Below the Key Metrics is the Search Coverage Gaps view. The searches with Coverage Gaps are not reviewing enough of the data in their time span when compared to their schedule. For instance, a search that is scheduled to run every 15 minutes but only looks at the past 5 minutes of data will ‘lose’ out on 10 minutes of data every time it runs. This is bad if the search is looking for critical errors or other notable events, as it will miss them entirely if it falls within this gap. Scheduling Inspector makes these searches easy to spot with color-coded highlighting.

Coverage Gaps

After identifying a search with a possible Coverage Gap, users can investigate further by selecting the Magnifying Glass to open the search on a new tab. This is useful for investigating if there were other reasons why this search didn’t meet best practices. Once confirmed as a viable target for fixing, users can select the Wrench icon for a modal that will enable them to automatically tune their search to meet best practices.

Wrench Icon 1

Clicking ‘Apply’ will apply the change in the Scheduled Search time range that will rectify the issue. This change is logged in the local directory of the appropriate application. It is recommended to not use this tool to change Enterprise Security searches or searches that result in an outputlookup command. It is beneficial to confirm the reasons for the search’s output and existence before updating the time span. By default, ES Helper Searches are excluded from the Scheduling Inspector output.

Wasteful Time Windows

Below the Search Coverage Gaps view is the Wasteful Time Windows report. These searches are a mirror image of the searches with Coverage Gaps.These searches have overlapping time windows that are searching the same data more than once. For instance, a search that is scheduled to run every 15 minutes but looks at the past 60 minutes of data will search the same bucket of events multiple times, wasting CPU resources and search slots. By keeping these searches in check, an environment can stay fast and efficient.

Wasteful Windows

After identifying a search with a possible Wasteful Time Window, users can investigate further by selecting the Magnifying Glass to open the search on a new tab. This is useful for investigating if there were other reasons why this search didn’t meet best practices. Once confirmed as a viable target for fixing, users can select the wrench icon for a modal that will enable them to automatically tune their search to meet best practices.

Wrench Icon 2

Selecting ‘Apply’ will update the Time Window of the Search to fix any overlaps based on the Scheduled Search cadence. This change will be saved on the local files of the corresponding application. It is recommended to not use this tool to change Enterprise Security searches or searches that result in an outputlookup command. It is beneficial to confirm the reasons for the search’s output and existence before updating the time span. By default, ES Helper Searches are excluded from the Scheduling Inspector output.

Orphaned Searches

The Orphaned Searches dashboard is used to keep track of searches that were created/owned by accounts that no longer exist. These are called Orphaned Searches and until they are assigned a new owner, they will not run. This could lead to missed alerts or broken dashboards. Scheduling Inspector allows the Admin to reassign these searches to another employee so they can run.

Inventory of Orphaned Reports

A list of Orphaned Searches (if there are any) will populate under the Inventory of Orphaned Report. In this report, Orphaned Searches are listed with additional information such as app source, permissions, and its Cron Schedule.

Inventory

After identifying an Orphaned Search that needs updating, selecting it will populate the Orphaned Search report below.

Individual Report

Users can click ‘View Search’ to investigate further, or can change the Owner, Sharing settings, and App location of the Search, and select ‘Save Changes’ to officially update the Scheduled Search. With these tools, admins can quickly identify and update Orphaned Searches so they get back to populating alerts and ensuring Splunk usability.