Configuring Dynamic Thresholds

Compatibility matrix Copied

The following versions of the ITRS Analytics Platform and ITRS Analytics apps have been actively tested and confirmed to be compatible with Dynamic Thresholds:

Note

To use Dynamic Thresholds, these apps must be installed on ITRS Analytics. While other versions may work, compatibility is not guaranteed.

For the documentation of the compatible ITRS Analytics Platform versions, refer to the following:

• Platform version 2.6.1
• Platform version 2.7.0
• Platform version 2.8.0

Known issues Copied

The following versions of the ITRS Analytics Platform and ITRS Analytics apps have known issues and are not compatible with Dynamic Thresholds:

• Platform version 2.9.0
• Platform version 2.8.1
• API Gateway App version 2.4.0

Exporting data to ITRS Analytics Copied

Exporting data to ITRS Analytics requires configuring the Results Exporter in Opsview.

Create an ITRS Analytics User Copied

To create and configure users in ITRS Analytics, follow the instructions in Role-based access control (RBAC) page from the ITRS Analytics documentation.

To import Dynamic Thresholds from an ITRS Analytics instance, the ITRS Analytics Importer process user must be configured with the credentials of an ITRS Analytics user having appropriate permissions. For on-premises ITRS Analytics instances, creating a dedicated user for this purpose is recommended.

In this example, a user named dynamic_thresholds is created.

1. Log in to the ITRS Analytics instance’s Keycloak page as administrator.

2. Switch to the obcerv realm by selecting it from the dropdown menu.

3. Create a new user with the username dynamic_thresholds.

4. Add a password to the dynamic_thresholds user by navigating to User > Credentials > Set Password. Ensure the Temporary option is not selected.

5. Add the user role to the dynamic_thresholds user by navigating to User > Role Mappings > Assign Roles. You may need to filter by realm roles to see the user role.

   Note

   To keep the system secure, it is recommended to create a new policy in ITRS Analytics’s Keycloak to allow the dynamic_thresholds user access to only the necessary entities.

6. Navigate to Clients > obcerv-platform > Authorization > Policies, and then create the following client policies of user type. See examples in ITRS Analytics documentation for reference.

   Name:    allow-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Positive
   

   Name:    deny-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Negative
   

7. Navigate to the Authorization > Scopes tab, and then create the following scope:

   Name: hierarchy = [host, service]
   Display Name: Opsview Dynamic Thresholds rules
   

8. Navigate to the Authorization > Resources tab, and then open the entities resource and add the newly created authorization scope to the Authorization scopes field. You may need to add a display name to save changes.

9. Create a new permission to deny a user the read scope for the resource. Navigate to the Authorization > Permissions tab, and then click Create scope-based permission to provide the following:

   Name:                    deny-entities-read-scope
   Resource:                entities
   Authorization scopes:    read
   Policies:                deny-ov-dynamic-thresholds
   Decision Strategy:       Unanimous
   

10. At this point, the user has no available scopes and cannot view any metrics from the query service. So, you must add a permission to allow the authorization scope created above. Return to the Authorization > Permissions tab, and then click Create Permission > Create scope-based permission to provide the following:

    Name:                    allow-entities-for-ov-dynamic-thresholds
    Resource:                entities
    Authorization scopes:    hierarchy = [host, service]
    Policies:                allow-ov-dynamic-thresholds
    Decision Strategy:       Unanimous
    

Configure backend Copied

The Dynamic Thresholds backend is configured by Opsview Deploy. While several options are available for backend configuration, the following must be specified in the user variables file at /opt/opsview/deploy/etc/user_vars.yml.

# Hostname of the Analytics server to connect to for importing dynamic thresholds
opsview_dynamic_thresholds_analytics_host: analyticsinstance.domain.com

# Username of the Analytics user used to connect to the Analytics server
opsview_dynamic_thresholds_analytics_username: dynamic_thresholds

# Password of the Analytics user used to connect to the Analytics server
opsview_dynamic_thresholds_analytics_password: amFtaWVsdWNrZXR0NzA3

Optionally, you can enable or disable server certificate checking. This defaults to true, and it uses the system CA certificate store to verify the ITRS Analytics server certificate:

# Whether to check the CA certificate of the server
opsview_dynamic_thresholds_obcerv_verify_cert: false

Further optional configuration options are documented in /opt/opsview/deploy/etc/examples/user_variables-example.yml. Once the configuration is added, run the sys-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/sys-install.yml

Retrieve ITRS Analytics ingestion credentials Copied

You need to retrieve the ITRS Analytics ingestion password and ingestion certificate to configure the Results Exporter to export data to ITRS Analytics successfully.

Ingestion host and certificate Copied

Access the ITRS Analytics UI and navigate to Admin > Certificate.

You can extract the ingestion host from the present URL by removing the scheme and port sections; for example, my.host.com is the ingestion host taken from the URL https://my.host.com:123.

The ingestion certificate contents can then be copied and saved to a file on your orchestrator system. This file must be readable by the opsview user and in a valid PEM format.

Ingestion password Copied

You can retrieve the password for the ingestion-api user by running the following kubectl command against the ITRS Analytics cluster.

kubectl -n <analytics resources namespace> get secret iam-initial-credentials -o jsonpath='{.data.obcerv-ingestion-api}' | base64 -d

Configure Results Exporter in Opsview Copied

Dynamic Thresholds relies on metric timeseries from an ITRS Analytics instance. To export this data to an ITRS Analytics system, you must configure the Results Exporter component.

To export data to an ITRS Analytics instance, add the following configuration to your Results Exporter configuration file at /opt/opsview/deploy/etc/user_results_exporter.yml. If opsview_results_exporter_outputs is already defined, then add the analytics output to the existing configuration.

opsview_dynamic_thresholds_ingestion_host: analyticsinstance.domain.com
opsview_dynamic_thresholds_ingestion_password: your-ingestion-api-password

opsview_results_exporter_outputs:
  analytics:
    dynamic_threshold_metrics:
      type: dynamic_thresholds
      filter: '*'
      parameters:
        host: '{{ opsview_dynamic_thresholds_ingestion_host }}'
        port: 443
        cert_path: /path/to/analytics/ingestion/cert.pem
        username: ingestion-api
        password: '{{ opsview_dynamic_thresholds_ingestion_password }}'

Once the configuration is added, run the results-exporter-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/results-exporter-install.yml

Configure Service Checks Copied

Dynamic Thresholds are configured on a per-host-service basis. You can enable or disable Dynamic Thresholds for service checks through the Opsview Configuration > Dynamic Thresholds page.

This page is only visible to users with the ADMINACCESS permission. See the section Adding a New Role for more information. Additionally, the Dynamics Threshold configuration page will only show Host-Services the current Opsview user has access to.

When you first access the Dynamic Thresholds configuration page, you will not see any service checks listed. No host-services are enabled for Dynamic Thresholds by default.

1. To enable Dynamic Threshold for a service check or multiple service checks, use the search bar to find the desired services. Pressing ENTER will display a list of matching services in the table.

   

2. Select the service checks you want to enable Dynamic Thresholds for by clicking the checkbox, then click Apply threshold changes. To disable Dynamic Thresholds for a service, uncheck the box and click Apply threshold changes again.

   

3. Once enabled, the service check will be listed in the table. The Current Thresholds column displays the latest threshold data for the service check (or Awaiting Data if no data has been imported yet).

   

Limitations Copied

• Dynamic Thresholds are only available for service checks that return a single metric value.
• Dynamic Thresholds currently only support service checks that calculate thresholds as VALUE > THRESHOLD_VALUE. Less-than thresholds are not supported.
• Dynamic Thresholds only support service checks that include the Dynamic Thresholds macros: $WARNINGDT or $CRITICALDT as values for warning and critical threshold arguments.
  • Currently only a subset of the bundled Service Checks support Dynamic Thresholds.
  • Dynamic Threshold macros are written in the format $WARNINGDT;DEFAULTVALUE$, where DEFAULTVALUE is the default threshold value if Dynamic Thresholds are not available/not configured.
  • Optionally, you can also set a hard limit for Dynamic Threshold macros. This is defined as $WARNINGDT;DEFAULTVALUE;HARDLIMIT$, where HARDLIMIT will cap any calculated dynamic thresholds at its value. Note that hard limits only apply to checks with Dynamic Thresholds enabled and do not affect default values.
• Threshold prediction accuracy will improve based on the length of time data has been exported to ITRS Analytics. While one week of exported data is recommended before using Dynamic Thresholds, optimal accuracy is achieved with up to four weeks of data.
  • For newly exported host-service metric data, Dynamic Thresholds may not be available for a brief period until sufficient data is collected.

Troubleshooting Copied

Dynamic Thresholds configuration page is missing Copied

The Dynamic Thresholds configuration page may not be accessible in the navigation bar if the Dynamic Thresholds feature is not included in your Opsview license.

Please check Managing your Subscription/Checking your Subscription Details for more information.

Host-services are missing in the Dynamic Thresholds search Copied

Host-services must be compatible with the Dynamic Thresholds feature to appear in the configuration page search results. Compatibility is determined by the following criteria:

• Host-service must be returning performance metrics.
• Service check definition must contain one of the Dynamic Thresholds macros: $WARNINGDT or $CRITICALDT.

My enabled host-services are in an Awaiting Data state Copied

If a host-service’s dynamic threshold configuration is showing Awaiting Data, it’s possible that service metric data isn’t reaching ITRS Analytics. Verify that the correct host-services are being exported through the ITRS Analytics Web Console and check Opsview log files for any related errors.

Log message details Copied

These are common log messages you might see when configuring Dynamic Thresholds encounters an error or issue.

Nothing to do right now Copied

INFO [opsview.opsview.processors.obcervimporter] ObcervImporterProcessor checking objects ...
INFO [opsview.opsview.processors.obcervimporter] Nothing to do right now

All host-services using Dynamic Thresholds have been refreshed within the last min_object_update_interval_secs seconds, and there have been no changes to which host-services are enabled or disabled for this feature. The obcervimporter will perform another check later.

Failed to extract result for 'Host.Service (oid=707) metric: response = {} Copied

INFO [opsview.opsview.processors.obcervimporter] ObcervImporterProcessor checking objects ...
WARNING [opsview.obcervimporter.opsview.obcervimporter.obcervclient] Failed to extract result for 'hostA.'ServiceA' (oid=7) metricA': response={}
WARNING [opsview.obcervimporter.opsview.obcervimporter.obcervclient] Failed to extract result for 'hostB.'ServiceB' (oid=97) metricB': response={}

Note

As discussed in the Limitations section, Dynamic Thresholds work optimally with host-services that have at least a week’s worth of data in ITRS Analytics. For newly monitored host-services, sufficient data for threshold calculations might not yet be available. In such cases, the obcervimporter will periodically retry and log a warning until enough data is collected.