Ingestion

The Ingestion app allows you to explicitly review and manage data point ingestion into ITRS Analytics.

All data points require a configuration defining the order of their dimensions. Dimensions are the ordered names and values within a data point that uniquely identify the data point’s entity. As of Platform 2.0.0, all data points require a configuration defining the order of their dimensions. Dimensions are the ordered names and values within a data point that uniquely identify the data point’s entity.

ITRS Analytics ships with known dimensions already configured – all Geneos and internal data (Kubernetes) are configured out-of-the-box.

Monitoring Copied

Ingestion Throughput Copied

This section displays the totals and time-series charts for the following:

• Ingested data points — all processed and stored data points.
• Received data points — all data points sent to the system.
• Dropped data points — data points that are dropped due to validation or missing containment rules.
• Filtered data points — data points filtered in one of the active filters.

Data Volumes Copied

This section shows the size and usage of the volumes in a full ITRS Analytics cluster, including apps. Volumes exceeding 75% capacity are flagged with a warning, and those at or above 90% capacity are flagged as critical.

Workloads Copied

This section displays a table for all the containers running in the ITRS Analytics install, including apps.

If a container has restarts, the row shows in light red. When you click on a container row, it shows a drawer displaying the CPU and memory time-series charts, container details, and a Last Status table if there are restarts.

You can also filter containers by enabling the Hide LinkerD or Hide Non-Running options.

Kafka Metrics Copied

Tracer Latency Copied

This tab displays a chart showing the maximum latency values per minute over the past hour for all unique consumer group client and topic combinations.

Tracers are empty messages periodically sent through the raw metrics ingestion pipeline of the ITRS Analytics platform. The summary view shows the maximum value across all partitions, while the detailed view shows values for all partitions of a single consumer group and topic combination. Partitions are ordered by highest latency and a maximum of 24 partitions can be displayed at a time.

Consumer Group Lag Copied

This tab displays a table showing the consumer lag for all unique consumer group client and topic combinations.

The summary view shows the number of partitions for each topic, along with the minimum, maximum, and total lag, and the lag ratio; whereas the detailed view shows the lag for each partition within a single consumer group and topic combination.

You can also filter data by enabling the Hide Unknown Consumers or Hide Zero Lag options.

Note

Lag ratio is calculated as total_lag / log_end_offset.

Configuration Copied

Entity Configuration Copied

Dropped data points Copied

Note

Only users with admin privileges can configure dimensions.

Any unknown dimension is listed in the Dropped Entities tab. These data points are not processed by the Platform.

If you wish to see this data, you must configure the order of dimensions.

To configure the order of unknown dimension keys, click the Order icon in the right-most column. This opens the Order dimension keys dialog, where you can drag the keys into the desired order.

Note

To see a full JSON example of the data point, click Sample.

The dimensions are ordered by hierarchy groups so that you can configure these at the same time. For example, if the following dimensions are listed:

[a][b]
[a][b][c]
[a][b][c][d]

Then configuring the order of [a][b] will also allow you to configure the other dimensions with common keys. It is important that hierarchies are configured following a consistent key order.

Once you have configured one or more dimension keys, the checkbox for the item will be selected in the list. To save the changes, click Configure.

After saving your changes, the configured dimensions are moved from the Dropped Entities tab to the Known Entities tab.

Known entities Copied

There are two sections for resolved dimensions: Resolved and Pre-configured (Read only).

Resolved Copied

These are the dimensions that previously appeared in the Dropped Entities tab. If the items have pending status, it means that they have been configured but no new data has arrived matching these dimensions. Non-pending items are fully configured and working as expected.

To remove resolved dimensions, select the checkbox for the dimension keys, and then click the Remove button.

Note

Removing dimensions from the data points does not automatically populate them in the Dropped Entities tab. They will only appear if new data is published that uses those exact dimensions, which may be unlikely to happen.

Pre-configured (Read only) Copied

The Pre-configured (Read only) tab shows the pre-configured dimensions for known dimension keys. This information is provided for reference only.

Filtering Copied

Dimension Filters Copied

Filters allow you to exclude certain data points based on some matching rules.

1. From the Filtering tab, click the Add button to create a new filter.
2. Input an expression or select the filters.
3. Once the value is set, select one or more options in the Datapoint drop-down list to apply the filter to.
4. Click Save.

To modify or delete a filter, select the checkbox for the filter, and then click Edit or Remove.

Geneos Filters Copied

You can use multiple Geneos filters to manage data points from Geneos that are ingested into ITRS Analytics.

To add your desired Geneos filters, go to the Geneos Filter tab and click Add Filter. You can also reorder, modify, and delete filters in this tab.

All filters added become part of your Geneos Filter set. You can enable or disable single Geneos filters.

• If the filter set is enabled, at least one filter must explicitly include a data point for that data point to be retained. If no filter includes it, it will be excluded.
• If the filter set is disabled, no data points will be excluded.

Note

The filter set is designed to prioritize exclusion.

Filters are applied in the following order to determine which data points are retained or excluded:

• Each filter is analyzed individually, considering both its “include” and “exclude” selectors. Within a single filter, the “exclude” selector always takes precedence over the “include” selector. The order of these selectors does not matter.
• If a filter neither includes nor excludes a data point, it has no impact on the overall results.
• If a filter explicitly includes or excludes a data point, it overrides any previous filter’s results for that data point.
• If no filter specifically selects a data point, the data point is excluded.

Data Retention Policies Copied

This section displays data retention policies, including metric policies and non-metric policies, that are being used in the Platform. You can configure the retention duration for each non-metric policy by specifying a duration between one hour and 10 years.

Eviction Rules Copied

Entity eviction rules define when inactive entities and their data are automatically removed. In the Eviction Rules section, you can view all enabled rules, their conditions, types, and details. These are configurable by an admin user in the Ingestion app by clicking Eviction Rules > Add Rule.

There are two types of rules:

1. Fixed Expiration — configure an expiration time for the entity, where inactive entities are evicted after a fixed period of time. This indicates that the entity is retained if the last update is within this duration. For example:

   Entity A is configured with a 2-hour fixed expiration. It is removed two hours after its last update. If the last update was at 13:00, it will be evicted at 15:00; however, if it updates at 14:45, the timer resets and it will be evicted at 16:45.

2. Lifespan-based Expiration — entities are evicted once they have been inactive for a period of time based on how long they were active for.

   • Min Expiration — the minimum time an entity must remain after it becomes inactive before it can be evicted.
   • TTL percent — the percentage of the active lifespan of the entity to be added after inactivity. It determines how long to keep an entity after it goes inactive, based on how long it was active.
   • Max Expiration — optional, but must be greater than the minimum expiration.

   For example, Entity B is configured with the following settings:

   If Entity B was active for four hours and the last update was at 13:00, the computed TTL window is two hours (four hours × 50%). Since two hours falls within the minimum and maximum expiration times, the entity will be evicted at 15:00.

Note

The default eviction rules created during installation can also be restored. Restoring defaults removes all rules created after installation.