Documentation
Table of Contents
2 Item value preprocessing
Overview
Preprocessing allows to define transformation rules for the received item values. One or several transformations are possible before saving to the database.
Transformations are executed in the order in which they are defined. Preprocessing is done by Zabbix server or proxy (if items are monitored by proxy).
Note that the conversion to desired value type (as defined in item configuration) is performed at the end of the preprocessing pipeline; conversions, however, may also take place if required by the corresponding preprocessing step. See preprocessing details for more technical information.
See also: Usage examples
Configuration
Preprocessing rules are defined in the Preprocessing tab of the item configuration form.
An item will become unsupported if any of the preprocessing steps fails, unless custom error handling has been specified using a Custom on fail option for supported transformations.
For log items, log metadata (without value) will always reset item unsupported state and make item supported again, even if the initial error occurred after receiving a log value from agent.
User macros and user macros with context are supported in item value preprocessing parameters, including JavaScript code.
Context is ignored when a macro is replaced with its value. Macro value is inserted in the code as is, it is not possible to add additional escaping before placing the value in the JavaScript code. Please be advised, that this can cause JavaScript errors in some cases.
| Type | ||
|---|---|---|
| Transformation | Description | |
| Text | ||
| Regular expression | Match the value to the <pattern> regular expression and replace value with <output>. The regular expression supports extraction of maximum 10 captured groups with the \N sequence. Failure to match the input value will make the item unsupported. Parameters: pattern - regular expression output - output formatting template. An \N (where N=1…9) escape sequence is replaced with the Nth matched group. A \0 escape sequence is replaced with the matched text. Please refer to regular expressions section for some existing examples. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value or set a specified error message. |
|
| Replace | Find the search string and replace it with another (or nothing). All occurrences of the search string will be replaced. Parameters: search string - the string to find and replace, case-sensitive (required) replacement - the string to replace the search string with. The replacement string may also be empty effectively allowing to delete the search string when found. It is possible to use escape sequences to search for or replace line breaks, carriage return, tabs and spaces "\n \r \t \s"; backslash can be escaped as "\\" and escape sequences can be escaped as "\\n". Escaping of line breaks, carriage return, tabs is automatically done during low-level discovery. |
|
| Trim | Remove specified characters from the beginning and end of the value. | |
| Right trim | Remove specified characters from the end of the value. | |
| Left trim | Remove specified characters from the beginning of the value. | |
| Structured data | ||
| XML XPath | Extract value or fragment from XML data using XPath functionality. For this option to work, Zabbix server must be compiled with libxml support. Examples: number(/document/item/value) will extract 10 from <document><item><value>10</value></item></document>number(/document/item/@attribute) will extract 10 from <document><item attribute="10"></item></document>/document/item will extract <item><value>10</value></item> from <document><item><value>10</value></item></document>Note that namespaces are not supported. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error-handling options: either to discard the value, set a specified value or set a specified error message. |
|
| JSON Path | Extract value or fragment from JSON data using JSONPath functionality. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error-handling options: either to discard the value, set a specified value or set a specified error message. |
|
| CSV to JSON | Convert CSV file data into JSON format. For more information, see: CSV to JSON preprocessing. |
|
| XML to JSON | Convert data in XML format to JSON. For more information, see: Serialization rules. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error-handling options: either to discard the value, set a specified value or set a specified error message. |
|
| Arithmetic | ||
| Custom multiplier | Multiply the value by the specified integer or floating-point value. Use this option to convert values received in KB, MBps, etc into B, Bps. Otherwise Zabbix cannot correctly set prefixes (K, M, G etc). Note that if the item type of information is Numeric (unsigned), incoming values with a fractional part will be trimmed (i.e. '0.9' will become '0') before the custom multiplier is applied. Supported: scientific notation, for example, 1e+70 (since version 2.2); user macros and LLD macros (since version 4.0); strings that include macros, for example, {#MACRO}e+10, {$MACRO1}e+{$MACRO2}(since version 5.2.3)The macros must resolve to an integer or a floating-point number. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Change | ||
| Simple change | Calculate the difference between the current and previous value. Evaluated as value-prev_value, where value - current value; prev_value - previously received value This setting can be useful to measure a constantly growing value. If the current value is smaller than the previous value, Zabbix discards that difference (stores nothing) and waits for another value. Only one change operation per item is allowed. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Change per second | Calculate the value change (difference between the current and previous value) speed per second. Evaluated as (value-prev_value)/(time-prev_time), where value - current value; prev_value - previously received value; time - current timestamp; prev_time - timestamp of previous value. This setting is extremely useful to get speed per second for a constantly growing value. If the current value is smaller than the previous value, Zabbix discards that difference (stores nothing) and waits for another value. This helps to work correctly with, for instance, a wrapping (overflow) of 32-bit SNMP counters. Note: As this calculation may produce floating-point numbers, it is recommended to set the 'Type of information' to Numeric (float), even if the incoming raw values are integers. This is especially relevant for small numbers where the decimal part matters. If the floating-point values are large and may exceed the 'float' field length in which case the entire value may be lost, it is actually suggested to use Numeric (unsigned) and thus trim only the decimal part. Only one change operation per item is allowed. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Numeral systems | ||
| Boolean to decimal | Convert the value from boolean format to decimal. The textual representation is translated into either 0 or 1. Thus, 'TRUE' is stored as 1 and 'FALSE' is stored as 0. All values are matched in a case-insensitive way. Currently recognized values are, for: TRUE - true, t, yes, y, on, up, running, enabled, available, ok, master FALSE - false, f, no, n, off, down, unused, disabled, unavailable, err, slave Additionally, any non-zero numeric value is considered to be TRUE and zero is considered to be FALSE. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Octal to decimal | Convert the value from octal format to decimal. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Hexadecimal to decimal | Convert the value from hexadecimal format to decimal. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Custom scripts | ||
| JavaScript | Enter JavaScript code in the block that appears when clicking in the parameter field or on a pencil icon. Note that available JavaScript length depends on the database used. For more information, see: Javascript preprocessing. |
|
| Validation | ||
| In range | Define a range that a value should be in by specifying minimum/maximum values (inclusive). Numeric values are accepted (including any number of digits, optional decimal part and optional exponential part, negative values). User macros and low-level discovery macros can be used. The minimum value should be less than the maximum. At least one value must exist. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Matches regular expression | Specify a regular expression that a value must match. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Does not match regular expression | Specify a regular expression that a value must not match. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Check for error in JSON | Check for an application-level error message located at JSONpath. Stop processing if succeeded and the message is not empty; otherwise, continue processing with the value that was before this preprocessing step. Note that these external service errors are reported to the user as is, without adding preprocessing step information. No error will be reported in case of failing to parse invalid JSON. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Check for error in XML | Check for an application-level error message located at XPath. Stop processing if succeeded and the message is not empty; otherwise, continue processing with the value that was before this preprocessing step. Note that these external service errors are reported to the user as is, without adding preprocessing step information. No error will be reported in case of failing to parse invalid XML. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Check for error using a regular expression | Check for an application-level error message using a regular expression. Stop processing if succeeded and the message is not empty; otherwise, continue processing with the value that was before this preprocessing step. Note that these external service errors are reported to the user as is, without adding preprocessing step information. Parameters: pattern - regular expression output - output formatting template. An \N (where N=1…9) escape sequence is replaced with the Nth matched group. A \0 escape sequence is replaced with the matched text. If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value, or set a specified error message. |
|
| Check for not supported value | Check if there was an error in retrieving item value. Normally that would lead to the item turning unsupported, but you may modify that behavior by specifying the Custom on fail error-handling options: to discard the value, to set a specified value (in this case the item will stay supported and the value can be used in triggers) or set a specified error message. Note that for this preprocessing step, the Custom on fail checkbox is grayed out and always marked. This step is always executed as the first preprocessing step and is placed above all others after saving changes to the item. It can be used only once. Supported since 5.2.0. |
|
| Throttling | ||
| Discard unchanged | Discard a value if it has not changed. If a value is discarded, it is not saved in the database and Zabbix server has no knowledge that this value was received. No trigger expressions will be evaluated, as a result, no problems for related triggers will be created/resolved. Functions will work only based on data that is actually saved in the database. As trends are built based on data in the database, if there is no value saved for an hour then there will also be no trends data for that hour. Only one throttling option can be specified for an item. Note that it is possible for items monitored by Zabbix proxy that very small value differences (less than 0.000001) are correctly not discarded by proxy, but are stored in the history as the same value if the Zabbix server database has not been upgraded. |
|
| Discard unchanged with heartbeat | Discard a value if it has not changed within the defined time period (in seconds). Positive integer values are supported to specify the seconds (minimum - 1 second). Time suffixes can be used in this field (e.g. 30s, 1m, 2h, 1d). User macros and low-level discovery macros can be used in this field. If a value is discarded, it is not saved in the database and Zabbix server has no knowledge that this value was received. No trigger expressions will be evaluated, as a result, no problems for related triggers will be created/resolved. Functions will work only based on data that is actually saved in the database. As trends are built based on data in the database, if there is no value saved for an hour then there will also be no trends data for that hour. Only one throttling option can be specified for an item. Note that it is possible for items monitored by Zabbix proxy that very small value differences (less than 0.000001) are correctly not discarded by proxy, but are stored in the history as the same value if the Zabbix server database has not been upgraded. |
|
| Prometheus | ||
| Prometheus pattern | Use the following query to extract required data from Prometheus metrics. See Prometheus checks for more details. |
|
| Prometheus to JSON | Convert required Prometheus metrics to JSON. See Prometheus checks for more details. |
|
For change and throttling preprocessing steps Zabbix has to remember the last value to calculate/compare the new value as required. These previous values are handled by the preprocessing manager. If Zabbix server or proxy is restarted or there is any change made to preprocessing steps the last value of the corresponding item is reset, resulting in:
- for Simple change, Change per second steps - the next value will be ignored because there is no previous value to calculated change from;
- for Discard unchanged. Discard unchanged with heartbeat steps - the next value will never be discarded, even if it should have been because of discarding rules.
If you use a custom multiplier or store value as Change per second for items with the type of information set to Numeric (unsigned) and the resulting calculated value is actually a float number, the calculated value is still accepted as a correct one by trimming the decimal part and storing the value as an integer.
Testing
Testing preprocessing steps is useful to make sure that complex preprocessing pipelines yield the results that are expected from them, without waiting for the item value to be received and preprocessed.
It is possible to test:
- against a hypothetical value
- against a real value from a host
Each preprocessing step can be tested individually as well as all steps can be tested together. When you click on the Test or Test all steps button respectively in the Actions block, a testing window is opened.
Testing hypothetical value
| Parameter | Description |
|---|---|
| Get value from host | If you want to test a hypothetical value, leave this checkbox unmarked. See also: Testing real value. |
| Value | Enter the input value to test. Clicking in the parameter field or on the view/edit button will open a text area window for entering the value or code block. |
| Not supported | Mark this checkbox to test an unsupported value. This option is useful to test the Check for not supported value preprocessing step. |
| Time | Time of the input value is displayed: now (read-only). |
| Previous value | Enter a previous input value to compare to. Only for Change and Throttling preprocessing steps. |
| Previous time | Enter the previous input value time to compare to. Only for Change and Throttling preprocessing steps. The default value is based on the 'Update interval' field value of the item (if '1m', then this field is filled with now-1m). If nothing is specified or the user has no access to the host, the default is now-30s. |
| Macros | If any macros are used, they are listed along with their values. The values are editable for testing purposes, but the changes will only be saved within the testing context. |
| End of line sequence | Select the end of line sequence for multiline input values: LF - LF (line feed) sequence CRLF - CRLF (carriage-return line-feed) sequence. |
| Preprocessing steps | Preprocessing steps are listed; the testing result is displayed for each step after the Test button is clicked. If the step failed in testing, an error icon is displayed. The error description is displayed on mouseover. In case "Custom on fail" is specified for the step and that action is performed, a new line appears right after the preprocessing test step row, showing what action was done and what outcome it produced (error or value). |
| Result | The final result of testing preprocessing steps is displayed in all cases when all steps are tested together (when you click on the Test all steps button). The type of conversion to the value type of the item is also displayed, for example Result converted to Numeric (unsigned). |
Click on Test to see the result after each preprocessing step.
Test values are stored between test sessions for either individual steps or all steps, allowing the user to change preprocessing steps or item configuration and then return to the testing window without having to re-enter information. Values are lost on a page refresh though.
The testing is done by Zabbix server. The frontend sends a corresponding request to the server and waits for the result. The request contains the input value and preprocessing steps (with expanded user macros). For Change and Throttling steps, an optional previous value and time can be specified. The server responds with results for each preprocessing step.
All technical errors or input validation errors are displayed in the error box at the top of the testing window.
Testing real value
To test preprocessing against a real value:
- Mark the Get value from host checkbox
- Enter or verify host parameters (host address, port, proxy name/no proxy) and item-specific details (such as SNMPv2 community or SNMPv3 security credentials). These fields are context-aware:
- The values are pre-filled when possible, i.e. for items requiring an agent, by taking the information from the selected agent interface of the host
- The values have to be filled manually for template items
- Plain-text macro values are resolved
- Fields where the value (or part of the value) is a secret or Vault macro are empty and have to be entered manually. If any item parameter contains a secret macro value, the following warning message is displayed: "Item contains user-defined macros with secret values. Values of these macros should be entered manually."
- The fields are disabled when not needed in the context of the item type (e.g. the host address and the proxy fields are disabled for calculated items)
- Click on Get value and test to test the preprocessing
If you have specified a value mapping in the item configuration form ('Show value' field), the item test dialog will show another line after the final result, named 'Result with value map applied'.
Parameters that are specific to getting a real value from a host:
| Parameter | Description |
|---|---|
| Get value from host | Mark this checkbox to get a real value from the host. |
| Host address | Enter the host address. This field is automatically filled by the address of the item host interface. |
| Port | Enter the host port. This field is automatically filled by the port of item host interface. |
| Additional fields for SNMP interfaces (SNMP version, SNMP community, Context name, etc) |
See Configuring SNMP monitoring for additional details on configuring an SNMP interface (v1, v2 and v3). These fields are automatically filled from the item host interface. |
| Proxy | Specify the proxy if the host is monitored by a proxy. This field is automatically filled by the proxy of the host (if any). |
For the rest of the parameters, see Testing hypothetical value above.