8 Known issues

See also: Compilation issues.

Proxy startup with MySQL 8.0.0-8.0.17

zabbix_proxy on MySQL versions 8.0.0-8.0.17 fails with the following "access denied" error:

[Z3001] connection to database 'zabbix' failed: [1227] Access denied; you need (at least one of) the SUPER, SYSTEM_VARIABLES_ADMIN or SESSION_VARIABLES_ADMIN privilege(s) for this operation

That is due to MySQL 8.0.0 starting to enforce special permissions for setting session variables. However, in 8.0.18 this behavior was removed: As of MySQL 8.0.18, setting the session value of this system variable is no longer a restricted operation.

The workaround is based on granting additional privileges to the zabbix user:

For MySQL versions 8.0.14 - 8.0.17:

grant SESSION_VARIABLES_ADMIN on *.* to 'zabbix'@'localhost';

For MySQL versions 8.0.0 - 8.0.13:

grant SYSTEM_VARIABLES_ADMIN on *.* to 'zabbix'@'localhost';

Upgrade

SQL mode setting for successful upgrade

The sql_mode setting in MySQL/MariaDB must have the "STRICT_TRANS_TABLES" mode set. If it is absent, the Zabbix database upgrade will fail (see also ZBX-19435).

Upgrade with MariaDB 10.2.1 and before

Upgrading Zabbix may fail if database tables were created with MariaDB 10.2.1 and before, because in those versions the default row format is compact. This can be fixed by changing the row format to dynamic (see also ZBX-17690).

Templates

Template compatibility in dual-stack (IPv4/IPv6) environments

In dual-stack environments (systems configured to support both IPv4 and IPv6), the hostname localhost typically resolves to both IPv4 and IPv6 addresses. Due to the common prioritization of IPv6 over IPv4 by many operating systems and DNS resolvers, Zabbix templates may fail to work correctly if the service being monitored is configured to listen only on IPv4.

Services that are not configured to listen on IPv6 addresses may become inaccessible, leading to monitoring failures. Users might configure access correctly for IPv4 but still face connectivity issues due to the default behavior of prioritizing IPv6.

A workaround for this is to ensure that the services (Nginx, Apache, PostgreSQL, etc.) are configured to listen on both IPv4 and IPv6 addresses, and Zabbix server/agent is allowed access via IPv6. Additionally, in Zabbix templates and configurations, use localhost explicitly instead of 127.0.0.1 to ensure compatibility with both IPv4 and IPv6.

For example, when monitoring PostgreSQL with the PostgreSQL by Zabbix agent 2 template, you may need to edit the pg_hba.conf file to allow connections for the zbx_monitor user. If the dual-stack environment prioritizes IPv6 (system resolves localhost to ::1) and you configure localhost but only add an IPv4 entry (127.0.0.1/32), the connection will fail because there is no matching IPv6 entry.

The following pg_hba.conf file example ensures that the zbx_monitor user can connect to any database from the local machine using both IPv4 and IPv6 addresses with different authentication methods:

# TYPE     DATABASE     USER            ADDRESS          METHOD
         host     all          zbx_monitor     localhost        trust
         host     all          zbx_monitor     127.0.0.1/32     md5
         host     all          zbx_monitor     ::1/128          scram-sha-256

If necessary, you can also use the IPv4 address (127.0.0.1) directly when configuring the PostgreSQL by Zabbix agent 2 template macro for the connection string.

Accidental installation of EPEL Zabbix packages

With EPEL repository installed and enabled, installing Zabbix from packages will lead to EPEL Zabbix packages being installed rather than official Zabbix packages.

In this case uninstall Zabbix packages from EPEL, i.e.:

dnf remove zabbix-server-mysql

Block Zabbix packages from EPEL. Add the following line in the /etc/yum.conf file:

exclude=zabbix7.0*

Install Zabbix server again:

dnf install zabbix-server-mysql

Notice that official Zabbix packages have the word release in their version string:

7.0.0-release1.el8

Zabbix packages for RHEL on Red Hat UBI environments

When installing Zabbix from Red Hat Enterprise Linux packages on Red Hat Universal Base Image environments, ensure access to required repositories and dependencies. Zabbix packages depend on libOpenIPMI.so and libOpenIPMIposix.so libraries, which are not provided by any package in the default package manager repositories enabled on UBI systems and will result in installation failures.

The libOpenIPMI.so and libOpenIPMIposix.so libraries are available in the OpenIPMI-libs package, which is provided by the redhat-#-for-<arch>-appstream-rpms repository. Access to this repository is curated by subscriptions, which, in the case of UBI environments, get propagated by mounting repository configuration and secrets directories of the RHEL host into the container file-system namespace.

For more information, see ZBX-24291.

Expired signing key for RHEL packages

When upgrading Zabbix on Red Hat Enterprise Linux or its derivatives, you may encounter an expired signing key issue for packages on Zabbix repository. When a signing key expires, attempts to verify package signatures will result in an error indicating that the certificate or key is no longer valid. For example:

error: Verifying a signature using certificate D9AA84C2B617479C6E4FCF4D19F2475308EFA7DD (Zabbix LLC (Jul 2022) <[email protected]>):
         1. Certificiate 19F2475308EFA7DD invalid: certificate is not alive
             because: The primary key is not live
             because: Expired on 2024-07-04T11:41:23Z
         2. Key 19F2475308EFA7DD invalid: key is not alive
             because: The primary key is not live
             because: Expired on 2024-07-04T11:41:23Z

To resolve such issues, manually reinstall the latest zabbix-release package for your specific variant of RHEL (replace the link below with the correct one from Zabbix repository).

For example, on RHEL 9, run:

rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-latest.el9.noarch.rpm

Then, update the repository information:

dnf update

For more information, see ZBX-24761.

Database TLS connection with MariaDB

Database TLS connection is not supported with the 'verify_ca' option for the DBTLSConnect parameter if MariaDB is used.

Possible deadlocks with MySQL/MariaDB

When running under high load, and with more than one LLD worker involved, it is possible to run into a deadlock caused by an InnoDB error related to the row-locking strategy (see upstream bug). The error has been fixed in MySQL since 8.0.29, but not in MariaDB. For more details, see ZBX-21506.

Global event correlation

Events may not get correlated correctly if the time interval between the first and second event is very small, i.e. half a second and less.

Numeric (float) data type range with PostgreSQL 11 and earlier

PostgreSQL 11 and earlier versions only support floating point value range of approximately -1.34E-154 to 1.34E+154.

NetBSD 8.0 and newer

Various Zabbix processes may randomly crash on startup on the NetBSD versions 8.X and 9.X. That is due to the too small default stack size (4MB), which must be increased by running:

ulimit -s 10240

For more information, please see the related problem report: ZBX-18275.

Regular expression limitations in Zabbix agent 2

Zabbix agent 2 does not support lookaheads and lookbehinds in regular expressions due to the standard Go regexp library limitations.

IPMI checks

IPMI checks will not work with the standard OpenIPMI library package on Debian prior to 9 (stretch) and Ubuntu prior to 16.04 (xenial). To fix that, recompile OpenIPMI library with OpenSSL enabled as discussed in ZBX-6139.

SSH checks

• Some Linux distributions like Debian, Ubuntu do not support encrypted private keys (with passphrase) if the libssh2 library is installed from packages. Please see ZBX-4850 for more details.

• When using libssh 0.9.x on some Linux distributions with OpenSSH 8, SSH checks may occasionally report "Cannot read data from SSH server". This is caused by a libssh issue (more detailed report). The error is expected to have been fixed by a stable libssh 0.9.5 release. See also ZBX-17756 for details.

• Using the pipe "|" in the SSH script may lead to a "Cannot read data from SSH server" error. In this case it is recommended to upgrade the libssh library version. See also ZBX-21337 for details.

ODBC checks

• MySQL unixODBC driver should not be used with Zabbix server or Zabbix proxy compiled against MariaDB connector library and vice versa, if possible it is also better to avoid using the same connector as the driver due to an upstream bug. Suggested setup:

  PostgreSQL, SQLite or Oracle connector → MariaDB or MySQL unixODBC driver MariaDB connector → MariaDB unixODBC driver MySQL connector → MySQL unixODBC driver

See ZBX-7665 for more information and available workarounds.

• XML data queried from Microsoft SQL Server may get truncated in various ways on Linux and UNIX systems.

• It has been observed that using ODBC checks for monitoring Oracle databases using various versions of Oracle Instant Client for Linux causes Zabbix server to crash.  See also: ZBX-18402, ZBX-20803.

• If using FreeTDS UnixODBC driver, you need to prepend a 'SET NOCOUNT ON' statement to an SQL query (for example, SET NOCOUNT ON DECLARE @strsql NVARCHAR(max) SET @strsql = ....). Otherwise, database monitor item in Zabbix will fail to retrieve the information with an error "SQL query returned empty result".
  See ZBX-19917 for more information.

Incorrect request method parameter in items

The request method parameter, used only in HTTP checks, may be incorrectly set to '1', a non-default value for all items as a result of upgrade from a pre-4.0 Zabbix version. For details on how to fix this situation, see ZBX-19308.

Web monitoring and HTTP agent

Zabbix server leaks memory on some Linux distributions due to an upstream bug when "SSL verify peer" is enabled in web scenarios or HTTP agent. Please see ZBX-10486 for more information and available workarounds.

Simple checks

There is a bug in fping versions earlier than v3.10 that mishandles duplicate echo replay packets. This may cause unexpected results for icmpping, icmppingloss, icmppingsec items. It is recommended to use the latest version of fping. Please see ZBX-11726 for more details.

Errors with fping execution in rootless containers

When containers are running in rootless mode or in a specific-restrictions environment, you may face errors related to fping execution when performing ICMP checks, such as fping: Operation not permitted or all packets to all resources lost.

To fix this problem add --cap-add=net_raw to "docker run" or "podman run" commands.

Additionally fping execution in non-root environments may require sysctl modification, i.e.:

sudo sysctl -w "net.ipv4.ping_group_range=0 1995"

where "1995" is the zabbix GID. For more details, see ZBX-22833.

SNMP checks

If the OpenBSD operating system is used, a use-after-free bug in the Net-SNMP library up to the 5.7.3 version can cause a crash of Zabbix server if the SourceIP parameter is set in the Zabbix server configuration file. As a workaround, please do not set the SourceIP parameter. The same problem applies also for Linux, but it does not cause Zabbix server to stop working. A local patch for the net-snmp package on OpenBSD was applied and will be released with OpenBSD 6.3.

SNMP data spikes

Spikes in SNMP data have been observed that may be related to certain physical factors like voltage spikes in the mains. See ZBX-14318 more details.

SNMP traps

The "net-snmp-perl" package, needed for SNMP traps, has been removed in RHEL 8.0-8.2; re-added in RHEL 8.3.

So if you are using RHEL 8.0-8.2, the best solution is to upgrade to RHEL 8.3.

Please also see ZBX-17192 for more information.

Alerter process crash in RHEL 7

Instances of a Zabbix server alerter process crash have been encountered in RHEL 7. Please see ZBX-10461 for details.

Upgrading Zabbix agent 2 (6.0.5 or older)

When upgrading Zabbix agent 2 (version 6.0.5 or older) from packages, a plugin-related file conflict error may occur. To fix the error, back up your agent 2 configuration (if necessary), uninstall agent 2 and install it anew.

On RHEL-based systems, run:

dnf remove zabbix-agent2
       dnf install zabbix-agent2

On Debian-based systems, run:

apt remove zabbix-agent2
       apt install zabbix-agent2

For more information, see ZBX-23250.

Flipping frontend locales

It has been observed that frontend locales may flip without apparent logic, i. e. some pages (or parts of pages) are displayed in one language while other pages (or parts of pages) in a different language. Typically the problem may appear when there are several users, some of whom use one locale, while others use another.

A known workaround to this is to disable multithreading in PHP and Apache.

The problem is related to how setting the locale works in PHP: locale information is maintained per process, not per thread. So in a multi-thread environment, when there are several projects run by same Apache process, it is possible that the locale gets changed in another thread and that changes how data can be processed in the Zabbix thread.

For more information, please see related problem reports:

• ZBX-10911 (Problem with flipping frontend locales)
• ZBX-16297 (Problem with number processing in graphs using the bcdiv function of BC Math functions)

Graphs

Daylight Saving Time

Changes to Daylight Saving Time (DST) result in irregularities when displaying X axis labels (date duplication, date missing, etc).

Sum aggregation

When using sum aggregation in a graph for period that is less than one hour, graphs display incorrect (multiplied) values when data come from trends.

Text overlapping

For some frontend languages (e.g., Japanese), local fonts can cause text overlapping in graph legend. To avoid this, use version 2.3.0 (or later) of PHP GD extension.

Log file monitoring

log[] and logrt[] items repeatedly reread log file from the beginning if file system is 100% full and the log file is being appended (see ZBX-10884 for more information).

Slow MySQL queries

Zabbix server generates slow SELECT queries in case of non-existing values for items. This issue is known to occur in MySQL 5.6/5.7 versions (for an extended discussion, see ZBX-10652), and, in specific cases, may also occur in later MySQL versions. A workaround to this is disabling the index_condition_pushdown or prefer_ordering_index optimizer in MySQL. Note, however, that this workaround may not fix all issues related to slow queries.

Slow configuration sync with Oracle

Configuration sync might be slow in Zabbix installations with Oracle DB that have high number of items and item preprocessing steps. This is caused by the Oracle database engine speed processing nclob type fields.

To improve performance, you can convert the field types from nclob to nvarchar2 by manually applying the database patch items_nvarchar_prepare.sql. Note that this conversion will reduce the maximum field size limit from 65535 bytes to 4000 bytes for item preprocessing parameters and item parameters such as Description, Script item's field Script, HTTP agent item's fields Request body and Headers, Database monitor item's field SQL query. Queries to determine template names that need to be deleted before applying the patch are provided in the patch as a comment. Alternatively, if MAX_STRING_SIZE is set you can change nvarchar2(4000) to nvarchar2(32767) in the patch queries to set the 32767 bytes field size limit.

For an extended discussion, see ZBX-22363.

API login

A large number of open user sessions can be created when using custom scripts with the user.login method without a following user.logout.

IPv6 address issue in SNMPv3 traps

Due to a net-snmp bug, IPv6 address may not be correctly displayed when using SNMPv3 in SNMP traps. For more details and a possible workaround, see ZBX-14541.

Trimmed long IPv6 IP address in failed login information

A failed login attempt message will display only the first 39 characters of a stored IP address as that's the character limit in the database field. That means that IPv6 IP addresses longer than 39 characters will be shown incompletely.

Zabbix agent checks on Windows

Non-existing DNS entries in a Server parameter of Zabbix agent configuration file (zabbix_agentd.conf) may increase Zabbix agent response time on Windows. This happens because Windows DNS caching daemon doesn't cache negative responses for IPv4 addresses. However, for IPv6 addresses negative responses are cached, so a possible workaround to this is disabling IPv4 on the host.

YAML export/import

There are some known issues with YAML export/import:

• Error messages are not translatable;
• Valid JSON with a .yaml file extension sometimes cannot be imported;
• Unquoted human-readable dates are automatically converted to Unix timestamps.

Setup wizard on SUSE with NGINX and php-fpm

Frontend setup wizard cannot save configuration file on SUSE with NGINX + php-fpm. This is caused by a setting in /usr/lib/systemd/system/php-fpm.service unit, which prevents Zabbix from writing to /etc. (introduced in PHP 7.4).

There are two workaround options available:

• Set the ProtectSystem option to 'true' instead of 'full' in the php-fpm systemd unit.
• Manually save /etc/zabbix/web/zabbix.conf.php file.

Chromium for Zabbix web service on Ubuntu 20

Though in most cases, Zabbix web service can run with Chromium, on Ubuntu 20.04 using Chromium causes the following error:

Cannot fetch data: chrome failed to start:cmd_run.go:994:
       WARNING: cannot create user data directory: cannot create 
       "/var/lib/zabbix/snap/chromium/1564": mkdir /var/lib/zabbix: permission denied
       Sorry, home directories outside of /home are not currently supported. See https://forum.snapcraft.io/t/11209 for details.

This error occurs because /var/lib/zabbix is used as a home directory of user 'zabbix'.

MySQL custom error codes

If Zabbix is used with MySQL installation on Azure, an unclear error message [9002] Some errors occurred may appear in Zabbix logs. This generic error text is sent to Zabbix server or proxy by the database. To get more information about the cause of the error, check Azure logs.

Invalid regular expressions after switching to PCRE2

In Zabbix 6.0 support for PCRE2 has been added. Even though PCRE is still supported, Zabbix installation packages for RHEL 7 and newer, SLES (all versions), Debian 9 and newer, Ubuntu 16.04 and newer have been updated to use PCRE2. While providing many benefits, switching to PCRE2 may cause certain existing PCRE regexp patterns becoming invalid or behaving differently. In particular, this affects the pattern ^[\w-\.]. In order to make this regexp valid again without affecting semantics, change the expression to ^[-\w\.] . This happens due to the fact that PCRE2 treats the dash sign as a delimiter, creating a range inside a character class.

Geomap widget error

The maps in the Geomap widget may not load correctly, if you have upgraded from an older Zabbix version with NGINX and didn't switch to the new NGINX configuration file during the upgrade.

To fix the issue, you can discard the old configuration file, use the configuration file from the current version package and reconfigure it as described in the download instructions in section e. Configure PHP for Zabbix frontend.

Alternatively, you can manually edit an existing NGINX configuration file (typically, /etc/zabbix/nginx.conf). To do so, open the file and locate the following block:

location ~ /(api\/|conf[^\.]|include|locale|vendor) {
               deny            all;
               return          404;
       }

Then, replace this block with:

location ~ /(api\/|conf[^\.]|include|locale) {
               deny            all;
               return          404;
       }
       
       location /vendor {
               deny            all;
               return          404;
       }

Use case with global variables shared across webhook calls

As global variables are shared across different webhook calls, the following code will result in the tag value counter gradually increasing:

try 
       {
          aa = aa + 1;
       }
       catch(e)
       {
          aa = 0;
       }

       result = {
               'tags': {
                   'endpoint': aa
               }
           };
       return JSON.stringify(result);

Using local variables instead of global ones is recommended to make sure that each script operates on its own data and that there are no collisions between simultaneous calls.

Server crash with PostgreSQL/TimescaleDB after upgrade from 7.0

Upgrading to Zabbix 7.0.1 (or later) from Zabbix 7.0.0 with PostgreSQL/TimescaleDB results in a server crash. This issue is caused by a workaround to a compression job issue in the auditlog table in Zabbix 7.0 that irreversibly changed the compression policy of the auditlog table.

To fix the issue, please perform a manual rebuild of the auditlog table. The buggy auditlog table can be detected using this query:

SELECT config FROM timescaledb_information.jobs WHERE application_name LIKE 'Compression%' AND hypertable_schema='public' AND hypertable_name='auditlog';.

If it returns a JSON object containing property compress_after (like {"hypertable_id": 14, "compress_after": 612000}) then you should rebuild the table.

The simplest way for rebuilding the auditlog table is:

CREATE TABLE auditlog_tmp (
           LIKE auditlog INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES
       );
       
       SELECT create_hypertable('auditlog_tmp', 'auditid', chunk_time_interval => 604800,
               time_partitioning_func => 'cuid_timestamp', migrate_data => true, if_not_exists => true);
       
       WITH moved_rows AS (
           DELETE FROM auditlog
           RETURNING *
       )
       INSERT INTO auditlog_tmp
       SELECT * FROM moved_rows;
       
       DROP TABLE auditlog;
       ALTER TABLE auditlog_tmp RENAME TO auditlog;

See also TimescaleDB documentation for more optimized ways to migrate data.

Database restore error with PostgreSQL/TimescaleDB after upgrade from 7.0.0-7.0.4

Using pg_restore to restore a PostgreSQL/TimescaleDB backup created in Zabbix 7.0.0-7.0.4 will result in a missing base36_decode function error, causing the restore to fail:

ERROR:  function base36_decode(text) does not exist
       LINE 1: CAST(base36_decode(substring(cuid FROM 2 FOR 8))/1000 AS int...
                    ^
       HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

This error occurs when restoring a backup created with pg_dump.

To fix this issue, please replace the cuid_timestamp function in your Zabbix database before creating the backup (stopping PostgreSQL/TimescaleDB before running the script is recommended):

CREATE OR REPLACE FUNCTION cuid_timestamp(cuid varchar(25)) RETURNS integer AS $$
       DECLARE
           base36 varchar;
           a char[];
           ret bigint;
           i int;
           val int;
           chars varchar;
       BEGIN
           base36 := substring(cuid FROM 2 FOR 8);
       
           chars := '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ';
       
           FOR i IN REVERSE char_length(base36)..1 LOOP
               a := a || substring(upper(base36) FROM i FOR 1)::char;
           END LOOP;
           i := 0;
           ret := 0;
           WHILE i < (array_length(a, 1)) LOOP
               val := position(a[i + 1] IN chars) - 1;
               ret := ret + (val * (36 ^ i));
               i := i + 1;
           END LOOP;
       
           RETURN CAST(ret/1000 AS integer);
       END;
       $$ LANGUAGE 'plpgsql' IMMUTABLE;
       DROP FUNCTION IF EXISTS base36_decode(character varying);

See also ZBX-24955 (for additional details on the error) and TimescaleDB documentation (for additional backup and restore options).