Receiving SNMP traps is the opposite to querying SNMP-enabled devices.
In this case, the information is sent from an SNMP-enabled device to snmptrapd and is collected or "trapped" by Zabbix server or Zabbix proxy from file.
Usually, traps are sent upon some condition change and the agent connects to the server on port 162 (as opposed to port 161 on the agent side that is used for queries). Using traps may detect some short problems that occur amidst the query interval and may be missed by the query data.
Receiving SNMP traps in Zabbix is designed to work with snmptrapd and one of the mechanisms for passing the traps to Zabbix - either a Bash or Perl script or SNMPTT.
The simplest way to set up trap monitoring after configuring Zabbix is to use the Bash script solution, because Perl and SNMPTT are often missing in modern distributions and require more complex configuration. However, this solution uses a script configured as traphandle
. For better performance on production systems, use the embedded Perl solution (either script with do perl
option or SNMPTT).
The workflow of receiving a trap:
snmptrapd
receives a trapsnmptrapd
passes the trap to the receiver script (Bash, Perl) or SNMPTTsnmptrap[regexp]
. The trap is set as the value of all matched items. If no matching item is found and there is an snmptrap.fallback
item, the trap is set as the value of that.During high-availability (HA) node switch, Zabbix will continue processing after the last record within the last ISO 8601 timestamp; if the same record is not found then only the timestamp will be used to identify last position.
Configuring the following fields in the frontend is specific for this item type:
In Data collection → Hosts, in the Host interface field set an SNMP interface with the correct IP or DNS address. The address from each received trap is compared to the IP and DNS addresses of all SNMP interfaces to find the corresponding hosts.
In the Key field use one of the SNMP trap keys:
Key | ||
---|---|---|
Description | Return value | Comments |
snmptrap[regexp] | ||
Catches all SNMP traps that match the regular expression specified in regexp. If regexp is unspecified, catches any trap. | SNMP trap | This item can be set only for SNMP interfaces. User macros and global regular expressions are supported in the parameter of this item key. |
snmptrap.fallback | ||
Catches all SNMP traps that were not caught by any of the snmptrap[] items for that interface. | SNMP trap | This item can be set only for SNMP interfaces. |
Multiline regular expression matching is not supported at this time.
Set the Type of information to 'Log' for the timestamps to be parsed. Note that other formats such as 'Numeric' are also acceptable but might require a custom trap handler.
For SNMP trap monitoring to work, it must first be set up correctly (see below).
To read the traps, Zabbix server or proxy must be configured to start the SNMP trapper process and point to the trap file that is being written by SNMPTT or a Bash/Perl trap receiver. To do that, edit the configuration file (zabbix_server.conf or zabbix_proxy.conf):
If systemd parameter PrivateTmp is used, this file is unlikely to work in /tmp.
Requirements: only snmptrapd.
A Bash trap receiver script can be used to pass traps to Zabbix server from snmptrapd using trapper file. To configure it, add the traphandle
option to snmptrapd configuration file (snmptrapd.conf
), see example.
snmptrapd might need to be restarted to pick up changes to its configuration.
Requirements: Perl, Net-SNMP compiled with --enable-embedded-perl (done by default since Net-SNMP 5.4)
A Perl trap receiver (look for misc/snmptrap/zabbix_trap_receiver.pl) can be used to pass traps to Zabbix server directly from snmptrapd. To configure it:
snmptrapd might need to be restarted to pick up changes to its configuration.
If the script name is not quoted, snmptrapd will refuse to start up with messages, similar to these:
At first, snmptrapd should be configured to use SNMPTT.
For the best performance, SNMPTT should be configured as a daemon using snmptthandler-embedded to pass the traps to it. See instructions for configuring SNMPTT.
When SNMPTT is configured to receive the traps, configure snmptt.ini
:
The "net-snmp-perl" package has been removed in RHEL 8.0-8.2; re-added in RHEL 8.3. For more information, see the known issues.
Now format the traps for Zabbix to recognize them (edit snmptt.conf):
EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
FORMAT ZBXTRAP $aA Device reinitialized (coldStart)
Do not use unknown traps - Zabbix will not be able to recognize them. Unknown traps can be handled by defining a general event in snmptt.conf:
All customized Perl trap receivers and SNMPTT trap configuration must format the trap in the following way:
where
Note that "ZBXTRAP" and "[address]" will be cut out from the message during processing. If the trap is formatted otherwise, Zabbix might parse the traps unexpectedly.
Example trap:
2024-01-11T15:28:47+0200 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" localhost - ZBXTRAP 192.168.1.1 Link down on interface 2. Admin state: 1. Operational state: 2
This will result in the following trap for SNMP interface with IP=192.168.1.1:
2024-01-11T15:28:47+0200 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events"
localhost - Link down on interface 2. Admin state: 1. Operational state: 2
Zabbix has large file support for SNMP trapper files. The maximum file size that Zabbix can read is 2^63 (8 EiB). Note that the filesystem may impose a lower limit on the file size.
Zabbix does not provide any log rotation system - that should be handled by the user. The log rotation should first rename the old file and only later delete it so that no traps are lost:
Because of the trap file implementation, Zabbix needs the file system to support inodes to differentiate files (the information is acquired by a stat() call).
This example uses snmptrapd and a Bash receiver script to pass traps to Zabbix server.
Setup:
zabbix_server.conf
:/usr/sbin/zabbix_trap_handler.sh
:curl -o /usr/sbin/zabbix_trap_handler.sh https://raw.githubusercontent.com/zabbix/zabbix-docker/7.2/Dockerfiles/snmptraps/alpine/conf/usr/sbin/zabbix_trap_handler.sh
Note: Before 7.0 release use this link for script download.
If necessary, adjust the ZABBIX_TRAPS_FILE variable in the script. To use the default value, create the parent directory first:
snmtrapd.conf
(refer to working example)snmptrapd might need to be restarted to pick up changes to its configuration.
Create an SNMP item for testing purposes:
Host SNMP interface IP: 127.0.0.1
Key: snmptrap["linkUp"]
Log time format: yyyy-MM-ddThh:mm:ss
Note that the ISO 8601 date and time format is used.
snmptrapd
for our chosen SNMP protocol version and send test traps using the snmptrap
utility.SNMPv1 and SNMPv2 protocols rely on "community string" authentication. In the example below we will use "secret" as community string. It must be set to the same value on SNMP trap senders.
Please note that while still widely used in production environments, SNMPv2 doesn't offer any encryption and real sender authentication. The data is sent as plain text and therefore these protocol versions should only be used in secure environments such as private network and should never be used over any public or third-party network.
SNMP version 1 isn't really used these days since it doesn't support 64-bit counters and is considered a legacy protocol.
To enable accepting SNMPv1 or SNMPv2 traps you should add the following line to snmptrapd.conf
. Replace "secret" with the SNMP community string configured on SNMP trap senders:
Next we can send a test trap using snmptrap
. We will use the common "link up" OID in this example:
SNMPv3 addresses SNMPv1/v2 security issues and provides authentication and encryption. You can use the MD5 or multiple SHA authentication methods and DES/multiple AES as cipher.
To enable accepting SNMPv3 add the following lines to snmptrapd.conf
:
Please note the "execute" keyword that allows to execute scripts for this user security model.
snmptrap -v 3 -n "" -a SHA -A mypassword -x AES -X mypassword -l authPriv -u traptest -e 0x8000000001020304 localhost 0 linkUp.0
If you wish to use strong encryption methods such as AES192 or AES256, please use net-snmp starting with version 5.8. You might have to recompile it with configure
option: --enable-blumenthal-aes
. Older versions of net-snmp do not support AES192/AES256. See also: http://www.net-snmp.org/wiki/index.php/Strong_Authentication_or_Encryption
In both examples you will see similar lines in your /var/lib/zabbix/snmptraps/snmptraps.log
:
2024-01-30T10:04:23+0200 ZBXTRAP 127.0.0.1
UDP: [127.0.0.1]:56585->[127.0.0.1]:162
DISMAN-EVENT-MIB::sysUpTimeInstance = 2538834
SNMPv2-MIB::snmpTrapOID.0 = IF-MIB::linkUp.0
The item value in Zabbix will be:
2024-01-30 10:04:23 2024-01-30 10:04:21
2024-01-30T10:04:21+0200 UDP: [127.0.0.1]:56585->[127.0.0.1]:162
DISMAN-EVENT-MIB::sysUpTimeInstance = 2538834
SNMPv2-MIB::snmpTrapOID.0 = IF-MIB::linkUp.0
Example with Perl:
2024-01-30T11:42:54+0200 ZBXTRAP 127.0.0.1
PDU INFO:
receivedfrom UDP: [127.0.0.1]:58649->[127.0.0.1]:162
notificationtype TRAP
version 1
community public
errorstatus 0
transactionid 1
requestid 2101882550
messageid 0
errorindex 0
VARBINDS:
DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (457671) 1:16:16.71
SNMPv2-MIB::snmpTrapOID.0 type=6 value=OID: IF-MIB::linkUp.0