Discovery rules (LLD)

Use low-level discovery as much as possible. This helps to avoid unsupported items as well as to improve templates flexibility.

Configuration

Naming

Choose a simple, descriptive name for each discovery rule. Make sure it always ends with the “discovery” word.

Items, triggers, graphs names generated from LLD should be prefixed with the discovery entity name they belong to. The only exception is the singleton discovery pattern.

Update interval

Low-level discovery is considered a heavy operation in Zabbix, so its frequency should be low. Consider always starting at 1 per hour.

If discovery uses another frequent item as a source (Item type = dependent item) - apply “Discard unchanged with heartbeat” preprocessing for such discovery. You can also use such preprocessing for other discoveries too.

In such case, you can also use discovery preprocessing to filter out toggling parts of the low-level discovery data, for example, for data coming from master item:

[ 
        {
         “volume_name”: “my disk1”,
         “volume_size”:  1000000000000,
         “volume_used”:  800000000000,
         “volume_updated_at”: “2019-07-01 00:00:00”
        },
        {
         “volume_name”: “my disk2”,
         “volume_size”:  1000000000000,
         “volume_used”:  800000000000,
         “volume_updated_at”: “2019-07-01 00:00:00”
        }
       ]

For such output consider transforming this array using JS or JSONPath preprocessing to:

[ 
        {
         “volume_name”: “my disk1”
        },
        {
         “volume_name”: “my disk2”
        }
       ]

Before applying throttling discard rule.

Keep lost resources period

Keep it to default: 30d.

Filters

Use user macros in filters

Consider using user macros in filters, two for each useful LLD macro.

That way, filters can be redefined on a linked template or host level without changing the template itself.

SNMP

When discovering SNMP OIDs, make sure to use regexes that match both MIB translated values and raw values. This would make discovery filters even if proper MIBs are not loaded into Zabbix server or proxy.

For example, filter for discovering disks when you need to discover only disks of resource type = hrStorageFixedDisk (1.3.6.1.2.1.25.2.1.4)

Tags

LLD items and triggers should contain the same tags as regular items and triggers, but LLD items may have additional custom tags with LLD macros.

For example, the low-level discovery item Interface {#IFNAME}({#IFALIAS}): Operational status might contain the following tags:

 component: network; description: {#IFALIAS}; interface: {#IFNAME}

Where applicable, configure LLD host discovery rule to assign the host tag service to discovered hosts to specify resource references.

For example, discovered hosts "load_balancer", "web01", "web02", "webforum" should all contains the following tag:

service:webserver

Discovery with Zabbix trapper

When pushing items via Zabbix trapper protocol – consider pushing low-level discovery data as well since discovery items support it.

Use preprocessing to build low-level discovery on the fly

With JavaScript preprocessing and other powerful features, you can create low-level discovery data on the fly. Prefer this method over external discovery scripts:

• To keep discovery rules clearly observable by all future template users
• To keep discovery as a part of the monitoring solution – easily transferable as part of the template
• To avoid external dependencies such as external discovery scripts

Example 1

Get Nginx Plus zones stats using Zabbix HTTP agent from URL such as this: http://demo.nginx.com/api/3/http/server_zones

{
         "hg.nginx.org": {
           "processing": 0,
           "requests": 175276,
           "responses": {
             "1xx": 0,
             "2xx": 162948,
             "3xx": 10117,
             "4xx": 2125,
             "5xx": 8,
             "total": 175198
           },
           "discarded": 78,
           "received": 50484208,
           "sent": 7356417338
         },
        "trac.nginx.org": {
           "processing": 7,
           "requests": 448613,
           "responses": {
             "1xx": 0,
             "2xx": 305562,
             "3xx": 87065,
             "4xx": 23136,
             "5xx": 5127,
             "total": 420890
           },
           "discarded": 27716,
           "received": 137307886,
           "sent": 3989556941
         }
       }

Feed this output to discovery rule via dependent item and apply Javascript preprocessing as this:

//parsing NGINX plus output:
       output = Object.keys(JSON.parse(value)).map(function(zone){
           return {"{#NGINX_ZONE}": zone}
       })
       return JSON.stringify({"data": output})

Making original JSON object a fully LLD compatible JSON Array that can be used for NGINX zones discovery.

Example 2

Get disks stats using Zabbix agent vfs.file.contents[/proc/diskstats] item:

   7       0 loop0 2 0 10 0 0 0 0 0 0 0 0
          7       1 loop1 0 0 0 0 0 0 0 0 0 0 0
          7       2 loop2 0 0 0 0 0 0 0 0 0 0 0
          7       3 loop3 0 0 0 0 0 0 0 0 0 0 0
          7       4 loop4 0 0 0 0 0 0 0 0 0 0 0
          7       5 loop5 0 0 0 0 0 0 0 0 0 0 0
          7       6 loop6 0 0 0 0 0 0 0 0 0 0 0
          7       7 loop7 0 0 0 0 0 0 0 0 0 0 0
          8       0 sda 192218 21315 11221888 13020540 28630719 8482221 801446972 388811708 0 265066852 401774948
          8       1 sda1 252 59 11294 5424 6 0 12 464 0 4160 5888
          8       2 sda2 4 0 8 72 0 0 0 0 0 72 72
          8       5 sda5 191918 21256 11208378 13014352 22872982 8482221 801446960 215739516 0 99497600 228699704
        252       0 dm-0 186763 0 10985130 22979168 31930494 0 799946248 396490524 0 265080476 419505356
        252       1 dm-1 26897 0 220608 688352 187589 0 1500712 23501956 0 212608 24190464

Feed this output to a regular item and then apply preprocessing as this:

JAVASCRIPT

var parsed = value.split("\n").reduce(function(acc, x, i) {
         acc["values"][x.split(/ +/)[3]] = x.split(/ +/).slice(1)
         acc["lld"].push({"{#DEVNAME}":x.split(/ +/)[3]});
         return acc;
       }, {"values":{}, "lld": []});
       
       return JSON.stringify(parsed);

Create new discovery rule with the item above as the master item. Apply additional preprocessing to this discovery rule:

JSONPATH

$.lld

Singleton discovery

While the low-level discovery was designed to automate the creation of items, triggers, and graphs for multiple similar entities such as network interfaces or disks, it can also be used as a simple filter for exclusive entities that either don’t exist or exist in the single instance.

This approach allows keeping a template clean, without users facing unsupported items when a template is applied to hosts with different configurations or versions of the monitored object.

To use singleton pattern, you need to do the following:

• Create discovery rule. Use regular items or dependent items to get some value that is not in LLD format. For a brief example, lets it a regular item that returns the text ‘found’ or ‘missing’.
• Use preprocessing in low-level discovery rule:
  • Check that received value matches your conditions and that items should be created
  • Using Javascript preprocessing, add an empty LLD macro named {#SINGLETON} inside LLD array of length 1

These two steps can be combined in a single line of JavaScript that would generate an LLD array.

return JSON.stringify(value === 'found' ? [{'{#SINGLETON}': ''}] : []);

Use this macro {#SINGLETON} inside square brackets of all item prototypes keys.

Append this macro to any graph prototype name.

Empty macro is required, so Zabbix can differentiate item or graph from the prototype. When macro is expanded after discovery - only clean item name or graph name can be seen absolutely identical to the one that you would statically define.

See MPM event discovery in Zabbix 4.4 “Template App Apache by HTTP” template as an example. We will also describe it in more detail in our blog.