13 Storage of secrets

Overview

It is possible to store some sensitive information secretly in HashiCorp Vault KV Secrets Engine - Version 2. Secrets can be saved for:

user macro values
database access credentials

Zabbix provides read-only access to the secrets in Vault, assuming that secrets are managed by someone else.

User macro values

It is possible to store user macro values secretly in Vault.

A "Vault secret" value of a user macro contains a reference path (as 'path:key', for example "secret/zabbix:password").

The following commands may be used to set the value for the path mentioned in example:

# Enable "secret/" mount point if not already enabled, note that "kv-v2" must be used
       vault secrets enable -path=secret/ kv-v2
       
       # Put new secret with key password under mount point "secret/" and path "secret/zabbix"
       vault kv put secret/zabbix password=<password>
       
       # Test that secret is successfully added
       vault kv get secret/zabbix
       
       # Finally test with Curl, note that "data" need to be manually added after mount point and "/v1" before the mount point, also see --capath parameter
       curl --header "X-Vault-Token: <VaultToken>" https://127.0.0.1:8200/v1/secret/data/zabbix

The secret value is retrieved by Zabbix server on every refresh of configuration data and is stored in configuration cache. The authentication token for a read-only access to the reference paths must be provided in server configuration ('VaultToken' parameter). If the macro value cannot be retrieved successfully the corresponding item using the value will turn unsupported.

It is also possible to trigger refresh of secret values from Vault, using a 'secrets_reload' command line option.

Zabbix proxy never communicates with Vault to get any secrets other than database credentials. Secret values on Zabbix proxy are retrieved from Zabbix server on each configuration sync and stored in configuration cache the same way as on Zabbix server.

That means a Zabbix proxy cannot start data collection after a restart until it receives the configuration data update from Zabbix server for the first time. Encryption must be enabled between Zabbix server and proxy; otherwise a server warning message is logged.

Database credentials

It is supported to store database credentials used by Zabbix server, proxies and frontend secretly in Vault:

Vault-related parameters for retrieving database credentials can be optionally entered in the frontend installation wizard.

Database credentials retrieved from Vault will be cached by the frontend. Note that the filesystem temporary file directory is used for database credential caching in frontend. You may use the ZBX_DATA_CACHE_TTL constant to control how often the data cache is refreshed/invalidated.

For server/proxy the VaultDBPath configuration parameter may be used to specify the path from where credentials for database will be retrieved by keys 'password' and 'username' (for example: secret/zabbix/database).

The following commands may be used to set the values for the path mentioned in example:

# Enable "secret/" mount point if not already enabled, note that "kv-v2" must be used
       vault secrets enable -path=secret/ kv-v2
       
       # Put new secrets with keys username and password under mount point "secret/" and path "secret/zabbix/database"
       vault kv put secret/zabbix/database username=zabbix password=<password>
       
       # Test that secret is successfully added
       vault kv get secret/zabbix/database
       
       # Finally test with Curl, note that "data" need to be manually added after mount point and "/v1" before the mount point, also see --capath parameter
       curl --header "X-Vault-Token: <VaultToken>" https://127.0.0.1:8200/v1/secret/data/zabbix/database

Configuration parameters

For Zabbix server/proxy new configuration parameters have been added for Vault authentication and retrieving database credentials:

VaultToken - Vault authentication token (see Zabbix server/proxy configuration file for details)
VaultURL - Vault server HTTP[S] URL
VaultDBPath - Vault path from where credentials for database will be retrieved by keys 'password' and 'username' (for example: secret/zabbix/database)

Zabbix server and Zabbix proxy read the Vault-related configuration parameters from zabbix_server.conf and zabbix_proxy.conf upon startup.

Zabbix server and Zabbix proxy will additionally read "VAULT_TOKEN" environment variable once during startup and unset it so that it would not be available through forked scripts; it is an error if both VaultToken and VAULT_TOKEN contain value.

Forward slash and colon are reserved symbols. Forward slash can only be used to separate mount point from path (e.g. secret/zabbix where mount point is "secret" and "zabbix" is path) and, in case of Vault macros, colon can only be used to separate path from key. It is possible to URL-encode "/" and ":" if there is need to create mount point with name that is separated by forward slash (e.g. foo/bar/zabbix where mount point is "foo/bar" and path is "zabbix" as "foo%2Fbar/zabbix") and if mount point name or path needs to contain colon.

Configuring TLS

Certificate signed by a certificate authority (CA) should be added to the default CA store. Alternatively a custom CA store location can be specified using the SSLCALocation configuration parameter; note that in this case the certificate directory must be prepared using the openssl c_rehash utility, for example configure SSLCALocation and copy "ca.pem" inside that directory, then run the following command:

c_rehash .

Documentation