»Vault as Secrets Backend Overview
By default, Consul Helm chart will expect that any credentials it needs are stored as Kubernetes secrets. As of Consul 1.11 and Consul Helm chart v0.38.0, we integrate more natively with Vault making it easier to use Consul Helm chart with Vault as the secrets storage backend.
At a high level, there are two points of integration with Vault:
- Gossip encryption - The encryption key for gossip communication is stored in Vault.
- TLS certificates and keys:
- Consul Server TLS credentials - TLS certificate and key for the Consul server is stored in Vault and issued from Vault.
- Service Mesh and Consul client TLS credentials - Consul uses Vault as the provider for mTLS certificates and keys for the service mesh services and TLS certificates and keys for the Consul clients.
- Vault 1.9+ and Vault-k8s 0.14+ is required.
- Vault must be installed and accessible to the Consul on Kubernetes installation.
global.tls.enableAutoencrypt=trueis required if TLS is enabled for the Consul installation when using the Vault secrets backend.
- The Vault installation must have been initialized, unsealed and the KV2 and PKI secrets engines enabled and the Kubernetes Auth Method enabled.
»Vault Helm Config
A minimal valid installation of Vault Kubernetes must include the Agent Injector which is utilized for accessing secrets from Vault. Vault servers could be deployed
external to Vault on Kubernetes as described via the
externalvaultaddr value in the Vault Helm Configuration
injector: enabled: "true"
»Vault Kubernetes Auth Method
Prior to creating Vault auth roles for the Consul servers and clients, ensure that the Vault Kubernetes auth method is enabled:
$ vault auth enable kubernetes
After enabling the Kubernetes auth method, in Vault, ensure that you have configured the Kubernetes Auth method properly as described in Kubernetes Auth Method Configuration. The command should look similar to the following with a custom
kubernetes_host config provided from the information provided via
$ vault write auth/kubernetes/config \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
»Vault KV Secrets Engine - Version 2
In order to utilize Vault as a secrets backend, we must enable the Vault KV secrets engine - Version 2.
$ vault secrets enable -path=consul kv-v2
»Vault PKI Engine
The Vault PKI Engine must be enabled in order to leverage Vault for issuing Consul Server TLS certificates. More details for configuring the PKI Engine is found in Bootstrapping the PKI Engine under the Server TLS section.
$ vault secrets enable pki
- Multi-DC Federation is not currently supported.
- Certificate rotation for Server TLS certs is not currently supported through the Helm chart. Ensure the TTL for your Server TLS certificates are sufficiently long. Should your certificates expire it will be necessary to issue a
consul reloadon each server after issuing new Server TLS certs from Vault.
- CA rotation is not currently supported through the Helm chart and must be manually rotated.
To utilize Vault as a secrets backend with Consul it is necessary to add several configuration fields to the Vault installation
which bootstrap Vault Auth roles and Policies for Consul to use. For the supported Vault secrets please see the individual secret
guides and ensure to, when combining the secrets, append the Vault Policies to your Vault Kube Auth Roles via a comma separated value (i.e.
$ vault write auth/kubernetes/role/consul-server \ bound_service_account_names=<Consul server service account> \ bound_service_account_namespaces=<Consul installation namespace> \ policies=gossip-policy,consul-ca,consul-server \ ttl=1h
The Vault integration with Consul on Kubernetes makes use of the Vault Agent Injectors. Kubernetes annotations are added to the deployments of the Consul components which cause the Vault Agent Injector to be added as an init-container that will then attach Vault secrets to Consul's pods at startup. Additionally the Vault Agent sidecar is added to the Consul component pods which is responsible for synchronizing and reissuing secrets at runtime. As a result of these additional sidecar containers the typical location for logging is expanded in the Consul components.
As a general rule the best way to troubleshoot startup issues for your Consul installation when using the Vault integration
is to establish if the
vault-agent-init container has completed or not via
kubectl logs -f <your-consul-component> -c vault-agent-int
and checking to see if the secrets have completed rendering.
- If the secrets are not properly rendered the underlying problem will be logged in
vault-agent-initinit-container and generally is related to the Vault Kube Auth Role not having the correct policies for the specific secret e.g.
global.secretsBackend.vault.consulServerRolenot having the correct policies for TLS.
- If the secrets are rendered and the
vault-agent-initcontainer has completed AND the Consul component has not become
Ready, this generally points to an issue with Consul being unable to utilize the Vault secret. This can occur if, for example, the Vault Role created for the PKI engine does not have the correct
alt_namesor otherwise is not properly configured. The best logs for this circumstance are the Consul container logs:
kubectl logs -f <your-consul-component> -c consul.