»Rotating Gossip Encryption Key

The following instructions provides a step-by-step manual process for rotating gossip encryption keys on Consul clusters that are deployed onto a Kubernetes cluster with Consul on Kubernetes.

Note: In case of federated Consul clusters, perform the following steps in the primary datacenter.

Generate a new key:
```
consul keygen
```
```
consul keygen
```
This should generate a new key which can be used as the gossip encryption key. In this example, we will be using Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= as the replacement gossip encryption key.

Add new key to consul keyring.

kubectl exec into a Consul Agent pod (Server or Client) and add the new key to the Consul Keyring. This can be performed by running the following command:

Note: If ACLs are enabled, export the bootstrap token as the CONSUL_HTTP_TOKEN to perform all consul keyring operations. The bootstrap token can be found in the Kubernetes secret consul-bootstrap-acl-token of the primary datacenter.

consul keyring -install="Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="
==> Installing new gossip encryption key...

consul keyring -install="Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="==> Installing new gossip encryption key...

Consul automatically propagates this encryption key across all Clients and Servers across the cluster and the federation if Consul federation is enabled.

List the keys in the keyring to verify the new key has been installed successfully.

consul keyring -list
==> Gathering installed encryption keys...

WAN:
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [2/2]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [2/2]

dc1 (LAN):
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

dc2 (LAN):
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

consul keyring -list==> Gathering installed encryption keys...
WAN:  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [2/2]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [2/2]
dc1 (LAN):  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]
dc2 (LAN):  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

Use the new key as the gossip encryption key.

Once the new key has been added to the keychain, you can install it as the new gossip encryption key. In order to do so run the following while remaining kubectl exec'd in the Consul Agent pod.

consul keyring -use="Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="
==> Changing primary gossip encryption key...

consul keyring -use="Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="==> Changing primary gossip encryption key...

You can ensure that the key has been propagated to all agents by verifying the number of agents that recognize the key over the number of total agents in the datacenter. Listing them provides that information.

consul keyring -list
==> Gathering installed encryption keys...

WAN:
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [2/2]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [2/2]

dc1 (LAN):
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

dc2 (LAN):
  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]
  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

consul keyring -list==> Gathering installed encryption keys...
WAN:  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [2/2]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [2/2]
dc1 (LAN):  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]
dc2 (LAN):  CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]  Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]

Update the Kubernetes secrets with the latest gossip encryption key.

In order for subsequent helm upgrades commands to be successfully executed without unexpected behavior, it is important to update the gossip encryption Kubernetes Secret with the value of the new gossip encryption key.

The name of the secret which stores the value of the gossip encryption key can be found in the Helm values file:

global:
  gossipEncryption:
     secretName: consul-gossip-encryption-key
     secretKey: key

global:  gossipEncryption:     secretName: consul-gossip-encryption-key     secretKey: key

kubectl patch secret consul-gossip-encryption-key -p '{"stringData":{"key": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'

kubectl patch secret consul-gossip-encryption-key -p '{"stringData":{"key": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'

Note: In the case of federated Consul clusters, update the federation-secret value for the gossip encryption key. The name of the secret and key can be found in the values file of the secondary datacenter.

global:
   gossipEncryption:
     secretName: consul-federation
     secretKey: gossipEncryptionKey

global:   gossipEncryption:     secretName: consul-federation     secretKey: gossipEncryptionKey

kubectl patch secret consul-federation -p '{"stringData":{"gossipEncryptionKey": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'

kubectl patch secret consul-federation -p '{"stringData":{"gossipEncryptionKey": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'

Remove the old key once the new one has been installed successfully.
kubectl exec into a Consul Agent pod (Server or Client) and add the new key to the Consul Keyring. This can be performed by running the following command:
Note: If ACLs are enabled, export the bootstrap token as the CONSUL_HTTP_TOKEN to perform all consul keyring operations.
```
consul keyring -remove="CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA="
==> Removing gossip encryption key...
```
```
consul keyring -remove="CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA="==> Removing gossip encryption key...
```