»Federation Between Kubernetes Clusters

This page describes how to federate multiple Kubernetes clusters. See Multi-Cluster Overview for more information on use-cases and how it works.

»Primary Datacenter

Consul treats each Kubernetes cluster as a separate Consul datacenter. In order to federate clusters, one cluster must be designated the primary datacenter. This datacenter will be responsible for creating the certificate authority that signs the TLS certificates Connect uses to encrypt and authorize traffic. It also handles validating global ACL tokens. All other clusters that are federated are considered secondaries.

»First Time Installation

If you haven't installed Consul on your cluster, continue reading below. If you've already installed Consul on a cluster and want to upgrade it to support federation, see Upgrading An Existing Cluster.

You will need to use the following config.yaml file for your primary cluster, with the possible modifications listed below.

global:
  name: consul
  datacenter: dc1

  # TLS configures whether Consul components use TLS.
  tls:
    # TLS must be enabled for federation in Kubernetes.
    enabled: true

  federation:
    enabled: true
    # This will cause a Kubernetes secret to be created that
    # can be imported by secondary datacenters to configure them
    # for federation.
    createFederationSecret: true

  acls:
    manageSystemACLs: true
    # If ACLs are enabled, we must create a token for secondary
    # datacenters to replicate ACLs.
    createReplicationToken: true

  # Gossip encryption secures the protocol Consul uses to quickly
  # discover new nodes and detect failure.
  gossipEncryption:
    secretName: consul-gossip-encryption-key
    secretKey: key

connectInject:
  # Consul Connect service mesh must be enabled for federation.
  enabled: true

controller:
  enabled: true

meshGateway:
  # Mesh gateways are gateways between datacenters. They must be enabled
  # for federation in Kubernetes since the communication between datacenters
  # goes through the mesh gateways.
  enabled: true
global:  name: consul  datacenter: dc1
  # TLS configures whether Consul components use TLS.  tls:    # TLS must be enabled for federation in Kubernetes.    enabled: true
  federation:    enabled: true    # This will cause a Kubernetes secret to be created that    # can be imported by secondary datacenters to configure them    # for federation.    createFederationSecret: true
  acls:    manageSystemACLs: true    # If ACLs are enabled, we must create a token for secondary    # datacenters to replicate ACLs.    createReplicationToken: true
  # Gossip encryption secures the protocol Consul uses to quickly  # discover new nodes and detect failure.  gossipEncryption:    secretName: consul-gossip-encryption-key    secretKey: key
connectInject:  # Consul Connect service mesh must be enabled for federation.  enabled: true
controller:  enabled: true
meshGateway:  # Mesh gateways are gateways between datacenters. They must be enabled  # for federation in Kubernetes since the communication between datacenters  # goes through the mesh gateways.  enabled: true

Modifications:

  1. The Consul datacenter name is dc1. The datacenter name in each federated cluster must be unique.

  2. ACLs are enabled in the above config file. They can be disabled by setting:

    global:
      acls:
        manageSystemACLs: false
        createReplicationToken: false
    
    global:  acls:    manageSystemACLs: false    createReplicationToken: false

    ACLs secure Consul by requiring every API call to present an ACL token that is validated to ensure it has the proper permissions. If you are only testing Consul, this is not required.

  3. Gossip encryption is enabled in the above config file. To disable it, comment out or delete the gossipEncryption key:

    global:
      # gossipEncryption:
      #   secretName: consul-gossip-encryption-key
      #   secretKey: key
    
    global:  # gossipEncryption:  #   secretName: consul-gossip-encryption-key  #   secretKey: key

    Gossip encryption encrypts the communication layer used to discover other nodes in the cluster and report on failure. If you are only testing Consul, this is not required.

    NOTE: This config assumes you've already created a Kubernetes secret called consul-gossip-encryption-key. See the docs for this setting for more information on how to create this secret.

  4. The default mesh gateway configuration creates a Kubernetes Load Balancer service. If you wish to customize the mesh gateway, for example using a Node Port service or a custom DNS entry, see the Helm reference for that setting.

With your config.yaml ready to go, follow our Installation Guide to install Consul on your primary cluster.

»Upgrading An Existing Cluster

If you have an existing cluster, you will need to upgrade it to ensure it has the following config:

global:
  tls:
    enabled: true
  federation:
    enabled: true
    createFederationSecret: true
  acls:
    manageSystemACLs: true
    createReplicationToken: true
meshGateway:
  enabled: true
global:  tls:    enabled: true  federation:    enabled: true    createFederationSecret: true  acls:    manageSystemACLs: true    createReplicationToken: truemeshGateway:  enabled: true
  1. global.tls.enabled must be true. See Configuring TLS on an Existing Cluster for more information on safely upgrading a cluster to use TLS.

If you've set enableAutoEncrypt: true, this is also supported.

  1. global.federation.enabled must be set to true. This is a new config setting.
  2. If using ACLs, you'll already have global.acls.manageSystemACLs: true. For the primary cluster, you'll also need to set global.acls.createReplicationToken: true. This ensures that an ACL token is created that secondary clusters can use to authenticate with the primary.
  3. Mesh Gateways are enabled with the default configuration. The default configuration creates a Kubernetes Load Balancer service. If you wish to customize the mesh gateway, see the Helm reference for that setting.

With the above settings added to your existing config, follow the Upgrading guide to upgrade your cluster and then come back to the Federation Secret section.

»ProxyDefaults

If you are using consul-helm 0.30.0+ you must also create a ProxyDefaults resource to configure Consul to use the mesh gateways for service mesh traffic.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  meshGateway:
    mode: 'local'
apiVersion: consul.hashicorp.com/v1alpha1kind: ProxyDefaultsmetadata:  name: globalspec:  meshGateway:    mode: 'local'

The spec.meshGateway.mode can be set to local or remote. If set to local, traffic from one datacenter to another will egress through the local mesh gateway. This may be useful if you prefer all your cross-cluster network traffic to egress from the same locations. If set to remote, traffic will be routed directly from the pod to the remote mesh gateway (resulting in one less hop).

Verify that the resource was synced to Consul:

$ kubectl get proxydefaults global
NAME              SYNCED   AGE
global            True     1s
$ kubectl get proxydefaults globalNAME              SYNCED   AGEglobal            True     1s

Its SYNCED status should be True.

»Federation Secret

The federation secret is a Kubernetes secret containing information needed for secondary datacenters/clusters to federate with the primary. This secret is created automatically by setting:

global:
  federation:
    createFederationSecret: true
global:  federation:    createFederationSecret: true

After the installation into your primary cluster you will need to export this secret:

$ kubectl get secret consul-federation -o yaml > consul-federation-secret.yaml
$ kubectl get secret consul-federation -o yaml > consul-federation-secret.yaml

Now you're ready to import the secret into your secondary cluster(s).

Switch kubectl context to your secondary Kubernetes cluster. In this example our context for our secondary cluster is dc2:

$ kubectl config use-context dc2
Switched to context "dc2".
$ kubectl config use-context dc2Switched to context "dc2".

And import the secret:

$ kubectl apply -f consul-federation-secret.yaml
secret/consul-federation configured
$ kubectl apply -f consul-federation-secret.yamlsecret/consul-federation configured

»Federation Secret Contents

The automatically generated federation secret contains:

  • Server certificate authority certificate - This is the certificate authority used to sign Consul server-to-server communication. This is required by secondary clusters because they must communicate with the Consul servers in the primary cluster.

  • Server certificate authority key - This is the signing key for the server certificate authority. This is required by secondary clusters because they need to create server certificates for each Consul server using the same certificate authority as the primary.

  • Consul server config - This is a JSON snippet that must be used as part of the server config for secondary datacenters. It sets:

    • primary_datacenter to the name of the primary datacenter.

    • primary_gateways to an array of IPs or hostnames for the mesh gateways in the primary datacenter. These are the addresses that Consul servers in secondary clusters will use to communicate with the primary datacenter.

      Even if there are multiple secondary datacenters, only the primary gateways need to be configured. Upon first connection with a primary datacenter, the addresses for other secondary datacenters will be discovered.

  • ACL replication token - If ACLs are enabled, secondary datacenters need an ACL token in order to authenticate with the primary datacenter. This ACL token is also used to replicate ACLs from the primary datacenter so that components in each datacenter can authenticate with one another.

  • Gossip encryption key - If gossip encryption is enabled, secondary datacenters need the gossip encryption key in order to be part of the gossip pool. Gossip is the method by which Consul discovers the addresses and health of other nodes.

»Secondary Cluster(s)

With the primary cluster up and running, and the federation secret imported into the secondary cluster, we can now install Consul into the secondary cluster.

You will need to use the following config.yaml file for your secondary cluster(s), with the possible modifications listed below.

global:
  name: consul
  datacenter: dc2
  tls:
    enabled: true

    # Here we're using the shared certificate authority from the primary
    # datacenter that was exported via the federation secret.
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey

  acls:
    manageSystemACLs: true

    # Here we're importing the replication token that was
    # exported from the primary via the federation secret.
    replicationToken:
      secretName: consul-federation
      secretKey: replicationToken

  federation:
    enabled: true
  gossipEncryption:
    secretName: consul-federation
    secretKey: gossipEncryptionKey
connectInject:
  enabled: true
controller:
  enabled: true
meshGateway:
  enabled: true
server:
  # Here we're including the server config exported from the primary
  # via the federation secret. This config includes the addresses of
  # the primary datacenter's mesh gateways so Consul can begin federation.
  extraVolumes:
    - type: secret
      name: consul-federation
      items:
        - key: serverConfigJSON
          path: config.json
      load: true
global:  name: consul  datacenter: dc2  tls:    enabled: true
    # Here we're using the shared certificate authority from the primary    # datacenter that was exported via the federation secret.    caCert:      secretName: consul-federation      secretKey: caCert    caKey:      secretName: consul-federation      secretKey: caKey
  acls:    manageSystemACLs: true
    # Here we're importing the replication token that was    # exported from the primary via the federation secret.    replicationToken:      secretName: consul-federation      secretKey: replicationToken
  federation:    enabled: true  gossipEncryption:    secretName: consul-federation    secretKey: gossipEncryptionKeyconnectInject:  enabled: truecontroller:  enabled: truemeshGateway:  enabled: trueserver:  # Here we're including the server config exported from the primary  # via the federation secret. This config includes the addresses of  # the primary datacenter's mesh gateways so Consul can begin federation.  extraVolumes:    - type: secret      name: consul-federation      items:        - key: serverConfigJSON          path: config.json      load: true

Modifications:

  1. The Consul datacenter name is dc2. The primary datacenter's name was dc1. The datacenter name in each federated cluster must be unique.

  2. ACLs are enabled in the above config file. They can be disabled by removing the whole acls block:

    acls:
      manageSystemACLs: false
      replicationToken:
        secretName: consul-federation
        secretKey: replicationToken
    
    acls:  manageSystemACLs: false  replicationToken:    secretName: consul-federation    secretKey: replicationToken

    If ACLs are enabled in one datacenter, they must be enabled in all datacenters because in order to communicate with that one datacenter ACL tokens are required.

  3. Gossip encryption is enabled in the above config file. To disable it, don't set the gossipEncryption key:

    global:
      # gossipEncryption:
      # secretName: consul-federation
      # secretKey: gossipEncryptionKey
    
    global:  # gossipEncryption:  # secretName: consul-federation  # secretKey: gossipEncryptionKey

    If gossip encryption is enabled in one datacenter, it must be enabled in all datacenters because in order to communicate with that one datacenter the encryption key is required.

  4. The default mesh gateway configuration creates a Kubernetes Load Balancer service. If you wish to customize the mesh gateway, for example using a Node Port service or a custom DNS entry, see the Helm reference for that setting.

With your config.yaml ready to go, follow our Installation Guide to install Consul on your secondary cluster(s).

»Verifying Federation

To verify that both datacenters are federated, run the consul members -wan command on one of the Consul server pods:

$ kubectl exec statefulset/consul-server -- consul members -wan
Node                 Address              Status  Type    Build     Protocol  DC   Segment
consul-server-0.dc1  10.32.4.216:8302     alive   server  1.8.0  2         dc1  <all>
consul-server-0.dc2  192.168.2.173:8302   alive   server  1.8.0  2         dc2  <all>
consul-server-1.dc1  10.32.5.161:8302     alive   server  1.8.0  2         dc1  <all>
consul-server-1.dc2  192.168.88.64:8302   alive   server  1.8.0  2         dc2  <all>
consul-server-2.dc1  10.32.1.175:8302     alive   server  1.8.0  2         dc1  <all>
consul-server-2.dc2  192.168.35.174:8302  alive   server  1.8.0  2         dc2  <all>
$ kubectl exec statefulset/consul-server -- consul members -wanNode                 Address              Status  Type    Build     Protocol  DC   Segmentconsul-server-0.dc1  10.32.4.216:8302     alive   server  1.8.0  2         dc1  <all>consul-server-0.dc2  192.168.2.173:8302   alive   server  1.8.0  2         dc2  <all>consul-server-1.dc1  10.32.5.161:8302     alive   server  1.8.0  2         dc1  <all>consul-server-1.dc2  192.168.88.64:8302   alive   server  1.8.0  2         dc2  <all>consul-server-2.dc1  10.32.1.175:8302     alive   server  1.8.0  2         dc1  <all>consul-server-2.dc2  192.168.35.174:8302  alive   server  1.8.0  2         dc2  <all>

In this example (run from dc1), you can see that this datacenter knows about the servers in dc2 and that they have status alive.

You can also use the consul catalog services command with the -datacenter flag to ensure each datacenter can read each other's services. In this example, our kubectl context is dc1 and we're querying for the list of services in dc2:

$ kubectl exec statefulset/consul-server -- consul catalog services -datacenter dc2
consul
mesh-gateway
$ kubectl exec statefulset/consul-server -- consul catalog services -datacenter dc2consulmesh-gateway

You can switch kubectl contexts and run the same command in dc2 with the flag -datacenter dc1 to ensure dc2 can communicate with dc1.

»Consul UI

We can also use the Consul UI to verify federation. See Viewing the Consul UI for instructions on how to view the UI.

With the UI open, you'll be able to switch between datacenters via the dropdown in the top left:

Consul Datacenter Dropdown

»Next Steps

With your Kubernetes clusters federated, try out using Consul service mesh to route between services deployed on each cluster by following our Learn tutorial: Secure and Route Service Mesh Communication Across Kubernetes.

You can also read our in-depth documentation on Consul Service Mesh In Kubernetes.

If you are still considering a move to Kubernetes, or to Consul on Kubernetes specifically, our Migrate to Microservices with Consul Service Mesh on Kubernetes collection uses an example application written by a fictional company to illustrate why and how organizations can migrate from monolith to microservices using Consul service mesh on Kubernetes. The case study in this collection should provide information valuable for understanding how to develop services that leverage Consul during any stage of your microservices journey.