»Upgrade An Existing Cluster to CRDs

If you have an existing Consul cluster running on Kubernetes you may need to perform extra steps to migrate to CRDs.

You will need to perform extra steps if you are using any of the following configurations:

  • Helm config connectInject.centralConfig.defaultProtocol, e.g.

    connectInject:
      centralConfig:
        defaultProtocol: http
    
  • Or setting the consul.hashicorp.com/connect-service-protocol annotation on your connect pods, e.g.

    annotations:
      'consul.hashicorp.com/connect-service-protocol': 'http'
    
  • Or Helm config connectInject.centralConfig.proxyDefaults, e.g.

    connectInject:
      centralConfig:
        proxyDefaults: |
          {
            "local_connect_timeout_ms": 1000
          }
    

»Why Migrate?

All of the above settings do not support modification after the initial installation of Consul, i.e. they cannot be updated through the Helm chart.

By switching to custom resources, these settings can now be modified.

»Migration Overview

The migration process will consist of identifying which config entries have been created in Consul and adding metadata to them so that they can be managed by a custom resource instead.

»Default Protocol

If you are setting connectInject.centralConfig.defaultProtocol then you must perform the follow steps to migrate to custom resources.

  1. Find existing service-defaults config entries:

    $ consul config list -kind service-defaults
    static-client
    static-server
    
  2. For each entry, export the config to a file:

    $ consul config read -name static-client -kind service-defaults > static-client.json
    
  3. Edit the file and add the key "Meta": {"consul.hashicorp.com/source-datacenter": "dc1"}. Where dc1 is the name of your datacenter. Make sure you add any missing trailing commas required for JSON:

    {
      "Kind": "service-defaults",
      "Name": "static-client",
      "Protocol": "http",
      "MeshGateway": {},
      "Expose": {},
      "CreateIndex": 26,
      "ModifyIndex": 26,
      "Meta": { "consul.hashicorp.com/source-datacenter": "dc1" }
    }
    
  4. Write the updated config entry:

    $ consul config write static-client.json
    Config entry written: service-defaults/static-client
    
  5. Now you're ready to create a custom resource that takes over control of this config entry. The custom resource will look like:

    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: static-client
    spec:
      protocol: 'http'
    

    Where metadata.name is the name of your service and spec.protocol is the default protocol you've set.

  6. When you run kubectl apply on this file, the ServiceDefaults custom resource should be created successfully and its synced status will be True:

    $ cat <<EOF | kubectl apply -f -
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: static-client
    spec:
      protocol: "http"
    EOF
    servicedefaults.consul.hashicorp.com/static-client created
    
    $ kubectl get servicedefaults static-client
    NAME            SYNCED   AGE
    static-client   True     1s
    
  7. Repeat steps 2-6 for each service-defaults config entry.

  8. Finally, delete your connectInject.centralConfig.defaultProtocol Helm config and run Helm upgrade.

    From here on out, you must create a ServiceDefaults custom resource for each new service to set its protocol. As there will no longer be a default.

»Connect Service Protocol Annotation

If you are setting the consul.hashicorp.com/connect-service-protocol annotation on your connect pods, then you must perform the follow steps to migrate to custom resources.

  1. Find existing service-defaults config entries:

    $ consul config list -kind service-defaults
    static-client
    static-server
    
  2. For each entry, export the config to a file:

    $ consul config read -name static-client -kind service-defaults > static-client.json
    
  3. Edit the file and add the key "Meta": {"consul.hashicorp.com/source-datacenter": "dc1"}. Where dc1 is the name of your datacenter. Make sure you add any missing trailing commas required for JSON:

    {
      "Kind": "service-defaults",
      "Name": "static-client",
      "Protocol": "http",
      "MeshGateway": {},
      "Expose": {},
      "CreateIndex": 26,
      "ModifyIndex": 26,
      "Meta": { "consul.hashicorp.com/source-datacenter": "dc1" }
    }
    
  4. Write the updated config entry:

    $ consul config write static-client.json
    Config entry written: service-defaults/static-client
    
  5. Now you're ready to create a custom resource that takes over control of this config entry. The custom resource will look like:

    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: static-client
    spec:
      protocol: 'http'
    

    Where metadata.name is the name of your service and spec.protocol is the default protocol you've set.

  6. When you run kubectl apply on this file, the ServiceDefaults custom resource should be created successfully and its synced status will be True:

    $ cat <<EOF | kubectl apply -f -
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: static-client
    spec:
      protocol: "http"
    EOF
    servicedefaults.consul.hashicorp.com/static-client created
    
    $ kubectl get servicedefaults static-client
    NAME            SYNCED   AGE
    static-client   True     1s
    
  7. Repeat steps 2-6 for each service-defaults config entry.

  8. You can now remove the consul.hashicorp.com/connect-service-protocol annotation from your deployments to indicate it is no longer used.

»Proxy Defaults

If you are setting connectInject.centralConfig.proxyDefaults then you must perform the follow steps to migrate to custom resources.

  1. Get your existing proxy-defaults config:

    $ consul config read -name global -kind proxy-defaults
    {
        "Kind": "proxy-defaults",
        "Name": "global",
        "Config": {
            "local_connect_timeout_ms": 1000
        },
        "MeshGateway": {
            "Mode": "local"
        },
        "Expose": {},
        "CreateIndex": 4,
        "ModifyIndex": 4
    }
    
  2. Export to a file:

    $ consul config read -name global -kind proxy-defaults > proxy-defaults.json
    
  3. Edit the file and add the key "Meta": {"consul.hashicorp.com/source-datacenter": "dc1"}. Where dc1 is the name of your datacenter. Make sure you add any missing trailing commas required for JSON:

    {
      "Kind": "proxy-defaults",
      "Name": "global",
      "Config": {
        "local_connect_timeout_ms": 1000
      },
      "MeshGateway": {
        "Mode": "local"
      },
      "Expose": {},
      "CreateIndex": 4,
      "ModifyIndex": 4,
      "Meta": { "consul.hashicorp.com/source-datacenter": "dc1" }
    }
    
  4. Write the updated config entry:

    $ consul config write proxy-defaults.json
    Config entry written: proxy-defaults/global
    
  5. Now you're ready to create a custom resource that takes over control of this config entry. The custom resource will look like:

    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
    metadata:
      name: global
    spec:
      config:
        local_connect_timeout_ms: 1000
      meshGateway:
        mode: local
    

    Any keys you had under "Config" must be set in YAML. If you previously had "MeshGateway" config this must also be set now under spec.meshGateway. Also, metadata.name must be global.

  6. When you run kubectl apply on this file, the ProxyDefaults custom resource should be created successfully and its synced status will be True:

    $ cat <<EOF | kubectl apply -f -
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
    metadata:
      name: global
    spec:
      config:
        local_connect_timeout_ms: 1000
      meshGateway:
        mode: local
    EOF
    proxydefaults.consul.hashicorp.com/global created
    
    $ kubectl get proxydefaults global
    NAME     SYNCED   AGE
    global   True     1s
    
  7. Finally, delete your connectInject.centralConfig.proxyDefaults Helm config and run Helm upgrade.

    This will have no effect because changes to this config are not picked up by Consul after initial installation.