»Connect Service Mesh on Kubernetes

Connect is a feature built into to Consul that enables automatic service-to-service authorization and connection encryption across your Consul services. Connect can be used with Kubernetes to secure pod communication with other pods and external Kubernetes services.

The Connect sidecar running Envoy can be automatically injected into pods in your cluster, making configuration for Kubernetes automatic. This functionality is provided by the consul-k8s project and can be automatically installed and configured using the Consul Helm chart.

»Usage

When the Connect injector is installed, the Connect sidecar can be automatically added to all pods. This sidecar can both accept and establish connections using Connect, enabling the pod to communicate to clients and dependencies exclusively over authorized and encrypted connections.

»Accepting Inbound Connections

An example Deployment is shown below with Connect enabled to accept inbound connections. Notice that the Deployment would still be fully functional without Connect. Minimal to zero modifications are required to enable Connect in Kubernetes. Notice also that even though we're using a Deployment here, the same configuration would work on a Pod, a StatefulSet, or a DaemonSet.

This Deployment specification starts a server that responds to any HTTP request with the static text "hello world".

apiVersion: v1
kind: Service
metadata:
  # This name will be the service name in Consul.
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http
      # If ACLs are enabled, the serviceAccountName must match the Consul service name.
      serviceAccountName: static-server

apiVersion: v1kind: Servicemetadata:  # This name will be the service name in Consul.  name: static-serverspec:  selector:    app: static-server  ports:    - protocol: TCP      port: 80      targetPort: 8080---apiVersion: v1kind: ServiceAccountmetadata:  name: static-server---apiVersion: apps/v1kind: Deploymentmetadata:  name: static-serverspec:  replicas: 1  selector:    matchLabels:      app: static-server  template:    metadata:      name: static-server      labels:        app: static-server      annotations:        'consul.hashicorp.com/connect-inject': 'true'    spec:      containers:        - name: static-server          image: hashicorp/http-echo:latest          args:            - -text="hello world"            - -listen=:8080          ports:            - containerPort: 8080              name: http      # If ACLs are enabled, the serviceAccountName must match the Consul service name.      serviceAccountName: static-server

The only change for Connect is the addition of the consul.hashicorp.com/connect-inject annotation. This enables injection for the Pod in this Deployment. The injector can also be configured to automatically inject unless explicitly disabled, but the default installation requires opt-in using the annotation shown above.

This will start a sidecar proxy that listens on port 20000 registered with Consul and proxies valid inbound connections to port 8080 in the pod. To establish a connection to the pod using Connect, a client must use another Connect proxy. The client Connect proxy will use Consul service discovery to find all available upstream proxies and their public ports.

In the example above, the server is listening on :8080. By default, the Consul Service Mesh runs in transparent proxy mode. This means that even though the server binds to all interfaces, the inbound and outbound connections will automatically go through to the sidecar proxy. It also allows you to use Kubernetes DNS like you normally would without the Consul Service Mesh.

The service name registered in Consul will be set to the name of the Kubernetes service associated with the Pod. This can be customized with the consul.hashicorp.com/connect-service annotation. If using ACLs, this name must be the same as the Pod's ServiceAccount name.

»Connecting to Connect-Enabled Services

The example Deployment specification below configures a Deployment that is capable of establishing connections to our previous example "static-server" service. The connection to this static text service happens over an authorized and encrypted connection via Connect.

apiVersion: v1
kind: Service
metadata:
  # This name will be the service name in Consul.
  name: static-client
spec:
  selector:
    app: static-client
  ports:
    - port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-client
  template:
    metadata:
      name: static-client
      labels:
        app: static-client
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
    spec:
      containers:
        - name: static-client
          image: curlimages/curl:latest
          # Just spin & wait forever, we'll use `kubectl exec` to demo
          command: ['/bin/sh', '-c', '--']
          args: ['while true; do sleep 30; done;']
      # If ACLs are enabled, the serviceAccountName must match the Consul service name.
      serviceAccountName: static-client

apiVersion: v1kind: Servicemetadata:  # This name will be the service name in Consul.  name: static-clientspec:  selector:    app: static-client  ports:    - port: 80---apiVersion: v1kind: ServiceAccountmetadata:  name: static-client---apiVersion: apps/v1kind: Deploymentmetadata:  name: static-clientspec:  replicas: 1  selector:    matchLabels:      app: static-client  template:    metadata:      name: static-client      labels:        app: static-client      annotations:        'consul.hashicorp.com/connect-inject': 'true'    spec:      containers:        - name: static-client          image: curlimages/curl:latest          # Just spin & wait forever, we'll use `kubectl exec` to demo          command: ['/bin/sh', '-c', '--']          args: ['while true; do sleep 30; done;']      # If ACLs are enabled, the serviceAccountName must match the Consul service name.      serviceAccountName: static-client

By default when ACLs are enabled or when ACLs default policy is allow, Consul will automatically configure proxies with all upstreams from the same datacenter. When ACLs are enabled with default deny policy, you must supply an intention to tell Consul which upstream you need to talk to.

When upstreams are specified explicitly with the consul.hashicorp.com/connect-service-upstreams annotation, the injector will also set environment variables <NAME>_CONNECT_SERVICE_HOST and <NAME>_CONNECT_SERVICE_PORT in every container in the Pod for every defined upstream. This is analogous to the standard Kubernetes service environment variables, but point instead to the correct local proxy port to establish connections via Connect.

We can verify access to the static text server using kubectl exec. Because transparent proxy is enabled by default, we use Kubernetes DNS to connect to our desired upstream.

$ kubectl exec deploy/static-client -- curl --silent http://static-server/
"hello world"

$ kubectl exec deploy/static-client -- curl --silent http://static-server/"hello world"

We can control access to the server using intentions. If you use the Consul UI or CLI to create a deny intention between "static-client" and "static-server", connections are immediately rejected without updating either of the running pods. You can then remove this intention to allow connections again.

$ kubectl exec deploy/static-client -- curl --silent http://static-server/
command terminated with exit code 52

$ kubectl exec deploy/static-client -- curl --silent http://static-server/command terminated with exit code 52

»Available Annotations

Pod annotations can be used to configure the injection behavior.

• consul.hashicorp.com/connect-inject - If this is "true" then injection is enabled. If this is "false" then injection is explicitly disabled. The default injector behavior requires pods to opt-in to injection by specifying this value as "true". This default can be changed in the injector's configuration if desired.

• consul.hashicorp.com/transparent-proxy - If this is "true", this Pod will run with transparent proxy enabled. This means you can use Kubernetes DNS to access upstream services and all inbound and outbound traffic within the pod is redirected to go through the proxy.

• consul.hashicorp.com/transparent-proxy-overwrite-probes - If this is "true" and transparent proxy is enabled, the Connect injector will overwrite Kubernetes HTTP probes to point to the Envoy proxy.

• consul.hashicorp.com/transparent-proxy-exclude-inbound-ports - A comma-separated list of inbound ports to exclude from traffic redirection when running in transparent proxy mode.

• consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs - A comma-separated list of outbound CIDRs to exclude from traffic redirection when running in transparent proxy mode.

• consul.hashicorp.com/transparent-proxy-exclude-outbound-ports - A comma-separated list of outbound ports to exclude from traffic redirection when running in transparent proxy mode.

• consul.hashicorp.com/transparent-proxy-exclude-uids - A comma-separated list of additional user IDs to exclude from traffic redirection when running in transparent proxy mode.

• consul.hashicorp.com/connect-service - For pods that accept inbound connections, this specifies the name of the service that is being served. This defaults to the name of the Kubernetes service associated with the pod.

  If using ACLs, this must be the same name as the Pod's ServiceAccount.

• consul.hashicorp.com/connect-service-port - For pods that accept inbound connections, this specifies the port to route inbound connections to. This is the port that the service is listening on. The service port defaults to the first exposed port on any container in the pod. If specified, the value can be the name of a configured port, such as "http" or it can be a direct port value such as "8080". This is the port of the service, the proxy public listener will listen on a dynamic port.

• consul.hashicorp.com/connect-service-upstreams - The list of upstream services that this pod needs to connect to via Connect along with a static local port to listen for those connections. When transparent proxy is enabled, this annotation is optional.

  • Services

    The name of the service is the name of the service registered with Consul. You can optionally specify datacenters with this annotation.

    annotations:
      "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter]"
    

    annotations:  "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter]"

    Copy

  • Consul Enterprise Namespaces

    If running Consul Enterprise 1.7+, your upstream services may be running in different namespaces. The upstream namespace can be specified after the service name as [service-name].[namespace]. See Consul Enterprise Namespaces below for more details on configuring the injector.

    annotations:
      "consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace]:[port]:[optional datacenter]"
    

    annotations:  "consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace]:[port]:[optional datacenter]"

    Copy

    NOTE: If the namespace is not specified it will default to the namespace of the source service.

    WARNING: Setting a namespace when not using Consul Enterprise or using a version < 1.7 is not supported. It will be treated as part of the service name.

  • Prepared Query

    annotations:
      'consul.hashicorp.com/connect-service-upstreams': 'prepared_query:[query name]:[port]'
    

    annotations:  'consul.hashicorp.com/connect-service-upstreams': 'prepared_query:[query name]:[port]'

    Copy

  • Multiple Upstreams

    If you would like to specify multiple services or upstreams, delimit them with commas

    annotations:
      "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],[service-name]:[port]:[optional datacenter]"
    

    annotations:  "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],[service-name]:[port]:[optional datacenter]"

    Copy

    annotations:
      "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],prepared_query:[query name]:[port]"
    

    annotations:  "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],prepared_query:[query name]:[port]"

    Copy

• consul.hashicorp.com/envoy-extra-args - A space-separated list of arguments to be passed to the injected envoy binary.

  annotations:
    consul.hashicorp.com/envoy-extra-args: '--log-level debug --disable-hot-restart'
  

  annotations:  consul.hashicorp.com/envoy-extra-args: '--log-level debug --disable-hot-restart'

  Copy

• consul.hashicorp.com/service-tags - A comma separated list of tags that will be applied to the Consul service and its sidecar.

  annotations:
    consul.hashicorp.com/service-tags: foo,bar,baz
  

  annotations:  consul.hashicorp.com/service-tags: foo,bar,baz

  Copy

• consul.hashicorp.com/service-meta-<YOUR_KEY> - Set Consul meta key/value pairs that will be applied to the Consul service and its sidecar. The key will be what comes after consul.hashicorp.com/service-meta-, e.g. consul.hashicorp.com/service-meta-foo: bar will result in foo: bar.

  annotations:
    consul.hashicorp.com/service-meta-foo: baz
    consul.hashicorp.com/service-meta-bar: baz
  

  annotations:  consul.hashicorp.com/service-meta-foo: baz  consul.hashicorp.com/service-meta-bar: baz

  Copy

• consul.hashicorp.com/sidecar-proxy- - Override default resource settings for the sidecar proxy container. The defaults are set in Helm config via the connectInject.sidecarProxy.resources key.

  • consul.hashicorp.com/sidecar-proxy-cpu-limit - Override the default CPU limit.
  • consul.hashicorp.com/sidecar-proxy-cpu-request - Override the default CPU request.
  • consul.hashicorp.com/sidecar-proxy-memory-limit - Override the default memory limit.
  • consul.hashicorp.com/sidecar-proxy-memory-request - Override the default memory request.

• consul.hashicorp.com/enable-metrics - Override the default Helm value connectInject.metrics.defaultEnabled.

• consul.hashicorp.com/enable-metrics-merging - Override the default Helm value connectInject.metrics.defaultEnableMerging.

• consul.hashicorp.com/merged-metrics-port - Override the default Helm value connectInject.metrics.defaultMergedMetricsPort.

• consul.hashicorp.com/prometheus-scrape-port - Override the default Helm value connectInject.metrics.defaultPrometheusScrapePort.

• consul.hashicorp.com/prometheus-scrape-path - Override the default Helm value connectInject.metrics.defaultPrometheusScrapePath.

• consul.hashicorp.com/service-metrics-port - Set the port where the Connect service exposes metrics.

• consul.hashicorp.com/service-metrics-path - Set the path where the Connect service exposes metrics.

»Installation and Configuration

The Connect sidecar proxy is injected via a mutating admission webhook provided by the consul-k8s project. This enables the automatic pod mutation shown in the usage section above. Installation of the mutating admission webhook is automated using the Helm chart.

To install the Connect injector, enable the Connect injection feature using Helm values and upgrade the installation using helm upgrade for existing installs or helm install for a fresh install.

connectInject:
  enabled: true

controller:
  enabled: true

connectInject:  enabled: true
controller:  enabled: true

This will configure the injector to inject when the injection annotation is set to true. Other values in the Helm chart can be used to limit the namespaces the injector runs in, enable injection by default, and more.

»Controlling Injection Via Annotation

By default, the injector will inject only when the injection annotation on the pod (not the deployment) is set to true:

annotations:
  'consul.hashicorp.com/connect-inject': 'true'

annotations:  'consul.hashicorp.com/connect-inject': 'true'

»Injection Defaults

If you wish for the injector to always inject, you can set the default to true in the Helm chart:

connectInject:
  enabled: true
  default: true

connectInject:  enabled: true  default: true

You can then exclude specific pods via annotation:

annotations:
  'consul.hashicorp.com/connect-inject': 'false'

annotations:  'consul.hashicorp.com/connect-inject': 'false'

»Controlling Injection Via Namespace

You can control which Kubernetes namespaces are allowed to be injected via the k8sAllowNamespaces and k8sDenyNamespaces keys:

connectInject:
  enabled: true
  k8sAllowNamespaces: ['*']
  k8sDenyNamespaces: []

connectInject:  enabled: true  k8sAllowNamespaces: ['*']  k8sDenyNamespaces: []

In the default configuration (shown above), services from all namespaces are allowed to be injected. Whether or not they're injected depends on the value of connectInject.default and the consul.hashicorp.com/connect-inject annotation.

If you wish to only enable injection in specific namespaces, you can list only those namespaces in the k8sAllowNamespaces key. In the configuration below only the my-ns-1 and my-ns-2 namespaces will be enabled for injection. All other namespaces will be ignored, even if the connect inject annotation is set.

connectInject:
  enabled: true
  k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
  k8sDenyNamespaces: []

connectInject:  enabled: true  k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']  k8sDenyNamespaces: []

If you wish to enable injection in every namespace except specific namespaces, you can use * in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:

connectInject:
  enabled: true
  k8sAllowNamespaces: ['*']
  k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']

connectInject:  enabled: true  k8sAllowNamespaces: ['*']  k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']

»Consul Enterprise Namespaces

Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered into Consul, you can control which Consul namespace they are registered into.

There are three options available:

1. Single Destination Namespace – Register all Kubernetes pods, regardless of namespace, into the same Consul namespace.

   This can be configured with:

   global:
     enableConsulNamespaces: true
   
   connectInject:
     enabled: true
     consulNamespaces:
       consulDestinationNamespace: 'my-consul-ns'
   

   global:  enableConsulNamespaces: true
   connectInject:  enabled: true  consulNamespaces:    consulDestinationNamespace: 'my-consul-ns'

   Copy

   NOTE: If the destination namespace does not exist we will create it.

2. Mirror Namespaces - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace. For example, pod foo in Kubernetes namespace ns-1 will be synced to the Consul namespace ns-1. If a mirrored namespace does not exist in Consul, it will be created.

   This can be configured with:

   global:
     enableConsulNamespaces: true
   
   connectInject:
     enabled: true
     consulNamespaces:
       mirroringK8S: true
   

   global:  enableConsulNamespaces: true
   connectInject:  enabled: true  consulNamespaces:    mirroringK8S: true

   Copy

3. Mirror Namespaces With Prefix - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace with a prefix. For example, given a prefix k8s-, pod foo in Kubernetes namespace ns-1 will be synced to the Consul namespace k8s-ns-1.

   This can be configured with:

   global:
     enableConsulNamespaces: true
   
   connectInject:
     enabled: true
     consulNamespaces:
       mirroringK8S: true
       mirroringK8SPrefix: 'k8s-'
   

   global:  enableConsulNamespaces: true
   connectInject:  enabled: true  consulNamespaces:    mirroringK8S: true    mirroringK8SPrefix: 'k8s-'

   Copy

»Consul Enterprise Namespace Upstreams

When transparent proxy is enabled and ACLs are disabled, the upstreams will be configured automatically across Consul namespaces. When ACLs are enabled, you must configure it by specifying an intention, allowing services across Consul namespaces to talk to each other.

If you wish to specify an upstream explicitly via the consul.hashicorp.com/connect-service-upstreams annotation, use the format [service-name].[namespace]:[port]:[optional datacenter]:

annotations:
  'consul.hashicorp.com/connect-inject': 'true'
  'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'

annotations:  'consul.hashicorp.com/connect-inject': 'true'  'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'

See consul.hashicorp.com/connect-service-upstreams for more details.

»Verifying the Installation

To verify the installation, run the "Accepting Inbound Connections" example from the "Usage" section above. After running this example, run kubectl get pod static-server --output yaml. In the raw YAML output, you should see injected Connect containers and an annotation consul.hashicorp.com/connect-inject-status set to injected. This confirms that injection is working properly.

If you do not see this, then use kubectl logs against the injector pod and note any errors.