June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
        • WAN Federation
      • Compatibility Matrix
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Overview
      • task
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.1.x
      • v0.2.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.5.x
      • v0.6.0-beta
    • Overview
    • ACL
  • Guides
Type '/' to Search

»Custom Resource Definitions

This topic describes how to manage Consul configuration entries via Kubernetes Custom Resources. Configuration entries provide cluster-wide defaults for the service mesh.

»Requirements

  • consul-helm 0.28.0 or later
  • consul-k8s 0.22.0 or later
  • consul 1.8.4 or later; some configuration entries require a newer version of Consul

»Supported Configuration Entries

You can specify the following values in the kind field. Click on a configuration entry to view its documentation:

  • Mesh (requires Consul 1.10.0+)
  • ExportedServices
    Enterprise
  • ProxyDefaults
  • ServiceDefaults
  • ServiceSplitter
  • ServiceRouter
  • ServiceResolver
  • ServiceIntentions (requires Consul 1.9.0+)
  • IngressGateway
  • TerminatingGateway

»Installation

Verify that the minimum version of the helm chart (0.28.0) is installed:

$ helm search repo hashicorp/consul
NAME                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/consul    0.28.0          1.9.1       Official HashiCorp Consul Chart
$ helm search repo hashicorp/consul
NAME                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/consul    0.28.0          1.9.1       Official HashiCorp Consul Chart

Update your helm repository cache if necessary:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Next, you must configure consul-helm via your values.yaml to install the custom resource definitions and enable the controller that acts on them:

global:
  name: consul

controller:
  enabled: true

connectInject:
  enabled: true
values.yaml
global:
  name: consul

controller:
  enabled: true

connectInject:
  enabled: true

Note that:

  1. controller.enabled: true installs the CRDs and enables the controller.
  2. Configuration entries are used to configure Consul service mesh so it's also expected that connectInject will be enabled.

See Install with Helm Chart for further installation instructions.

»Upgrading An Existing Cluster to CRDs

If you have an existing Consul cluster running on Kubernetes you may need to perform extra steps to migrate to CRDs. See Upgrade An Existing Cluster to CRDs for full instructions.

»Usage

Once installed, you can use kubectl to create and manage Consul's configuration entries.

»Create

You can create configuration entries via kubectl apply.

$ cat <<EOF | kubectl apply --filename -
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: foo
spec:
  protocol: "http"
EOF

servicedefaults.consul.hashicorp.com/foo created
$ cat <<EOF | kubectl apply --filename -
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: foo
spec:
  protocol: "http"
EOF

servicedefaults.consul.hashicorp.com/foo created

See Configuration Entries for detailed schema documentation.

»Get

You can use kubectl get [kind] [name] to get the status of the configuration entry:

$ kubectl get servicedefaults foo
NAME  SYNCED
foo   True
$ kubectl get servicedefaults foo
NAME  SYNCED
foo   True

The SYNCED status shows whether the configuration entry was successfully created in Consul.

»Describe

You can use kubectl describe [kind] [name] to investigate the status of the configuration entry. If SYNCED is false, the status will contain the reason why.

$ kubectl describe servicedefaults foo
Status:
  Conditions:
    Last Transition Time:  2020-10-09T21:15:50Z
    Status:                True
    Type:                  Synced
$ kubectl describe servicedefaults foo
Status:
  Conditions:
    Last Transition Time:  2020-10-09T21:15:50Z
    Status:                True
    Type:                  Synced

»Edit

You can use kubectl edit [kind] [name] to edit the configuration entry:

$ kubectl edit servicedefaults foo
# change protocol: http => protocol: tcp
servicedefaults.consul.hashicorp.com/foo edited
$ kubectl edit servicedefaults foo
# change protocol: http => protocol: tcp
servicedefaults.consul.hashicorp.com/foo edited

You can then use kubectl get to ensure the change was synced to Consul:

$ kubectl get servicedefaults foo
NAME  SYNCED
foo   True
$ kubectl get servicedefaults foo
NAME  SYNCED
foo   True

»Delete

You can use kubectl delete [kind] [name] to delete the configuration entry:

$ kubectl delete servicedefaults foo
servicedefaults.consul.hashicorp.com "foo" deleted
$ kubectl delete servicedefaults foo
servicedefaults.consul.hashicorp.com "foo" deleted

You can then use kubectl get to ensure the configuration entry was deleted:

$ kubectl get servicedefaults foo
Error from server (NotFound): servicedefaults.consul.hashicorp.com "foo" not found
$ kubectl get servicedefaults foo
Error from server (NotFound): servicedefaults.consul.hashicorp.com "foo" not found

»Delete Hanging

If running kubectl delete hangs without exiting, there may be a dependent configuration entry registered with Consul that prevents the target configuration entry from being deleted. For example, if you set the protocol of your service to http via ServiceDefaults and then create a ServiceSplitter, you won't be able to delete the ServiceDefaults.

This is because by deleting the ServiceDefaults config, you are setting the protocol back to the default which is tcp. Since ServiceSplitter requires that the service has an http protocol, Consul will not allow the ServiceDefaults to be deleted since that would put Consul into a broken state.

In order to delete the ServiceDefaults config, you would need to first delete the ServiceSplitter.

»Kubernetes Namespaces

»Consul OSS

Consul Open Source (Consul OSS) ignores Kubernetes namespaces and registers all services into the same global Consul registry based on their names. For example, service web in Kubernetes namespace web-ns and service admin in Kubernetes namespace admin-ns will be registered into Consul as web and admin with the Kubernetes source namespace ignored.

When creating custom resources to configure these services, the namespace of the custom resource is also ignored. For example, you can create a ServiceDefaults custom resource for service web in the Kubernetes namespace admin-ns even though the web service is actually running in the web-ns namespace (although this is not recommended):

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
  namespace: admin-ns
spec:
  protocol: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: web-ns
spec: ...
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
  namespace: admin-ns
spec:
  protocol: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: web-ns
spec: ...

NOTE: If two custom resources of the same kind and the same name are attempted to be created in different Kubernetes namespaces, the last one created will not be synced.

»ServiceIntentions Special Case

ServiceIntentions are different from the other custom resources because the name of the resource doesn't matter. For other resources, the name of the resource determines which service it configures. For example, this resource configures the service web:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
spec:
  protocol: http
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
spec:
  protocol: http

For ServiceIntentions, because we need to support the ability to create wildcard intentions (e.g. foo => * (allow) meaning that foo can talk to any service), and because * is not a valid Kubernetes resource name, we instead use the field spec.destination.name to configure the destination service for the intention:

# foo => * (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: '*'
  sources:
    - name: foo
      action: allow
---
# foo => web (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: web
  sources:
    - name: foo
      action: allow
# foo => * (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: '*'
  sources:
    - name: foo
      action: allow
---
# foo => web (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: web
  sources:
    - name: foo
      action: allow

NOTE: If two ServiceIntentions resources set the same spec.destination.name, the last one created will not be synced.

»Consul Enterprise
Enterprise

Consul Enterprise supports multiple configurations for how Kubernetes namespaces are mapped to Consul namespaces. The Consul namespace that the custom resource is registered into depends on the configuration being used but in general, you should create your custom resources in the same Kubernetes namespace as the service they're configuring and everything will work as expected.

The details on each configuration are:

  1. Mirroring - The Kubernetes namespace will be "mirrored" into Consul, i.e. service web in Kubernetes namespace web-ns will be registered as service web in the Consul namespace web-ns. In the same vein, a ServiceDefaults custom resource with name web in Kubernetes namespace web-ns will configure that same service.

    This is configured via connectInject.consulNamespaces:

    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        mirroringK8S: true
    
    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        mirroringK8S: true
    
  2. Mirroring with prefix - The Kubernetes namespace will be "mirrored" into Consul with a prefix added to the Consul namespace, i.e. if the prefix is k8s- then service web in Kubernetes namespace web-ns will be registered as service web in the Consul namespace k8s-web-ns. In the same vein, a ServiceDefaults custom resource with name web in Kubernetes namespace web-ns will configure that same service.

    This is configured via connectInject.consulNamespaces:

    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        mirroringK8S: true
        mirroringK8SPrefix: k8s-
    
    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        mirroringK8S: true
        mirroringK8SPrefix: k8s-
    
  3. Single destination namespace - The Kubernetes namespace is ignored and all services will be registered into the same Consul namespace, i.e. if the destination Consul namespace is my-ns then service web in Kubernetes namespace web-ns will be registered as service web in Consul namespace my-ns.

    In this configuration, the Kubernetes namespace of the custom resource is ignored. For example, a ServiceDefaults custom resource with the name web in Kubernetes namespace admin-ns will configure the service with name web even though that service is running in Kubernetes namespace web-ns because the ServiceDefaults resource ends up registered into the same Consul namespace my-ns.

    This is configured via connectInject.consulNamespaces:

    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        consulDestinationNamespace: 'my-ns'
    
    global:
      name: consul
      enableConsulNamespaces: true
      image: hashicorp/consul-enterprise:<tag>-ent
    connectInject:
      consulNamespaces:
        consulDestinationNamespace: 'my-ns'
    

    NOTE: In this configuration, if two custom resources of the same kind and the same name are attempted to be created in two Kubernetes namespaces, the last one created will not be synced.

»ServiceIntentions Special Case (Enterprise)

ServiceIntentions are different from the other custom resources because the name of the resource doesn't matter. For other resources, the name of the resource determines which service it configures. For example, this resource configures the service web:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
spec:
  protocol: http
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: web
spec:
  protocol: http

For ServiceIntentions, because we need to support the ability to create wildcard intentions (e.g. foo => * (allow) meaning that foo can talk to any service), and because * is not a valid Kubernetes resource name, we instead use the field spec.destination.name to configure the destination service for the intention:

# foo => * (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: '*'
  sources:
    - name: foo
      action: allow
---
# foo => web (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: web
  sources:
    - name: foo
      action: allow
# foo => * (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: '*'
  sources:
    - name: foo
      action: allow
---
# foo => web (allow)
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: name-does-not-matter
spec:
  destination:
    name: web
  sources:
    - name: foo
      action: allow

In addition, we support the field spec.destination.namespace to configure the destination service's Consul namespace. If spec.destination.namespace is empty, then the Consul namespace used will be the same as the other config entries as outlined above.

github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager